🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy - #1000
Open
mwcraig wants to merge 5 commits into
Open
🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy#1000mwcraig wants to merge 5 commits into
mwcraig wants to merge 5 commits into
Conversation
sigma_func wraps astropy.stats.median_absolute_deviation, which is numpy-only, so on every other array namespace the data was converted to numpy (and on array-api-strict the call failed outright). This is one of the two remaining astropy.stats call sites behind astropy#929. Keep the numpy path exactly as it was -- same astropy call, same masked CCDData handling, no new work before the is_numpy_namespace check -- and route every other namespace to a new private core._mad_fallback that computes the median absolute deviation purely in terms of the array API on the input's device, using the sort-based medians from _nanfuncs (nanmedian for ignore_nan=True, median otherwise). The fallback promotes integer and boolean input to the namespace's default real floating dtype, flattens for axis=None (background_deviation_box), reduces over tuples of axes by permuting them last and merging them, validates axes, and excludes the masked pixels of a CCDData. sigma_func stays the same function object because median_combine tests uncertainty_func identity. The new tests exercise the fallback directly on every backend (the strict job uploads no coverage) against astropy over the axis and ignore_nan grid, including int/bool/float32 input and all-NaN slices, and check the public entry point against astropy.stats.mad_std, the namespace/device of the result, the fallback branch on numpy via a patched is_numpy_namespace, and the CCDData mask. In test_combiner.py, test_combiner_with_scaling now builds its reference stack with xp.stack instead of xp.asarray on a tuple of arrays, which array-api-strict rejects; the test previously failed earlier, in sigma_func, so this never surfaced. Verified: numpy 616 passed; array-api-strict 5 failed (the three Combiner.sigma_clipping tests, astropy#936, astropy#983 -- down from 13), 0 xpassed; jax 615 passed; dask 609 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
Add the changelog entry (PR number to be filled in) and bring the "What limitations should I be aware of?" list in docs/array_api.rst up to date: it only mentioned the nanmedian fallback, but the combiner also falls back for nansum/nanmean/nanstd, subtract_overscan for median, and sigma_func now for the median absolute deviation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
sigma_func no longer converts non-numpy data to numpy, so the escape is not observed any more. Regenerated with a full-suite dask run (CCDPROC_ARRAY_LIBRARY=dask CCDPROC_LOG_ARRAY_ESCAPES=1 CCDPROC_WRITE_ESCAPE_BASELINE=1); the only change is this deleted line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
This was referenced Aug 27, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1000 +/- ##
==========================================
+ Coverage 97.78% 97.84% +0.05%
==========================================
Files 9 9
Lines 1808 1852 +44
==========================================
+ Hits 1768 1812 +44
Misses 40 40
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…for astropy+bottleneck astropy's median_absolute_deviation only honours the mask of an explicit numpy.ma.MaskedArray, so hand it one when the CCDData has a mask instead of relying on numpy.nanmedian noticing the mask of CCDData.__array__'s output (which only happens on its small-array path and never with bottleneck). The astropy reference in test_mad_fallback_matches_astropy gets a tuple axis with negative entries normalised: astropy's bottleneck dispatch transposes with the tuple as given and raises on a negative entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JnWCg95xE93SbhME52jxGJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #929 (the
sigma_funchalf). Companion PR: #1001 (theCombiner.sigma_clippinghalf), built off the samemain.sigma_func— the defaultuncertainty_funcofmedian_combine, also used bybackground_deviation_box/background_deviation_filter— wrappedastropy.stats.median_absolute_deviation, which is numpy-only: on any other array namespace the data was converted to numpy, and onarray-api-strictthe call failed outright (8 of the 13 remaining strict failures).Policy (numpy is untouched): when the namespace is numpy,
sigma_funcstill calls astropy exactly as before — same call, same masked-CCDDatahandling, no new work before thearray_api_compat.is_numpy_namespacecheck. Every other namespace goes to a new privatecore._mad_fallbackthat computes the median absolute deviation purely in terms of the array API, on the input's device, using the sort-based medians in_nanfuncs(nanmedianforignore_nan=True,medianotherwise). The fallback promotes int/bool input to the namespace default float, flattens foraxis=None, reduces over tuples of axes (permute last + merge), validates axes, and excludes the masked pixels of aCCDData.sigma_funcstays the same function object becausemedian_combinetestsuncertainty_func is sigma_func.Also:
test_combiner_with_scalingbuilt its reference withxp.asarray((a, b, c))on a tuple of arrays, whicharray-api-strictrejects; it now usesxp.stack. The test never got that far before because it failed first insigma_func.Tests: the fallback is exercised directly on every backend (the strict job uploads no coverage) against astropy over
axis ∈ {None, int, numpy int, tuples incl. negative}×ignore_nan, with int/bool/float32 input and all-NaN slices; the public entry point is checked againstastropy.stats.mad_std, for namespace/device of the result, for the fallback branch on numpy (patchedis_numpy_namespace), and for theCCDDatamask. Design was prototype-verified with zero mismatches vs astropy on numpy, strict (device1), jax and dask.Docs: changelog entry; the "What limitations should I be aware of?" list in
docs/array_api.rstis refreshed (it only mentioned thenanmedianfallback, butnansum/nanmean/nanstd,medianand now the MAD also fall back). Thecore.py sigma_funcline leaves the escape baseline (confirmed by a full-suite dask regeneration; the diff is that one line).Verified locally
sigma_clippingtests for the companion PR, Array API: units/Quantity handling with non-numpy arrays #936, Consider marray as a uniform masked-array representation across array backends #983), 0 xpassed🤖 Generated with Claude Code
https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V