Array-API fallback for Combiner.sigma_clipping when the namespace is not numpy - #1001
Open
mwcraig wants to merge 4 commits into
Open
Array-API fallback for Combiner.sigma_clipping when the namespace is not numpy#1001mwcraig wants to merge 4 commits into
mwcraig wants to merge 4 commits into
Conversation
Combiner.sigma_clipping wrapped astropy.stats.sigma_clip unconditionally, which converts the data to numpy and fails outright on array-api-strict (ccdproc#929). NumPy users must not be slowed down, so numpy data still go to astropy exactly as before (masked=True, all extra keyword arguments forwarded, compiled fast path intact). Every other namespace now uses _sigma_clip_mask, an iterative clip written purely in terms of the array API standard that reproduces astropy's mask: the last iteration's bounds applied to the original data, non-finite values always masked, None/0 thresholds meaning 3 as in astropy. The 'median'/'mean'/'std'/'mad_std' string options map to the same tiered NaN-aware reductions the combination methods use (new private _nanmadstd for 'mad_std'); astropy-only keyword arguments raise TypeError off numpy. An integer maxiters runs without any host synchronisation, so the clip stays a single graph on dask. Also fixes a latent bug: axis, copy and maxiters were read with kwd.get and then forwarded again through **kwd, so passing any of them raised "TypeError: got multiple values for keyword argument". They are popped now. Docstring fixes: func defaults to 'mean' (not 'median'); None thresholds are treated as 3 rather than "no rejection"; kwd semantics per namespace. Docs: image_combination.rst recommends func="median" over np.ma.median; array_api.rst lists the limitations off numpy. The sigma_clipping escape baseline entry is deleted (confirmed by a full-suite dask regeneration). Verified: the design was checked against astropy.stats.sigma_clip(..., masked=True, copy=False).mask over 864 structured cases (numpy and array-api-strict on device1), 2400 random numpy trials, and 243 cases each on jax and dask, with 0 mismatches. The new differential test (5 data sets x 3 cenfunc x 3 stdfunc x 4 thresholds x 3 maxiters) runs on every backend. Full suite: numpy 1123 passed; array-api-strict 10 failed (down from 13; all remaining are ccdproc#929 sigma_func, ccdproc#936 and ccdproc#983), 0 xpassed; jax 1122 passed; dask 1116 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
This was referenced Aug 27, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1001 +/- ##
==========================================
+ Coverage 97.78% 97.92% +0.13%
==========================================
Files 9 9
Lines 1808 1878 +70
==========================================
+ Hits 1768 1839 +71
+ Misses 40 39 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Record in the test file why _sigma_clip_mask is tested by comparison against astropy over a full cross product rather than a curated list (the corners of astropy's behaviour only appear for particular combinations of center, deviation, thresholds and maxiters), what each data set is for, and what the grid costs. Comments only; no behaviour change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
astropy 8.1 (astropy#19858) changed the python loop that sigma_clip takes for a callable cenfunc/stdfunc: with masked=True it now masks the union of every iteration's rejections regardless of copy, where it used to apply the final bounds to the data for copy=False. Its compiled path (string options) still applies the final bounds, which is what _sigma_clip_mask reproduces. Derive the reference mask from return_bounds=True, which both astropy paths and versions agree on, and keep checking astropy's own mask against it on the compiled path. Docstrings note the corner where astropy >= 8.1's numpy path can differ for a callable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JnWCg95xE93SbhME52jxGJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #929 (the
Combiner.sigma_clippinghalf). Companion PR: #1000 (thesigma_funchalf), built off the samemain; the two touch different files and merge cleanly in either order (the only overlap is adjacent bullets inCHANGES.rst/docs/array_api.rst).Combiner.sigma_clippingwrappedastropy.stats.sigma_clipunconditionally, which converts the data to numpy and fails outright onarray-api-strict(3 of the 13 remaining strict failures).Policy (numpy is untouched): when the namespace is numpy, the data still go to
astropy.stats.sigma_clipexactly as before —masked=True, all extra keyword arguments forwarded, compiled fast path intact. Every other namespace uses a new private_sigma_clip_mask, an iterative clip written purely in terms of the array API that reproduces astropy's mask: the last iteration's bounds applied to the original data, non-finite values always masked,None/0thresholds meaning 3 as in astropy. The'median'/'mean'/'std'/'mad_std'string options map to the same tiered (bottleneck → native → fallback) NaN-aware reductions the combination methods use (new private_nanmadstdfor'mad_std'); astropy-only keyword arguments (grow,masked,return_bounds, …) raiseTypeErroroff numpy. An integermaxitersruns with no host synchronisation, so on dask the clip stays one graph.Latent bug fixed:
axis,copyandmaxiterswere read withkwd.getand then forwarded again through**kwd, so passing any of them raisedTypeError: got multiple values for keyword argument. They are popped now.Docstring fixes:
funcdefaults to'mean', not'median';Nonethresholds are treated as 3 (as astropy does) rather than "no rejection" — the docstring still described the pre-2.4 behaviour. TheNone/0/negative threshold semantics are a behaviour question, not part of this PR — see #1002.Tests: the three
test_combiner_sigmaclip_*tests are parametrized over a backend-generic MAD callable and"mad_std"and no longer use numpy-only.all(). A differential test runs_sigma_clip_maskon every backend againstastropy.stats.sigma_clip(..., masked=True, copy=False).maskover 5 data sets (outliers, NaN/inf incl. an all-NaN column, zero spread, a column that collapses mid-loop, ints) × 3 cenfuncs × 3 stdfuncs × 4 threshold pairs ×maxiters ∈ {1, 3, None}; plus tests for silence on all-NaN slices,None/0→ 3, final-bounds-not-union semantics, numpy scalar thresholds, bad axes/strings,_nanmadstdvsmad_std, a spy proving astropy is called iff numpy, theaxis/maxitersregression, astropy-only kwargs, theuse_astropydeprecation, existing-mask preservation, andcombine(sigma_clip=True)(previously not exercised at all). Design was prototype-verified against astropy over 864 structured cases (numpy + strictdevice1), 2400 random numpy trials and 243 cases each on jax and dask, 0 mismatches.Docs: changelog entry; one bullet in
docs/array_api.rst;docs/image_combination.rstnow recommendsfunc="median"overnp.ma.median. Thecombiner.py sigma_clippingline leaves the escape baseline (confirmed by a full-suite dask regeneration; the diff is that one line).Verified locally
sigma_functests for the companion PR, Array API: units/Quantity handling with non-numpy arrays #936, Consider marray as a uniform masked-array representation across array backends #983), 0 xpassed🤖 Generated with Claude Code
https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V