Skip to content

🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy - #1000

Open
mwcraig wants to merge 5 commits into
astropy:mainfrom
mwcraig:sigma-func-array-api
Open

🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy#1000
mwcraig wants to merge 5 commits into
astropy:mainfrom
mwcraig:sigma-func-array-api

Conversation

@mwcraig

@mwcraig mwcraig commented Aug 27, 2026

Copy link
Copy Markdown
Member

Part of #929 (the sigma_func half). Companion PR: #1001 (the Combiner.sigma_clipping half), built off the same main.

sigma_func — the default uncertainty_func of median_combine, also used by background_deviation_box/background_deviation_filter — wrapped astropy.stats.median_absolute_deviation, which is numpy-only: on any other array namespace the data was converted to numpy, and on array-api-strict the call failed outright (8 of the 13 remaining strict failures).

Policy (numpy is untouched): when the namespace is numpy, sigma_func still calls astropy exactly as before — same call, same masked-CCDData handling, no new work before the array_api_compat.is_numpy_namespace check. Every other namespace goes to a new private core._mad_fallback that computes the median absolute deviation purely in terms of the array API, on the input's device, using the sort-based medians in _nanfuncs (nanmedian for ignore_nan=True, median otherwise). The fallback promotes int/bool input to the namespace default float, flattens for axis=None, reduces over tuples of axes (permute last + merge), validates axes, and excludes the masked pixels of a CCDData. sigma_func stays the same function object because median_combine tests uncertainty_func is sigma_func.

Also: test_combiner_with_scaling built its reference with xp.asarray((a, b, c)) on a tuple of arrays, which array-api-strict rejects; it now uses xp.stack. The test never got that far before because it failed first in sigma_func.

Tests: the fallback is exercised directly on every backend (the strict job uploads no coverage) against astropy over axis ∈ {None, int, numpy int, tuples incl. negative} × ignore_nan, with int/bool/float32 input and all-NaN slices; the public entry point is checked against astropy.stats.mad_std, for namespace/device of the result, for the fallback branch on numpy (patched is_numpy_namespace), and for the CCDData mask. Design was prototype-verified with zero mismatches vs astropy on numpy, strict (device1), jax and dask.

Docs: changelog entry; the "What limitations should I be aware of?" list in docs/array_api.rst is refreshed (it only mentioned the nanmedian fallback, but nansum/nanmean/nanstd, median and now the MAD also fall back). The core.py sigma_func line leaves the escape baseline (confirmed by a full-suite dask regeneration; the diff is that one line).

Verified locally

🤖 Generated with Claude Code

https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V

mwcraig and others added 4 commits August 26, 2026 20:30
sigma_func wraps astropy.stats.median_absolute_deviation, which is
numpy-only, so on every other array namespace the data was converted to
numpy (and on array-api-strict the call failed outright). This is one of
the two remaining astropy.stats call sites behind astropy#929.

Keep the numpy path exactly as it was -- same astropy call, same masked
CCDData handling, no new work before the is_numpy_namespace check -- and
route every other namespace to a new private core._mad_fallback that
computes the median absolute deviation purely in terms of the array API
on the input's device, using the sort-based medians from _nanfuncs
(nanmedian for ignore_nan=True, median otherwise). The fallback promotes
integer and boolean input to the namespace's default real floating
dtype, flattens for axis=None (background_deviation_box), reduces over
tuples of axes by permuting them last and merging them, validates axes,
and excludes the masked pixels of a CCDData. sigma_func stays the same
function object because median_combine tests uncertainty_func identity.

The new tests exercise the fallback directly on every backend (the
strict job uploads no coverage) against astropy over the axis and
ignore_nan grid, including int/bool/float32 input and all-NaN slices,
and check the public entry point against astropy.stats.mad_std, the
namespace/device of the result, the fallback branch on numpy via a
patched is_numpy_namespace, and the CCDData mask. In test_combiner.py,
test_combiner_with_scaling now builds its reference stack with xp.stack
instead of xp.asarray on a tuple of arrays, which array-api-strict
rejects; the test previously failed earlier, in sigma_func, so this
never surfaced.

Verified: numpy 616 passed; array-api-strict 5 failed (the three
Combiner.sigma_clipping tests, astropy#936, astropy#983 -- down from 13), 0 xpassed;
jax 615 passed; dask 609 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
Add the changelog entry (PR number to be filled in) and bring the
"What limitations should I be aware of?" list in docs/array_api.rst up
to date: it only mentioned the nanmedian fallback, but the combiner also
falls back for nansum/nanmean/nanstd, subtract_overscan for median, and
sigma_func now for the median absolute deviation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
sigma_func no longer converts non-numpy data to numpy, so the escape is
not observed any more. Regenerated with a full-suite dask run
(CCDPROC_ARRAY_LIBRARY=dask CCDPROC_LOG_ARRAY_ESCAPES=1
CCDPROC_WRITE_ESCAPE_BASELINE=1); the only change is this deleted line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGArZMnLPC1G7YP62vN18V
@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.84%. Comparing base (57fd985) to head (b8b0e9f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1000      +/-   ##
==========================================
+ Coverage   97.78%   97.84%   +0.05%     
==========================================
  Files           9        9              
  Lines        1808     1852      +44     
==========================================
+ Hits         1768     1812      +44     
  Misses         40       40              
Flag Coverage Δ
dask 96.85% <93.61%> (-0.09%) ⬇️
jax 97.02% <93.61%> (-0.10%) ⬇️
numpy 97.73% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mwcraig mwcraig changed the title Array-API fallback for sigma_func (MAD) when the namespace is not numpy 🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy Aug 27, 2026
…for astropy+bottleneck

astropy's median_absolute_deviation only honours the mask of an explicit
numpy.ma.MaskedArray, so hand it one when the CCDData has a mask instead of
relying on numpy.nanmedian noticing the mask of CCDData.__array__'s output
(which only happens on its small-array path and never with bottleneck).

The astropy reference in test_mad_fallback_matches_astropy gets a tuple axis
with negative entries normalised: astropy's bottleneck dispatch transposes
with the tuple as given and raises on a negative entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JnWCg95xE93SbhME52jxGJ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant