Add array-API NaN-aware sum, mean and std fallbacks

[mwcraig]

Closes #986.

nansum/nanmean/nanstd are not in the array API standard, so on a conforming-but-minimal namespace _default_average, _default_sum and _default_std raised RuntimeError instead of computing — and told the user to install bottleneck, which cannot help there, since those helpers only reach for bottleneck when the namespace is numpy. _default_median got a spec-only fallback in #906/#978; its three siblings did not.

What is here

ccdproc/_nanfuncs.py — spec-only nansum, nanmean and nanstd, and the three helpers wired to them exactly as _default_median is wired to _nanmedian.nanmedian. All three land together because average_combine and sum_combine reach mean/sum before std, so fixing either alone would only move the error a few lines down.

The awkward parts, all of which nanmedian had to handle too:

• All-NaN slices match numpy in value — nansum gives 0.0, nanmean and nanstd give NaN — but do it silently. numpy.nanmean and numpy.nanstd emit a RuntimeWarning there, and this project's pytest configuration turns warnings into errors, so a warning would fail every Combiner test with a fully masked pixel. Zero denominators are swapped for ones before the division rather than patched up after, so no 0/0 is ever evaluated.
• Dtype promotion goes through xp.__array_namespace_info__().default_dtypes(device=...)["real floating"] rather than a hardcoded float64, since jax without JAX_ENABLE_X64 has no float64 and warns when asked for one. This is where nansum deliberately parts company with numpy.nansum, which preserves an integer dtype.
• Device: every scalar built inside the fallbacks carries device=array_api_compat.device(x).
• Axis: a single integer, NotImplementedError otherwise — the same contract nanmedian offers, and all call sites use axis=0.
• nanstd is the two-pass form (ddof=0, matching numpy.nanstd and bottleneck.nanstd). A test with values 1e8 + {0,1,2,3} pins it there: the single-pass sum(x**2) - sum(x)**2/n form loses every significant digit at CCD-count scale.

ccdproc/tests/test_nanfuncs.py — modelled on test_nanmedian.py: all three fallbacks against the numpy reference across dtypes (float, int, bool), 1-D/2-D/3-D, all-NaN and single-value slices, negative and non-zero axes, and the non-default device, independently of whether the active backend happens to provide native versions.

test_bottleneck_defaults_respect_array_namespace is parametrized off a _DEFAULT_FUNCS table naming each helper's fallback, and a new test_defaults_fall_back_without_native_nan_function covers all four against a stand-in namespace.

tox.ini — a strict environment so the array-api-strict suite can be run locally. It pins only the interpreter and inherits setenv, deps, extras and commands through the existing strict factor, so it stays in lockstep with CI by construction; tox -e strict --showconfig is identical to py313-strict apart from the env name. The pin is the point — without a py3xx factor in the name, tox builds the env against whatever interpreter tox itself is installed under, which is often too old.

Verification

• numpy: 492 passed. dask: 481 passed. No regressions.
• array-api-strict (tox -e strict): no No NaN-aware ... errors remain; 266 → 330 passed.

This does not, by itself, lower the strict failure count

It stays at exactly 99. #986 predicted 99 → ~76, but all 23 of the tests it named now get past the reduction and fail one frame later:

next failure	count
'Array' object has no attribute 'any' (combiner.py:517, _get_nan_substituted_data)	21
'Array' object has no attribute 'sum'	2
dtype must be one of the supported dtypes, got <class 'bool'>	1

That .any() wall already blocked six median_combine tests on main, so it is independent of this change — #986's estimate double-counted a blocker it had itself listed under "what remains". The fallbacks are still the necessary fix; they bring average/sum/std to parity with median. Fixing combiner.py:517/:536 is now the highest-leverage strict change, with roughly 23 tests queued behind it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RQMJZUaaxfqGDk41GLSaFK

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.29%. Comparing base (00bfd12) to head (702866c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #987      +/-   ##
==========================================
+ Coverage   96.88%   97.29%   +0.41%     
==========================================
  Files           9        9              
  Lines        1701     1738      +37     
==========================================
+ Hits         1648     1691      +43     
+ Misses         53       47       -6     

Flag	Coverage Δ
dask	96.42% <100.00%> (+0.42%)	⬆️
jax	96.53% <100.00%> (+0.42%)	⬆️
numpy	97.18% <100.00%> (+0.41%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mwcraig]

Why is silence a design goal?

[mwcraig]

Silence isn't the goal — consistency with every other path through the combiner is. Verified: bottleneck.nanmean/nanstd (what the numpy default actually uses) return NaN on an all-NaN slice with no warning, jax's native nanmean is silent, and the _nanmedian fallback is silent. numpy's nan-reductions are the odd ones out. An all-NaN slice is also a routine state here — it's just a fully masked pixel — not an anomaly worth a per-call warning; and with pytest's warnings-as-errors, a warning would fail every Combiner test containing one.

Suggest rewording the docstring so it carries that justification instead of just asserting the behavior, e.g.: "Slices that are entirely NaN yield NaN silently, matching bottleneck.nanmean (the numpy-backend default); numpy.nanmean warns here, but a fully masked pixel is a routine input for the combiner, not an anomaly." Happy to apply.

— Written by Claude at @mwcraig's direction.

[mwcraig]

Critical review focused on (a) correctness of the new fallbacks and (b) test size.

Verdict: the fallbacks are correct. A differential fuzz against numpy (27,000 cases: float32/float64, 1-D through 4-D, negative axes, NaN densities 0/0.3/1.0) found zero mismatches, and the new tests pass on numpy, dask, and array-api-strict (65 passed each). The combiner failures in a full strict run are the pre-existing .any() wall at combiner.py:518 that the PR description already documents; this change does not touch them.

The main opportunity is test size: roughly 50 of the 135 lines in test_nanfuncs.py restate coverage that test_matches_numpy already provides, and merging with test_nanmedian.py would save ~95 lines total. Details inline.

Non-issues I checked deliberately:

• All combiner call sites pass axis=0 by keyword, so the keyword-only fallback signature is compatible with every path (average_combine, sum_combine, _weighted_sum, uncertainty).
• _safe_divide genuinely never evaluates 0/0: with warnings.simplefilter("error") active, none of the three fallbacks warn on an all-NaN slice.
• Slices containing inf match numpy.nanstd (both give NaN).
• float32 input stays float32; only int/bool promote.
• The int64-precision divergence from numpy.nansum (e.g. 2**53 + 1 loses a bit on promotion to float64) is real but exactly what the module docstring discloses.

One nit not worth an inline thread: the 18-line comment on [testenv:strict] in tox.ini could lose its last paragraph — the changedir comment it cites already explains the no-test-factor rule.

[mwcraig]

isinstance(axis, int) has two quirks, both inherited verbatim from _nanmedian: it rejects np.int64(0) with NotImplementedError (numpy integers are not int), yet silently accepts axis=True as axis 1, because bool subclasses int. All ccdproc call sites pass a literal 0, and consistency with nanmedian is a fair defense — but if you touch this, operator.index(axis) plus an explicit bool rejection fixes both, ideally in both modules.

[mwcraig]

Both quirks confirmed locally (isinstance(np.int64(0), int) is False; True % 2 == 1, so axis=True silently reduces axis 1). Since the bool half is silent wrongness rather than a loud refusal, worth fixing now — will apply the operator.index(axis) + explicit bool rejection to _setup here and to _nanmedian in the same commit so the two modules keep the identical contract:

if axis is None or isinstance(axis, bool):
    raise NotImplementedError(...)
try:
    axis = operator.index(axis)
except TypeError:
    raise NotImplementedError(...)

— Written by Claude at @mwcraig's direction.

[Copilot]

Pull request overview

Adds array-API-compatible NaN-aware reduction fallbacks for Combiner.

Changes:

• Implements nansum, nanmean, and two-pass nanstd.
• Uses fallbacks when array namespaces lack native reductions.
• Adds comprehensive tests and a local strict tox environment.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
ccdproc/_nanfuncs.py	Implements the new fallbacks.
ccdproc/combiner.py	Integrates fallbacks into combination operations.
ccdproc/tests/test_nanfuncs.py	Tests values, axes, dtypes, devices, and edge cases.
ccdproc/tests/test_combiner.py	Verifies fallback selection.
tox.ini	Adds the Python 3.13 strict environment.
CHANGES.rst	Documents the fixes and tox environment.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mwcraig]

Responding to the review summary:

On the one summary-level nit — the last paragraph of the [testenv:strict] comment in tox.ini: half agree. It is a cross-reference, not a duplication — the reader it protects is the one about to rename the env, and without a pointer at the rename site the explanation in [testenv] never reaches them (adding a test factor would silently move the run into .tmp/{envname} and break the __file__-anchored escapes baseline). But it doesn't need three lines; will compress to one:

# No "test" factor in the name on purpose -- see the changedir comment in [testenv].

On the test-size theme: agreed in the inline threads to drop test_mean_and_std_all_nan_slice_is_nan and test_nanstd_is_population_deviation, replace the 17-line contextmanager with two filterwarnings marks, and tighten the test_combiner.py parametrization; pushed back on merging with test_nanmedian.py (file↔module mapping) and on dropping test_nansum_all_nan_slice_is_zero (the one counterintuitive absolute value deserves its own pin). Details in each thread.

— Written by Claude at @mwcraig's direction.