Tracking: remaining work for the array-API migration

[mwcraig]

This is the canonical checklist for finishing the array-API (array-library-agnostic) migration. It supersedes #909 as the place to track status; #909 and its children (#910–#913) remain as the CuPy-specific findings.

State as of 2026-08-25 (post-#989/#990): the August batch (#972–#980, plus #985 CI coverage), #987 (NaN-aware sum/mean/std/median fallbacks, closing #986), #988 (combiner internals: xp.any/xp.count_nonzero, namespace bool mask dtypes, nbytes-free memory sizing), #989 (array-API median fallback for subtract_overscan, wrapping nanmedian with numpy's NaN-propagating semantics) and #990 (test-only hygiene batch addressing #969/#970 and the test-body numpy-isms) are all merged. #992 (combine-method mask= bucket, 63 → 54) and #993 (gain_correct/flat_correct device=, 54 → 41) are merged too. #995 (namespace dtypes: ccd_process bool, transform_image mask, combine()/ImageFileCollection FITS byte order) and #994 (clip_extrema rank-comparison rewrite) are merged too (2026-08-25, 6724c8e): strict is now 31 on main, and every section-1 bucket that was on the strict failure list is closed. What is left on strict is 18 test-body fixes (section 2) and 13 upstream-blocked tests (section 3). CI exercises numpy, jax, dask (including dask+bottleneck and a dask escape-baseline gate); array-api-strict runs as a soft (continue-on-error) job. CuPy and torch are not in CI. User docs live in docs/array_api.rst. 36 backend_xfail markers remain in the test suite. Later on 2026-08-25, three PRs are open on top of 6724c8e: #996 (narrows the two stale test_cosmicray.py markers, 5 XPASS → 0), #997 (#982 and the correlated add/subtract leak in the wrapper) and #998 (the section-2 test bodies, strict 31 → 15). Merged together locally they give strict 15 failed, 496 passed, 36 skipped, 45 xfailed, 0 xpassed; numpy 557 passed / 35 skipped; jax and dask green apart from xfails. Of the 15, 11 are #929, one each is #936, #983 (test_rebin_ccddata[True-True]) and the two new ccdproc items below (combine(output_file=) writing a namespace array, combine(dtype=int)). All three merged 2026-08-25 (main = 4d7d21b); strict on main verified at exactly those 15. #999 then took the two ccdproc items plus the len(oscan) one-liner; merged 2026-08-26 (main = be80055). Strict on main verified at 13 failed / 503 passed / 36 skipped / 45 xfailed / 0 xpassed, and every remaining failure is upstream-blocked: 8 × #929 via the default sigma_func (median_absolute_deviation), 3 × #929 via Combiner.sigma_clipping (sigma_clip), #936, #983. Section 1 is complete. #946 and #943 are closed. 2026-08-26, later: the #929 half that is fixable on the ccdproc side is now in two PRs off be80055: #1000 (sigma_func → xp-native MAD when the namespace is not numpy; strict 13 → 5) and #1001 (Combiner.sigma_clipping → xp-native clip reproducing astropy.stats.sigma_clip; strict 13 → 10). Numpy keeps astropy in both. Merged together locally: strict 2 failed / 1129 passed / 0 xpassed (#936, #983 only), the two BOUNDARY lines leave the escape baseline, and the dask enforce run is clean.

Current strict result on main (1b9b621): 63 failed, 418 passed, 36 skipped, 46 xfailed, 5 xpassed — down from 85 with #989 + #990. With #993 (711bb26) the strict job was 41; #995 and #994 (6724c8e) take it to 31 failed, 469 passed, 36 skipped, 43 xfailed, 5 xpassed with no new failures; the 5 XPASSes are still the stale test_cosmicray.py markers. The job is continue-on-error, so read the pytest summary line in its log rather than the job conclusion. Every one of the 63 has been traced to a cause (2026-08-25):

#	cause	where	section
12	average_combine/sum_combine/median_combine build the result with CCDData(..., mask=mask); astropy's NDData.mask setter forces numpy and cannot convert a device1 strict array	combiner.py:726, :804, :616	1
9	gain_correct calls xp.asarray(gain_value) without device= ("two different devices")	core.py:968	1
2	flat_correct calls xp.asarray(flat_mean) without device= (same class)	core.py:1057	1
4	ccd_process bad-pixel mask uses builtin bool instead of xp.bool	core.py:383	1
4	clip_extrema fancy-index assignment	combiner.py:406	1
2	combine() passes a numpy dtype (ccd.data.dtype.type) to the requested namespace's asarray — strict warns, filterwarnings=error fails	combiner.py:1082	1
2	uncertainty propagation for multiply/divide delegates to astropy's _propagate_multiply_divide, whose np.abs(...) (nduncertainty.py:832) densifies the strict array and the mixed expression then fails	_ccddata_wrapper_for_array_api.py:305–330	1
1	transform_image applies the user transform to the bool mask; strict rejects 10 * bool_array (numpy silently promotes)	core.py:1153	1
8	astropy.stats densification: default sigma_func (median_absolute_deviation, 5) and sigma_clipping (sigma_clip, 3)	core.py:1291, combiner.py:491	3 (#929)
6	test bodies call astropy CCDData.multiply(...) on strict data to build scaled inputs; astropy's _prepare_then_do_arithmetic does np.result_type(ref, operand) on a strict dtype ("Could not convert Array … to a NumPy dtype") — this is the NDData-arithmetic upstream item already in #940, not a ccdproc code path	test_combiner.py (test_combiner_with_scaling*, test_combiner_result_dtype, test_combine_overwrite_output)	2
5	combine() given FITS filenames / numpy CCDData with no array_package= returns numpy data, and the tests then compare it with strict arrays via xp.all(xpx.isclose(...)); array_package is opt-in by design, so the tests should pass it	test_combine_average_fitsimages, test_combine_numpyndarray, test_combine_average_ccddata, test_combine_limitedmem_fitsimages, test_combine_limitedmem_scale_fitsimages	2
2	test bodies ccd.write() a device1 strict CCDData; astropy.io.fits cannot coerce it	test_combiner_image_file_collection_input, test_combine_image_file_collection_input	2
2	test body xp.asarray(np_mgrid[...]) / 10.0 — integer strict array divided by a float	test_ccdproc.py:304 (test_subtract_overscan_model)	2
1	test helper _make_mean_scaler uses .mean() / np.ma.average — this is the "Combiner.scaling setter bug" from #990's deferred list; the setter is fine	test_combiner.py:52 (test_3d_combiner_with_scaling)	2
1	astropy _arithmetic decorator leaks _config_ccd_requires_unit=False (astropy/astropy#20268)	test_generator_ccds_without_unit	2/3
1	astropy NDData.mask setter (#983's blocker)	test_rebin_ccddata[True-True]	2/3
1	Quantity machinery on a device1 strict array	test_unit_mismatch_behaves_as_expected	3 (#936)

So: 36 are ccdproc bugs fixable now (section 1), 17 are test-body fixes (section 2), and 10 are upstream (#929 ×8, #936, astropy#20268) — plus the rebin/mask-setter one that is fixable with the _mask workaround if we want it. The 5 xpassed are all in test_cosmicray.py: the two test_cosmicray_median_masked_column[masked_array-*] cases (stale since #979) and the three gain_apply=False cases of test_cosmicray_gain_correct_uncertainty_namespace (the gain_apply=True cases still xfail, so that marker needs narrowing to the True parametrizations, not removing).

1. Fixable-now bugs in ccdproc (no upstream dependency)

• Combiner.__init__ builds a nested Array via xp.asarray([ccd.data ...]); array-api-strict requires xp.stack (44 failures) #965 Combiner.__init__ builds a nested xp.asarray([...]); needs xp.stack (44 strict failures — biggest single win)
• subtract_dark(scale=True): device-propagation bug misreported as a unit mismatch under array-api-strict #966 subtract_dark(scale=True) device-propagation bug
• rebin() uses numpy-only .astype() method instead of xp.astype() #967 rebin() uses the .astype() method instead of xp.astype
• create_deviation: float * bool-mask multiplication rejected by array-api-strict (core.py:459) #968 create_deviation float × bool-mask multiplication (core.py:459)
• background_deviation_box discards the xpx.at result — silent no-op on immutable backends #963 background_deviation_box discards the xpx.at result (silent no-op on immutable backends)
• Variance and InverseVariance uncertainty propagation escapes to numpy on jax (only the StdDev wrapper is array-API-aware) #962 Variance/InverseVariance uncertainty wrappers escape to numpy on jax (only StdDev is array-API-aware)
• Array API: dead code in cosmicray_median clobbers masked-input handling #932 dead code in cosmicray_median clobbers masked-input handling
• Consider adding fallback median implementation #906 xp-native fallback median for backends without scipy.ndimage
• Combiner: no array-API fallback for NaN-aware mean/sum/std (only median has one) #986 no array-API fallback for NaN-aware mean/sum/std -- fixed in Add array-API NaN-aware sum, mean and std fallbacks #987, which also merged the Consider adding fallback median implementation #906 median fallback into ccdproc/_nanfuncs.py
• subtract_overscan calls xp.median (core.py:629), which is not in the array API standard — fixed in Add an array-API median fallback for subtract_overscan #989 with a median fallback in _nanfuncs.py
• Combiner.average_combine/sum_combine/median_combine pass mask= to the CCDData(...) constructor (combiner.py:726, :804, :616), so astropy's numpy-forcing mask setter runs on the result. ccd_process already sidesteps the same setter by assigning nccd._mask (core.py:381); do the same here. 12 strict failures — biggest remaining bucket — PR Keep combine result masks in the array namespace #992 (also fixes the same setter sites inside combine() and its numpy-only mask.copy(); the remaining members of the bucket now stop at Array API: Combiner.sigma_clipping and the default sigma_func densify via astropy.stats #929 (median_combine's default sigma_func) or at test_writeable_after_combine, which writes a device1 array to FITS and belongs with the section-2 "write a numpy copy" tests)
• gain_correct calls xp.asarray(gain_value) without device= (core.py:968), so the scalar lands on the default device and the multiply fails when the image is elsewhere. Surfaced by Add an array-API median fallback for subtract_overscan #989; 9 strict failures (test_gain.py ×5, test_gain_correct* ×3, test_ccd_process_does_not_change_input) — PR Fix gain_correct and flat_correct on a non-default device #993 (also makes a plain-number gain a float, since the standard does not promote int64 with float64)
• flat_correct has the same bug for xp.asarray(flat_mean) (core.py:1057). 2 strict failures (test_flat_correct, test_flat_correct_norm_value) — PR Fix gain_correct and flat_correct on a non-default device #993 (the two tests then needed their flats created on the testing device; fixed in the same PR)
• ccd_process builds its bad-pixel mask with xp.asarray(mask, dtype=bool) — Python builtin bool where xp.bool is required (core.py:383; same class as create_deviation: float * bool-mask multiplication rejected by array-api-strict (core.py:459) #968/Make combiner internals array-API compatible #988's fixes). Surfaced by Add an array-API median fallback for subtract_overscan #989; 4 strict failures — PR Fix array-namespace dtype handling in ccd_process, transform_image, combine, and ImageFileCollection #995 (the same four tests then hit log_meta._replace_array_with_placeholder, which assumed anything without __len__ is NDData and raised AttributeError on a strict array; fixed in the same PR, plus the test bodies now build a bool mask and compare with ==)
• Combiner.clip_extrema assigns through fancy indexing (combiner.py:406, IndexError: Fancy indexing __setitem__ is not supported); needs xpx.at or a where-based rewrite. 4 strict failures — PR Compute Combiner.clip_extrema mask with rank comparison instead of scatter #994 (no scatter at all: argsort twice gives each image's per-pixel rank, and ranks < nlow | ranks >= n - nhigh is the mask; bit-identical to the old code on 8400 random trials incl. ties/NaNs)
• combine() converts file-read data with xp.asarray(ccd.data, dtype=ccd.data.dtype.type) (combiner.py:1082, and the uncertainty line below it), handing a numpy dtype to the target namespace; strict warns and the suite's filterwarnings = error turns it into a failure. Let xp.asarray infer the dtype or map it through the namespace. 2 strict failures (test_combine_ccd_with_uncertainty_and_mask_from_fits) — PR Fix array-namespace dtype handling in ccd_process, transform_image, combine, and ImageFileCollection #995 (dropping dtype= alone is not enough: strict and jax reject big-endian >f8 outright, so a _native_numpy helper converts to native byte order in numpy first; the same three sites in ImageFileCollection.ccds/data get the same fix, and its mask has to go through _mask because the public setter forces numpy). Only [mean] clears; [function] then stops at _make_mean_scaler (section 2)
• _CCDDataWrapperForArrayAPI uncertainty: _propagate_multiply/_propagate_divide call super()._propagate_multiply_divide, and astropy's implementation uses np.abs(...) (nduncertainty.py:832), which densifies the strict operand; the wrapper needs to own that formula the way it owns add/subtract (upstream fix would be abs() instead of np.abs; worth a note in Tracking: upstream array-API blockers (astropy, reproject, astroscrappy) #940). 2 strict failures (test_flat_correct_deviation, test_flat_correct_data_uncertainty) — PR Fix gain_correct and flat_correct on a non-default device #993 (the mixin now owns _propagate_multiply_divide; after the device= fix this bucket also sat behind the five test_gain.py failures; the two strict xfail markers in test_ccddata_wrapper_for_array_api.py covering it are dropped)
• Same class of leak for add/subtract with uncertainty_correlation != 0: the wrapper still delegates to astropy's _propagate_add_sub, whose correlation term uses np.sqrt(this * other) (nduncertainty.py:742), so a correlated add/subtract through the wrapper fails on strict with TypeError: Expected Array or Python scalar; got numpy.ndarray. Uncorrelated add/subtract is fine. Not covered by any test and not on the strict failure list; found while covering Fix gain_correct and flat_correct on a non-default device #993's multiply/divide override. Fix is the same shape as Fix gain_correct and flat_correct on a non-default device #993 (the mixin owns the formula with xp.sqrt) — PR Normalise combine()'s raw-module array_package and fix correlated add/subtract uncertainty leak #997 (test_wrapped_arithmetic_correlated_uncertainty now covers add/subtract too)
• transform_image applies transform_func to the bool mask directly (core.py:1153); numpy promotes 10 * mask silently, strict raises. Cast the mask to a numeric dtype before transforming (the mask > 0 afterwards already assumes a numeric result). 1 strict failure (test_transform_image[True-True]) — PR Fix array-namespace dtype handling in ccd_process, transform_image, combine, and ImageFileCollection #995 (cast to the data dtype; the test body also discarded the xpx.at(...).set() result, same class as background_deviation_box discards the xpx.at result — silent no-op on immutable backends #963, and now checks the transformed mask's contents)
• combine() does not normalise a raw module passed as array_package #982 combine() does not normalise a raw module passed as array_package, so combine(files, array_package=dask.array) fails; Combiner.__init__ was fixed in Build Combiner data and mask arrays with xp.stack #976 and the two entry points should agree — PR Normalise combine()'s raw-module array_package and fix correlated add/subtract uncertainty leak #997 (the crash itself went inert with Fix array-namespace dtype handling in ccd_process, transform_image, combine, and ImageFileCollection #995, whose device=-free conversion a raw dask.array accepts, and combine() re-derives xp from the converted array anyway; the normalisation now matches Combiner, an array is still accepted as array_package, and regression tests cover raw numpy/dask.array)
• combine(dtype=...)/Combiner(dtype=...) hand the user's dtype straight to xp.astype/xp.asarray (combiner.py:1104, :191); a builtin Python int is a valid NumPy dtype but array-api-strict rejects it (AttributeError: type object 'int' has no attribute '_np_dtype'). 1 strict failure (test_combiner_result_dtype, whose dtype=int case is the point of the test), reachable since Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 cleared the .multiply() failure in front of it — PR Write combine(output_file=) from a NumPy copy; map dtype to the array namespace #999 (core._namespace_dtype resolves the name with numpy.dtype and looks it up on the namespace; the namespace's own dtype objects pass through)
• subtract_overscan calls len(oscan) on the overscan array (core.py:698, xp.arange(len(oscan))); array-api-strict has no __len__. Masked on the strict job because test_subtract_overscan_model is xfailed on Array API: subtract_overscan model fitting goes through numpy-only astropy.modeling #933 (with oscan.shape[0] in its place the test stops in astropy.modeling.fitting._convert_input, so the marker is right), but it is a one-line ccdproc numpy-ism — PR Write combine(output_file=) from a NumPy copy; map dtype to the array namespace #999
• Combiner.scaling setter has an array-API bug exposed when test bodies stop using numpy .mean() — diagnosed 2026-08-25: not a setter bug. test_3d_combiner_with_scaling fails inside the test helper _make_mean_scaler (test_combiner.py:52, .mean()/np.ma.average), and test_combiner_with_scaling fails earlier on ccd_data.multiply(3) in the test body (astropy NDData arithmetic). Both moved to section 2
• Combiner._get_nan_substituted_data calls the numpy method self._data_arr_mask.any() (combiner.py:517) — fixed in Make combiner internals array-API compatible #988, along with the further numpy-isms the fix exposed in the same functions (.sum(axis=0) mask counts, len() on arrays, list passed to xp.reshape, int→float promotion before sqrt)
• combine() memory sizing uses .nbytes on data/uncertainty/mask/flags — fixed in Make combiner internals array-API compatible #988 via _array_size_in_bytes (element count × finfo/iinfo bit width)
• Builtin bool passed where xp.bool is required, in the mask paths of Combiner and _CCDDataWrapperForArrayAPI — fixed in Make combiner internals array-API compatible #988 (wrapper mask setters plus the test sites building masks in the strict namespace)
• Combiner: median combine and the uncertainty calculation still drop to plain numpy (see comments around combiner.py median/uncertainty paths; CCDData images returned by combiner cannot be written as FITS files #351)

2. Test-suite hygiene

• Test suite: test_subtract_overscan calls numpy-only .copy() on array-API arrays #969 test_subtract_overscan calls numpy-only .copy() — fixed in Fix numpy-isms in strict-backend test bodies (test hygiene) #990 (the two median=True parametrizations went green with Add an array-API median fallback for subtract_overscan #989)
• Test suite: raw np.zeros_like on backend-aware arrays in test_ccdproc_logging.py and test_rebin.py #970 raw np.zeros_like in test_ccdproc_logging.py / test_rebin.py — fixed in Fix numpy-isms in strict-backend test bodies (test hygiene) #990 (test_rebin_ccddata[True-True] stays red on astropy's numpy-forcing CCDData.mask setter, Consider marray as a uniform masked-array representation across array backends #983's blocker)
• Numpy .sum() / .mean() method calls in test_combiner.py test bodies — fixed in Fix numpy-isms in strict-backend test bodies (test hygiene) #990, along with ellipsis indexing, int/float promotion and xp.asarray(list)→xp.stack cases
• test__overscan_schange calls xp.allclose — fixed in Fix numpy-isms in strict-backend test bodies (test hygiene) #990
• Six test_combiner.py tests build scaled inputs with astropy CCDData.multiply(...) on strict data (test_combiner_with_scaling, test_combiner_with_scaling_uncertainty ×3, test_combiner_result_dtype, test_combine_overwrite_output); astropy's np.result_type in _prepare_then_do_arithmetic cannot take a strict dtype. Build the inputs in the namespace (CCDData(ccd.data * 3, unit=...)) instead — this is the NDData-arithmetic limitation already listed in Tracking: upstream array-API blockers (astropy, reproject, astroscrappy) #940 and is what _CCDDataWrapperForArrayAPI exists for. 6 strict failures — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 (the four jax backend_xfail markers for the same .multiply() calls are dropped, they XPASS; test_combiner_with_scaling_uncertainty[average|sum] clear, and the other four stop on the next cause: test_combine_overwrite_output on combine(output_file=) (section 3), test_combiner_result_dtype on combine(dtype=int) (section 1), test_combiner_with_scaling and test_combiner_with_scaling_uncertainty[median_combine] on Array API: Combiner.sigma_clipping and the default sigma_func densify via astropy.stats #929)
• Five combine() tests feed FITS filenames or numpy CCDData without array_package= and then compare the (numpy) result against strict arrays with xp.all(xpx.isclose(...)) (test_combine_average_fitsimages, test_combine_numpyndarray, test_combine_average_ccddata, test_combine_limitedmem_fitsimages, test_combine_limitedmem_scale_fitsimages). Pass array_package=xp (or compare in numpy). 5 strict failures — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 (array_package=xp plus Combiner(ccd_list, xp=xp) for the reference; test_combine_average_ccddata converts its CCDData list by hand because array_package is ignored for CCDData input)
• test_combiner_image_file_collection_input / test_combine_image_file_collection_input write a device1 strict CCDData with ccd.write(); astropy.io.fits cannot coerce it. Write a numpy copy. 2 strict failures — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 (numpy_copy/numpy_ccddata helpers in pytest_fixtures.py; also clears test_writeable_after_combine[average|sum_combine], while [median_combine] stays on Array API: Combiner.sigma_clipping and the default sigma_func densify via astropy.stats #929)
• test_subtract_overscan_model divides an integer strict array by a float (test_ccdproc.py:304); cast np_mgrid to float first. 2 strict failures — expect these to then land on Array API: subtract_overscan model fitting goes through numpy-only astropy.modeling #933 (astropy.modeling) — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 (they do, via the len(oscan) numpy-ism listed in section 1; now backend_xfail("array-api-strict") citing Array API: subtract_overscan model fitting goes through numpy-only astropy.modeling #933)
• _make_mean_scaler (test_combiner.py:52) uses .mean() and np.ma.average — rewrite in xp. 2 strict failures (test_3d_combiner_with_scaling, and test_combine_ccd_with_uncertainty_and_mask_from_fits[function] once Fix array-namespace dtype handling in ccd_process, transform_image, combine, and ImageFileCollection #995 lands) — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 (both pass)
• array-api-strict job: device-mismatch noise from inline array creation in test bodies #946 device-mismatch noise from inline array creation in strict-job test bodies — after the section-1 device= fixes above the remaining device errors are all astropy setters/Quantity, not test bodies; re-check once those land and close if nothing is left — re-checked after Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998: none of the 15 remaining strict failures is a test-body device error; array-api-strict job: device-mismatch noise from inline array creation in test bodies #946 closed 2026-08-25
• test_generator_ccds_without_unit fails on the strict job because of an astropy bug, not test ordering per se: the _arithmetic decorator in astropy/nddata/ccddata.py sets the module-global _config_ccd_requires_unit = False and restores it with no try/finally, so any exception inside CCDData.add/subtract/multiply/divide leaves the unit requirement disabled process-wide. test_unit_mismatch_behaves_as_expected raises inside .subtract() by design, so the flag leaks on every backend — reproduced on plain numpy with just those two tests. The full numpy run stays green only because intervening tests (e.g. test_gain) complete a successful arithmetic pass, which resets the flag; on strict those tests fail too, so nothing repairs it. Filed upstream as CCDData arithmetic leaves unit requirement disabled process-wide if the operation raises astropy#20268; listed in Tracking: upstream array-API blockers (astropy, reproject, astroscrappy) #940 — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998: removing the .multiply() calls from test_combiner.py also removed the incidental successful arithmetic that had been resetting the flag, so the leak surfaced on plain numpy too; the test now resets astropy.nddata.ccddata._config_ccd_requires_unit = True itself and passes on every backend regardless of order, and the marker is gone
• Platform-dependent jax backend behavior: ccds() does not raise ValueError for unit-less files on Linux CI #943 jax ccds() platform-dependent ValueError behaviour (Linux CI only) — same astropy leak as above; the xfail reason at test_image_collection.py:438 should cite CCDData arithmetic leaves unit requirement disabled process-wide if the operation raises astropy#20268 rather than "platform-dependent" — PR Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 drops the marker altogether (see the previous bullet); Platform-dependent jax backend behavior: ccds() does not raise ValueError for unit-less files on Linux CI #943 closed 2026-08-25
• Correct the xfail reason added in Propagate gain correction to L.A.Cosmic uncertainties #958 that cites astroscrappy (astroscrappy is monkeypatched out in that test) — already done on main in d33d7cf

3. Design issues / blocked on upstream (tracked in #940)

• Array API: Combiner.sigma_clipping and the default sigma_func densify via astropy.stats #929 Combiner.sigma_clipping and the default sigma_func densify via astropy.stats — in progress: PR 🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy #1000 (sigma_func) and PR Array-API fallback for Combiner.sigma_clipping when the namespace is not numpy #1001 (sigma_clipping), numpy unchanged, every other namespace xp-native; leaves strict at 2 (Array API: units/Quantity handling with non-numpy arrays #936, Consider marray as a uniform masked-array representation across array backends #983). History: 8 strict failures (5 via median_absolute_deviation at core.py:1291, 3 via sigma_clip at combiner.py:491); the largest remaining bucket once section 1 is done. After Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 it is 11: the three median_combine callers test_combiner_with_scaling, test_combiner_with_scaling_uncertainty[median_combine] and test_writeable_after_combine[median_combine] reach the default sigma_func once their first cause is cleared. With Write combine(output_file=) from a NumPy copy; map dtype to the array namespace #999 merged the split is 8 via sigma_func (core.py:1409; test_sigma_func_for_ccddata, test_combiner_dtype, test_combiner_median, test_combiner_with_scaling, test_combine_result_uncertainty_and_mask[median_combine-*], test_writeable_after_combine[median_combine], test_combiner_with_scaling_uncertainty[median_combine]) and 3 via sigma_clipping (combiner.py:480; test_combiner_sigmaclip_{high,low,single_pix}). The sigma_func half is cheap on the ccdproc side (MAD = median(|x - median(x)|) on top of the _nanfuncs.py median); the sigma_clipping half is harder because the method forwards **kwd and string/callable func/dev_func to astropy — options are an xp-native clip or backend_xfail markers
• combine(output_file=...) writes the namespace result with ccd.write(output_file, ...) (combiner.py:1273); astropy.io.fits needs NumPy, so on strict it fails with AttributeError: 'Array' object has no attribute 'astype' (ccddata.py:399). 1 strict failure (test_combine_overwrite_output, reachable since Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998). Same class as the CPU-only policy (Array API: decide a policy for inherently CPU-only operations #935): the writer needs a NumPy copy — PR Write combine(output_file=) from a NumPy copy; map dtype to the array namespace #999 (new core._to_numpy host-copy helper, device-generic; the writer gets a NumPy CCDData and the namespace result is returned unchanged. This is the first library-side host-copy helper, so Array API: decide a policy for inherently CPU-only operations #935 can build on it)
• Array API: units/Quantity handling with non-numpy arrays #936 Quantity/units handling with non-numpy arrays — 1 strict failure (test_unit_mismatch_behaves_as_expected)
• Exclude masked pixels from the median filter in cosmicray_median #984 cosmicray_median does not exclude masked pixels from the median filter itself, so bright masked regions pull the local median (follow-up to Array API: dead code in cosmicray_median clobbers masked-input handling #932/Honor the input mask in cosmicray_median #979; runs in numpy, so it also belongs with the CPU-only policy in Array API: decide a policy for inherently CPU-only operations #935)
• Consider marray as a uniform masked-array representation across array backends #983 consider marray as a uniform masked-array representation across backends; bounded by astropy's numpy-only NDData.mask setter
• Array API: subtract_overscan model fitting goes through numpy-only astropy.modeling #933 subtract_overscan model fitting goes through numpy-only astropy.modeling
• Array API: wcs_project mixes numpy output from reproject with xp operations #930 wcs_project mixes numpy output from reproject with xp operations
• Tracking: upstream array-API blockers (astropy, reproject, astroscrappy) #940 upstream blockers: astropy.stats.sigma_clip, median_absolute_deviation, block_reduce/block_replicate, NDData arithmetic (np.result_type in _prepare_then_do_arithmetic, np.abs in _propagate_multiply_divide; the _CCDDataWrapperForArrayAPI workaround), the _config_ccd_requires_unit leak (CCDData arithmetic leaves unit requirement disabled process-wide if the operation raises astropy#20268), astropy.modeling, Quantity, reproject (CPU-only), astroscrappy (permanent → document under the CPU-only policy, Array API: decide a policy for inherently CPU-only operations #935), __array__(copy=) warning (Array API: locate the source of __array__(copy=...) DeprecationWarnings in CuPy runs #934)

4. Tooling / infrastructure

• Escape-baseline keys are fragile across Python versions (comprehension frame names, PEP 709) #945 escape-baseline keys are fragile across Python versions (PEP 709)
• Refactor _escape_triage.py: split concerns and register pytest hooks explicitly #947 refactor _escape_triage.py; register pytest hooks explicitly
• Add sparse test backend as a densification detector (and eventual supported library) #938 sparse backend as a densification detector (sub-items in that issue)
• Escape logger only patches np.asarray/np.asanyarray/np.ma.asanyarray; it misses np.array() and ufunc-driven coercion, so it is a sampler rather than a complete detector
• Decide between a parametrized xp fixture (SciPy-style, all backends in one run) and the current one-backend-per-CCDPROC_ARRAY_LIBRARY approach
• Promote the array-api-strict job from soft to required — sections 1–2 have landed (2026-08-26); the residue is exactly 13 upstream-blocked tests (11 × Array API: Combiner.sigma_clipping and the default sigma_func densify via astropy.stats #929, Array API: units/Quantity handling with non-numpy arrays #936, Consider marray as a uniform masked-array representation across array backends #983), of which 11 are fixed by 🍾 Array-API fallback for sigma_func (MAD) when the namespace is not numpy #1000/Array-API fallback for Combiner.sigma_clipping when the namespace is not numpy #1001; the residue after those is Array API: units/Quantity handling with non-numpy arrays #936 and Consider marray as a uniform masked-array representation across array backends #983, which need backend_xfail markers citing the astropy items so the job can be green
• Decide whether CuPy and/or torch get a CI job

5. Close out

• Re-triage the backend_xfail markers after the fixes above and drop the stale ones — the five known-stale ones below are narrowed in PR Narrow two stale backend_xfail markers in test_cosmicray.py #996 (merged; strict XPASS 5 → 0), and Fix strict array-API test bodies in test_combiner/test_ccdproc/test_image_collection #998 drops five jax markers (four .multiply() ones in test_combiner.py, the test_generator_ccds_without_unit one). Known stale before Narrow two stale backend_xfail markers in test_cosmicray.py #996: test_cosmicray.py::test_cosmicray_median_masked_column[masked_array-1.0] and [masked_array-None] XPASS on the strict job, and their reason (a scipy.ndimage.median_filter device failure) no longer applies after Honor the input mask in cosmicray_median #979; the test_cosmicray_gain_correct_uncertainty_namespace marker (test_cosmicray.py:154) XPASSes for the three gain_apply=False cases and should be narrowed to the gain_apply=True parametrizations
• Re-run the suite against CuPy and close the parts of CuPy issues #909–CuPy issues in test_image_collection #913 that are subsumed by the issues above
• Confirm docs/array_api.rst and CHANGES.rst reflect the final set of supported backends

[mwcraig]

Filed the astropy bug from section 2 upstream: astropy/astropy#20268 — the _arithmetic decorator in astropy/nddata/ccddata.py restores _config_ccd_requires_unit = True without try/finally, so a failed CCDData arithmetic call (e.g. the unit-mismatch raise asserted in test_unit_mismatch_behaves_as_expected) disables the unit requirement process-wide, which is why test_generator_ccds_without_unit stops raising when the two tests share a pytest process. Reproduced on plain numpy with astropy 8.0.1; the code on astropy main is identical.

[mwcraig]

Checklist updated 2026-08-25 after #989 and #990 merged. Strict on main (1b9b621): 63 failed, 418 passed, 36 skipped, 46 xfailed, 5 xpassed (was 85). All 63 are now traced to a cause in the table at the top of the description. Changes from the previous revision:

• Checked off the xp.median item (Add an array-API median fallback for subtract_overscan #989), Test suite: test_subtract_overscan calls numpy-only .copy() on array-API arrays #969/Test suite: raw np.zeros_like on backend-aware arrays in test_ccdproc_logging.py and test_rebin.py #970 and the test-body numpy-isms (Fix numpy-isms in strict-backend test bodies (test hygiene) #990).
• The "Combiner.scaling setter bug" from Fix numpy-isms in strict-backend test bodies (test hygiene) #990's deferred list is not a setter bug: the failure is in the test helper _make_mean_scaler (test_combiner.py:52). Moved to section 2.
• The "astropy ndarithmetic cannot convert strict arrays in scaled combinations" item is not a ccdproc code path either: the six tests call CCDData.multiply(...) directly on strict data, which hits np.result_type in astropy's _prepare_then_do_arithmetic. That is the NDData-arithmetic limitation already in Tracking: upstream array-API blockers (astropy, reproject, astroscrappy) #940; the tests should build their inputs in the namespace. Moved to section 2.
• New section-1 items: the combine methods pass mask= to CCDData(...) and so run astropy's numpy-forcing mask setter (12 failures, the biggest remaining bucket; ccd_process already dodges this with _mask), flat_correct has the same missing-device= bug as gain_correct, combine() hands a numpy dtype to the target namespace, the wrapper's multiply/divide uncertainty propagation leaks through astropy's np.abs, and transform_image transforms the bool mask directly.
• Split of the 63: 36 fixable in ccdproc now, 17 test-body fixes, 10 upstream-blocked (Array API: Combiner.sigma_clipping and the default sigma_func densify via astropy.stats #929 ×8, Array API: units/Quantity handling with non-numpy arrays #936, CCDData arithmetic leaves unit requirement disabled process-wide if the operation raises astropy#20268).