Fix array-namespace dtype handling in ccd_process, transform_image, combine, and ImageFileCollection - #995
Merged
Merged
Conversation
…ombine, and ImageFileCollection array_api_strict rejected several ccdproc calls into the array API because they leaked NumPy-specific dtype conventions across the namespace boundary: - ccd_process built its bad-pixel mask with the builtin bool instead of the namespace bool (xp.bool), which strict namespaces reject outright. - transform_image ran the transform function directly on a boolean mask; most transform functions (e.g. scipy.ndimage.shift) only accept numeric input, so the mask is now cast to the data dtype first and re-thresholded afterward, mirroring the existing gain_correct pattern. - combine() and ImageFileCollection.ccds()/data() converted FITS data (read as NumPy, possibly big-endian) to the target namespace by handing a NumPy dtype object to xp.asarray(..., dtype=...). That triggers a UserWarning under array_api_strict (an error under this project warning filters) and outright errors on other backends. A new _native_numpy helper in core.py converts the data to native byte order first, so a plain NumPy array can be handed to xp.asarray. The same fix also uses the private _mask attribute rather than the public mask setter for ImageFileCollection, since CCDData public setter always converts its value back to NumPy. While fixing the above, log_meta.py _replace_array_with_placeholder turned out to crash for a bare (non-NDData) array-API array that does not implement __len__, such as array_api_strict arrays: it fell through to value.data, which raises AttributeError rather than the TypeError it was written to expect. That masked the real fixes above under strict, so it is fixed here too. Part of astropy#971 (the "namespace dtypes" bucket). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
- test_transform_image: fix the xpx.at(...).set(...) call discarding its return value (a silent no-op on JAX/strict, the same class of bug as astropy#963), build the mask as a real boolean array, and assert the transformed mask is True only at the expected pixel. - test_ccd_process / test_ccd_process_gain_corrected: build the bad-pixel mask as boolean (matching the ccd_process fix) and compare it against the result with == instead of xpx.isclose, which strict namespaces only accept numeric dtypes for. - Add test_generator_ccds_converts_to_array_namespace, exercising the same FITS-to-namespace conversion in ImageFileCollection.ccds() that combine() already had a regression test for. Part of astropy#971 (the "namespace dtypes" bucket). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
The previous wording blamed the array's byte order for the strict warning; the warning actually came from handing the NumPy type object to xp.asarray as dtype=, which is what the helper lets us stop doing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
58 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #995 +/- ##
==========================================
+ Coverage 97.47% 97.75% +0.28%
==========================================
Files 9 9
Lines 1781 1785 +4
==========================================
+ Hits 1736 1745 +9
+ Misses 45 40 -5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
These cover the two lines flagged by codecov/patch on PR astropy#995: image_collection.py's data() generator conversion to the collection's array namespace, and log_meta's length=42 fallback in _replace_array_with_placeholder for array-API objects without __len__ or a .data attribute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
mwcraig
commented
Aug 25, 2026
Requested in review: the docstring summary should stay short and the explanation of why the conversion is needed belongs under Notes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME
This was referenced Aug 25, 2026
Normalise combine()'s raw-module array_package and fix correlated add/subtract uncertainty leak
#997
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #971 (the "namespace dtypes" bucket).
Strict-backend count on top of
main711bb26: 41 failed -> 35 failed (456 -> 463 passed; 43 xfailed, 5 xpassed unchanged). No test regresses on strict, numpy, jax, or dask.ccd_process,transform_image,combine(), andImageFileCollection.ccds()/data()all leaked NumPy-specific dtype conventions across the array-namespace boundary: a builtinboolinstead ofxp.boolforccd_process's bad-pixel mask, running a transform function directly on a boolean mask intransform_image, and handing a NumPy dtype object (rather than a plain array) toxp.asarray(..., dtype=...)when converting FITS data to a requested namespace incombine()andImageFileCollection. Each is fixed to speak the array API idiom already used elsewhere in the codebase (xp.bool, casting the mask to the data dtype before transforming, converting to native byte order and lettingxp.asarraydo a plain conversion). A new_native_numpyhelper incore.py, next to_is_array, is shared bycombine()andImageFileCollection.ImageFileCollection's mask conversion also had to switch from the public.masksetter to the private_maskattribute, sinceCCDData's public setter unconditionally converts its value back to NumPy - the same workaroundccd_processalready used, with aTODOto remove it onceCCDDatasupports array namespaces.Fixing
ccd_process's mask dtype uncovered a second, previously-masked bug:log_meta.py's_replace_array_with_placeholder(used by the@log_to_metadatadecorator to build the auto-logging string) crashed for a bare, non-NDDataarray-API array that does not implement__len__- whicharray_api_strictdeliberately does not, per the array API standard. It fell through tovalue.data, which such an array does not have, raisingAttributeErrorinstead of theTypeErrorthe code was written to catch. That crash was maskingccd_process's tests even after the mask-dtype fix, so it is fixed here too (broadened theexceptto also catchAttributeError).Test-body fixes bundled with the corresponding source fix:
test_transform_imagediscarded the return value ofxpx.at(...).set(...)(a silent no-op on JAX/strict, the same class of bug as #963) and did not build a real boolean mask; both are fixed, and the test now asserts the transformed mask isTrueonly at the expected pixel.test_ccd_process/test_ccd_process_gain_correctedbuilt their bad-pixel mask as a float array and compared it against the (now boolean) result withxpx.isclose, which strict namespaces reject for non-numeric dtypes; both now build a boolean mask and compare with==. A newtest_generator_ccds_converts_to_array_namespaceintest_image_collection.pyexercises the same FITS-to-namespace conversion inImageFileCollection.ccds()thatcombine()already had a regression test for (A4 in the working plan; not on the original strict-failure list since no existing test passedarray_package=toImageFileCollection, but the code path is byte-for-byte the same pattern ascombine(), so it is included here).One target test remains failing on strict for a reason outside this PR's scope:
test_combine_ccd_with_uncertainty_and_mask_from_fits[function]reaches a second, unrelated bug once this PR's fix tocombine()clears the first one -test_combiner.py's_make_mean_scalercallsccd_data.data.mean()as a method, whicharray_api_strictarrays do not support (only the top-levelxp.meanfunction form is part of the standard). That is a bug intest_combiner.py, a file intentionally left untouched here to avoid conflicting with a parallel PR (fixingCombiner.clip_extrema) also touching that file;[mean](the other parametrization) passes.🤖 Generated with Claude Code
https://claude.ai/code/session_01S36ZzAAVXVm32vuTdtCQME