Add float16 data type (v3 and v2) - #93
Open
konstibob wants to merge 1 commit into
Open
Conversation
float16 is part of the Zarr v3 core spec, not an optional extension -- the
zarr-extensions entry for it is only a pointer back to the core spec's data
types section. Before this change, opening any float16 array failed at metadata
parse with "Cannot deserialize value of type DataType from String float16".
No new codec is needed: the existing bytes codec handles any fixed-itemsize
data type. The work is the in-memory representation, because neither Java nor
ucar.ma2 has a 16-bit float. float16 is therefore held as ucar.ma2.FLOAT and
converted at the codec boundary, which makes it the first data type whose
encoded width (2 bytes) differs from its in-memory width (4 bytes).
- utils/Float16: IEEE 754 binary16 <-> float conversions. Float.float16ToFloat
is Java 20+ and this project targets Java 8, so the bit manipulation is
written out, covering subnormals, signed zeros, infinities, NaN payloads and
round-half-to-even ties.
- core/DataType.isHalfPrecisionFloat(): default false, so the bytes codec can
detect the width mismatch without importing the v2/v3 enums.
- core/codec/core/BytesCodec: widen on decode, narrow on encode. The float16
branch precedes the ArrayFloat branch, since float16 is held in an ArrayFloat
and must not be written as float32.
- core/ArrayMetadata.parseFillValue: accepts numbers, NaN, +/-Infinity, and the
hex/binary forms. The 2 bytes of a hex fill value are a binary16 bit pattern,
not a truncated float32, so they are widened rather than read as a float.
- v3/DataType.FLOAT16 ("float16"), v2/DataType.FLOAT16 ("<f2") and FLOAT16_BE
(">f2"). v2 needed no byte-count change: getByteCount() parses the dtype
string, giving 2 for f2. Blosc typesize follows getByteCount() and so becomes
2, matching the encoded width.
Narrowing is lossy and documented as such: values at or above 65520.0 become
infinity, values below 2^-25 become signed zero, and everything else is rounded
to 11 significand bits. ZarrTest.testdata/assertIsTestdata now pre-quantize
float test values, because 1024 of the 4096 values they generate are not
exactly representable in binary16 (2049 stores as 2048, 4095 as 4096); the
quantized values match np.arange(4096, dtype='float16') element for element.
Verified against numpy 2.4.6 and zarr-python 3.1.6:
- 39 conversion cases asserted against numpy bit patterns, plus an exhaustive
sweep proving all 65536 binary16 patterns round-trip (NaN stays NaN).
- Round-trips in both endiannesses, through blosc/zstd/gzip/sharding/crc32c,
and for v2 and v3.
- Cross-implementation in both directions: zarr-python writes / zarr-java reads
and zarr-java writes / zarr-python reads, for v2 and v3 and both byte orders.
The chunk bytes are byte-identical to zarr-python's in all four
configurations, including the partial trailing chunk.
- Confirmed the interop test has teeth: injecting a symmetric byte-swap into
both encode and decode still passes the zarr-java round-trip, while failing
all 8 interop cases.
Follow-ups, deliberately not attempted here:
- complex64/complex128: ucar.ma2 has no complex type, so the in-memory
representation (trailing dimension of 2 vs. a custom ArrayObject) is a design
decision for the maintainers rather than something to settle in this PR.
- r* (r8, r16, ...) and fixed_length_utf32: parameterized data types. r* needs
its name parsed as a pattern and fixed_length_utf32 carries a configuration
object, so data_type stops being a plain JSON string and the @jsonvalue enum
cannot model either. That is a separate refactor.
- string and variable_length_bytes: these break the fixed-itemsize invariant of
getByteCount() and belong with the vlen work.
- ZarrPythonTests still drives its data type matrix from the hand-maintained
dataTypeProviderV3/V2 lists, which FLOAT16 has been added to. If those are
switched to DataType.values(), note that zarr_python_write.py builds test
data with np.arange, whose float16 values above 2048 are quantized -- the
Java side already accounts for this via ZarrTest.quantize.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
float16 is part of the Zarr v3 core spec, not an optional extension -- the zarr-extensions entry for it is only a pointer back to the core spec's data types section. Before this change, opening any float16 array failed at metadata parse with "Cannot deserialize value of type DataType from String float16".
No new codec is needed: the existing bytes codec handles any fixed-itemsize data type. The work is the in-memory representation, because neither Java nor ucar.ma2 has a 16-bit float. float16 is therefore held as ucar.ma2.FLOAT and converted at the codec boundary, which makes it the first data type whose encoded width (2 bytes) differs from its in-memory width (4 bytes).