diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b9147c..04696df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,12 @@ after its public API and format compatibility policies are established. ### Added +- The canonical `keep.flat-chunks/v1` durable layout specification, typed + `LayoutId` grammar, checked flat-plan bounds, domain-separated checksum, + exact golden records, field-complete `LayoutId` refusal tables and + cardinality-before-aggregate first-failure plan mutation ledger, and + verified storage-profile boundary replay law. Production encoding, decoding, + admission, ingestion, and reconstruction remain assigned to issue #10. - Canonical version-1 `ChunkId` calculation in a domain distinct from `BlobId`, with independent golden vectors. - A constant-memory `FastCdc` detector for `fastcdc-64k-v1` that preserves diff --git a/README.md b/README.md index d3a81fa..5e984b7 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,13 @@ for that profile and a domain-separated `ChunkId`. The detector consumes arbitrary borrowed feed partitions, emits identified spans without retaining candidate bytes, and flushes a final runt only when the caller declares EOF. +The next M2 design slice freezes the canonical `keep.flat-chunks/v1` layout +record and `LayoutId`: one bounded, ordered sequence of exact chunk identities +and logical offsets bound to one `BlobId` and one `StorageProfileId`. Its +language-neutral golden and mutation corpus exists, but the encoder, decoder, +ingestion path, and reconstruction path remain assigned to the next +implementation slice. + Keep does **not** expose ingestion, layouts, or physical storage yet. Durability, retention, recovery, verification of stored structures, and garbage collection remain planned work. Calculating a `BlobId` or `ChunkId` @@ -110,7 +117,10 @@ implemented proof boundary and explicit nonclaims. The separate [CDC profile corpus](conformance/cdc-profile/v1/README.md) freezes M2 boundary semantics, and the [chunk identity invariant](docs/invariants/chunk-identity/README.md) defines -the implemented `ChunkId` and `FastCdc` proof boundary. +the implemented `ChunkId` and `FastCdc` proof boundary. The +[Flat Chunk Layout v1 specification](docs/formats/flat-chunk-layout-v1/README.md) +and [layout corpus](conformance/layout/v1/README.md) freeze the next format +boundary without claiming that its implementation exists. ## Contributing diff --git a/conformance/layout/v1/ORIGIN.md b/conformance/layout/v1/ORIGIN.md new file mode 100644 index 0000000..1254bea --- /dev/null +++ b/conformance/layout/v1/ORIGIN.md @@ -0,0 +1,62 @@ +# Flat Chunk Layout Corpus Origin + +## Provenance + +- Created: 2026-07-27 +- Governing specification: + `docs/formats/flat-chunk-layout-v1/README.md` +- Layout identity envelope: + `docs/adr/0002-separate-identity-from-physical-storage.md` +- Blob witnesses: + `conformance/golden-file-worldline/v1/identities.tsv` +- Chunk witnesses: `conformance/chunk-id/v1/identities.tsv` +- Storage-profile witness: `conformance/cdc-profile/v1/profile.tsv` +- Independent digest executable: `b3sum` 1.8.5 +- Byte-assembly runtime: Node.js 24.18.0 `Buffer` fixed-width big-endian + operations + +## Construction + +The fixtures were constructed before any production layout encoder or decoder +existed. + +The one-off construction process: + +1. assembled the 144-byte header from literal magic bytes and fixed-width + unsigned big-endian integers; +2. copied existing canonical binary `BlobId` witnesses or independently + calculated the typed ADR-0001 preimage for the deterministic source; +3. copied the accepted `StorageProfileId` digest; +4. copied independently checked version-1 `ChunkId` lengths and digests; +5. assembled each 44-byte entry in logical order; +6. sent the exact typed checksum preimage to the external `b3sum` executable; +7. appended the resulting raw 32-byte checksum; +8. sent the exact ADR-0002 `LayoutId` preimage to `b3sum`; and +9. rendered the final record bytes as lowercase hex for review. + +Node.js assembled bytes but did not supply a BLAKE3 implementation. `b3sum` +was the only digest oracle. No Keep production type, serializer, encoder, +decoder, or chunk detector generated expected layout bytes. + +The `profile-boundary-mismatch` mutation's replacement `ChunkId` digests were +independently calculated from 262,143 and 2 zero bytes through the same typed +version-1 preimage and external `b3sum` oracle. + +## Review controls + +The checked-in tables expose every source recipe, identity, entry coordinate, +record length, checksum, and layout identity needed for an independent +implementation to reproduce the fixtures. + +Issue #10 MUST add an independent checker that: + +- decodes the hex fixtures without using the production layout decoder; +- reconstructs every field at its documented offset; +- recomputes both checksum and `LayoutId` through an independently admitted + BLAKE3 capability; +- executes every text and binary `LayoutId` refusal vector; +- applies every mutation in `mutations.tsv`; and +- cross-checks production encoding only after the independent values pass. + +Regenerating a fixture because production output differs is forbidden. Resolve +the disagreement against the specification and independent digest oracle. diff --git a/conformance/layout/v1/README.md b/conformance/layout/v1/README.md new file mode 100644 index 0000000..ee5e34e --- /dev/null +++ b/conformance/layout/v1/README.md @@ -0,0 +1,99 @@ +# Flat Chunk Layout Corpus v1 + +This directory is the implementation-independent design corpus for +`keep.flat-chunks/v1`. It freezes canonical record bytes, `LayoutId` +coordinates, and mutation expectations without supplying a production encoder +or decoder. + +Issue [#10](https://github.com/flyingrobots/keep/issues/10) owns the +implementation, independent checker, corruption tests, property tests, and +continuous fuzz target. Until that issue lands, these fixtures are protocol +evidence, not evidence that Keep can ingest, decode, admit, or reconstruct a +layout. + +The normative format is the +[Flat Chunk Layout v1 specification](../../../docs/formats/flat-chunk-layout-v1/README.md). + +## Files + +- `layouts.tsv` binds each case to source bytes, `BlobId`, record length, + checksum, `LayoutId`, and exact record fixture. +- `entries.tsv` lists every entry's index, logical offset, exact length, and + version-1 `ChunkId` digest. +- `*.layout.hex` contains one lowercase hexadecimal encoding of the complete + canonical binary record followed by exactly one LF. Hex is fixture transport; + decoded bytes are the durable record. +- `invalid-layout-id-text.tsv` supplies hexadecimal input bytes and exact + refusal classes for malformed or unsupported text coordinates. +- `invalid-layout-id-binary.tsv` supplies exact byte mutations and refusal + classes for malformed or mismatched binary coordinates. +- `mutations.tsv` defines structural and verification mutations for issue #10. +- `ORIGIN.md` records independent construction and review provenance. + +All TSV files are UTF-8, tab-delimited, and terminated by exactly one final LF. +Fields use canonical unsigned decimal unless their name ends in `_hex`. `-` +means inapplicable; it is not an empty field. + +## Golden cases + +The four records exercise distinct laws: + +- `empty` binds the empty `BlobId` and contains zero entries; +- `one-zero` binds one byte to one one-byte `ChunkId`; +- `max-plus-one-zeros` binds 262,145 zero bytes to one hard-maximum chunk and + one final one-byte runt; and +- `zeros-long` binds four repeated hard-maximum chunks, proving that repeated + `ChunkId` values are lawful at distinct logical offsets. + +The chunk digests are existing independent witnesses from +`conformance/chunk-id/v1/identities.tsv`. The registered profile identity is +the accepted ADR-0003 witness. + +## Mutation protocol + +`mutations.tsv` uses these operations: + +- `replace-v1` replaces exactly `span_length` bytes with the same number of + decoded `parameter` hexadecimal bytes; +- `xor-v1` XORs exactly `span_length` bytes with equal-length decoded + `parameter` hexadecimal bytes; +- `insert-v1` inserts decoded `parameter` bytes before `offset`; +- `delete-v1` removes exactly `span_length` bytes and requires parameter `-`; + and +- `swap-v1` swaps the span at `offset` with the equal-length span whose + canonical decimal offset is `parameter`. + +`recompute-v1` recalculates the record checksum after a same-length mutation so +the targeted semantic law, rather than checksum corruption, decides the +result. `preserve-v1` leaves every existing checksum byte unchanged. + +Expected outcomes follow the specification's first-failure order. Inserting a +duplicate flags-width field at offset 24 shifts the canonical header and makes +`header_length` the first invalid field. Swapping the middle entries of +`zeros-long` creates a gap at the first swapped position before the later +offset decrease is observed. + +Mutations of opaque identity digests are not all structural errors. A +rechecksummed `BlobId` digest or `ChunkId` digest can form another structurally +valid layout. Those cases remain admitted until verified content proves +`layout.reconstruction-mismatch` or `layout.chunk-mismatch`. The ledger names +that later phase explicitly. + +The `profile-boundary-mismatch` mutation replaces the natural 262,144-byte +hard-maximum boundary for 262,145 zero bytes with structurally valid +262,143-byte and 2-byte chunks. Both replacement `ChunkId` values and the +target `BlobId` name the exact source bytes. Verification MUST refuse the plan +because replaying `fastcdc-64k-v1` emits the original boundary. + +## `LayoutId` coordinate mutations + +Issue #10 applies every row in `invalid-layout-id-binary.tsv` to the named +case's `layout_id_binary_hex` value. Operations have semantics identical to the +plan mutation protocol. Wider replacements deliberately distinguish an +out-of-bounds plan length, an in-bounds but incongruent length, and a different +valid record length that mismatches the named record. + +`invalid-layout-id-text.tsv` stores raw input as hexadecimal so empty input, +whitespace, and other noncanonical bytes remain unambiguous in a +line-oriented table. The production codecs MUST assert the exact typed +outcomes in both files rather than only generic failure. diff --git a/conformance/layout/v1/empty.layout.hex b/conformance/layout/v1/empty.layout.hex new file mode 100644 index 0000000..8265a64 --- /dev/null +++ b/conformance/layout/v1/empty.layout.hex @@ -0,0 +1 @@ +4b4545503a4c41594f55543a504c414e00010001000000000090002c00000000000000b000000000010100014b4545503a424c4f423a4944000000000001010000000000000000c0074a279c09f9d019dc10e4c821f79f1450cfb8541ab4627132ab9f3c75e33f000101aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb000000000000625119aa215bafe19f952bb3f0114147a8ef5097779113abd97fb2eb8bd0f615 diff --git a/conformance/layout/v1/entries.tsv b/conformance/layout/v1/entries.tsv new file mode 100644 index 0000000..025463d --- /dev/null +++ b/conformance/layout/v1/entries.tsv @@ -0,0 +1,9 @@ +keep.flat-chunk-layout-entries/v1 +case entry_index logical_offset chunk_length chunk_digest_hex +one-zero 0 0 1 9b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff62 +max-plus-one-zeros 0 0 262144 c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b +max-plus-one-zeros 1 262144 1 9b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff62 +zeros-long 0 0 262144 c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b +zeros-long 1 262144 262144 c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b +zeros-long 2 524288 262144 c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b +zeros-long 3 786432 262144 c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b diff --git a/conformance/layout/v1/invalid-layout-id-binary.tsv b/conformance/layout/v1/invalid-layout-id-binary.tsv new file mode 100644 index 0000000..eedab10 --- /dev/null +++ b/conformance/layout/v1/invalid-layout-id-binary.tsv @@ -0,0 +1,11 @@ +keep.flat-chunk-layout-id.invalid-binary/v1 +case base_case operation offset span_length parameter expected_outcome +wrong-identity-magic empty xor-v1 0 1 01 layout-id.wrong-magic +unsupported-identity-version empty replace-v1 16 2 0002 layout-id.unsupported-version +unsupported-layout-codec empty replace-v1 18 2 0002 layout-id.unsupported-codec +plan-length-out-of-bounds empty replace-v1 20 8 0000000002c000b1 layout-id.plan-length-out-of-bounds +plan-length-not-congruent empty replace-v1 20 8 00000000000000b1 layout-id.plan-length-not-congruent +plan-length-mismatch empty replace-v1 20 8 00000000000000dc layout-id.plan-length-mismatch +digest-mismatch empty xor-v1 28 1 01 layout-id.mismatch +truncated-coordinate empty delete-v1 59 1 - layout-id.wrong-length +trailing-byte empty insert-v1 60 0 00 layout-id.wrong-length diff --git a/conformance/layout/v1/invalid-layout-id-text.tsv b/conformance/layout/v1/invalid-layout-id-text.tsv new file mode 100644 index 0000000..0028dde --- /dev/null +++ b/conformance/layout/v1/invalid-layout-id-text.tsv @@ -0,0 +1,21 @@ +keep.flat-chunk-layout-id.invalid-text/v1 +case input_hex expected_outcome +input-too-long 616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161 layout-id.input-too-long +empty-input layout-id.malformed-structure +wrong-scheme 6b6570743a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.wrong-scheme +wrong-kind 6b6565703a626c6f623a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.wrong-kind +malformed-version 6b6565703a6c61796f75743a313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.malformed-version +unsupported-version 6b6565703a6c61796f75743a76323a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.unsupported-version +unsupported-codec 6b6565703a6c61796f75743a76313a747265652d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.unsupported-codec +unsupported-algorithm 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a7368613235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.unsupported-algorithm +leading-zero-plan-length 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a303137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.noncanonical-plan-length +signed-plan-length 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a2b3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.noncanonical-plan-length +plan-length-overflow 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a31383434363734343037333730393535313631363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.plan-length-overflow +plan-length-out-of-bounds 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137353a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.plan-length-out-of-bounds +plan-length-not-congruent 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137373a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.plan-length-not-congruent +short-digest 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a353339353136636335326665373433336533613938346134653261343837373637363637336262363264363230336138386466653163653235336332633066 layout-id.invalid-digest-length +uppercase-digest 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306646 layout-id.noncanonical-digest-case +nonhex-digest 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306667 layout-id.invalid-digest-alphabet +trailing-field 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a353339353136636335326665373433336533613938346134653261343837373637363637336262363264363230336138386466653163653235336332633066383a6578747261 layout-id.trailing-data +leading-space 206b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638 layout-id.wrong-scheme +trailing-newline 6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a353339353136636335326665373433336533613938346134653261343837373637363637336262363264363230336138386466653163653235336332633066380a layout-id.invalid-digest-length diff --git a/conformance/layout/v1/layouts.tsv b/conformance/layout/v1/layouts.tsv new file mode 100644 index 0000000..897b769 --- /dev/null +++ b/conformance/layout/v1/layouts.tsv @@ -0,0 +1,6 @@ +keep.flat-chunk-layouts/v1 +case source_recipe source_parameter source_count logical_length blob_id storage_profile_id entry_count record_length record_checksum_hex layout_id layout_id_binary_hex record_fixture +empty empty-v1 - 0 0 keep:blob:v1:blake3-256:0:c0074a279c09f9d019dc10e4c821f79f1450cfb8541ab4627132ab9f3c75e33f keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb 0 176 625119aa215bafe19f952bb3f0114147a8ef5097779113abd97fb2eb8bd0f615 keep:layout:v1:flat-chunks-v1:blake3-256:176:539516cc52fe7433e3a984a4e2a4877676673bb62d6203a88dfe1ce253c2c0f8 4b4545503a4c41594f55543a494400000001000100000000000000b0539516cc52fe7433e3a984a4e2a4877676673bb62d6203a88dfe1ce253c2c0f8 empty.layout.hex +one-zero repeated-byte-v1 00 1 1 keep:blob:v1:blake3-256:1:1cfb8fa9e917aba15a1f592095f377ff180755fe1212b0d7d2ec750bd128b606 keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb 1 220 093e7ab79a7e454629ab504b1736e4e6b655a64f1ba6780cb97d3e074384db7c keep:layout:v1:flat-chunks-v1:blake3-256:220:887da23f1a7483359a78fc9a7fde80030ec2c4690603803f0ab7d0edb56575b8 4b4545503a4c41594f55543a494400000001000100000000000000dc887da23f1a7483359a78fc9a7fde80030ec2c4690603803f0ab7d0edb56575b8 one-zero.layout.hex +max-plus-one-zeros repeated-byte-v1 00 262145 262145 keep:blob:v1:blake3-256:262145:dbe1c952ccd18c317f9c7a83301da48e251730d5b9279ecfb101cd80da9b6f07 keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb 2 264 9e23e3a56d449e4f19919270602d29f2eb8a614f4a022ea42d1c0243d409eb42 keep:layout:v1:flat-chunks-v1:blake3-256:264:2b281c5e381edfad1dc513d3c2a8818fb4b95d0b188072e27ecc1e18a748afa6 4b4545503a4c41594f55543a494400000001000100000000000001082b281c5e381edfad1dc513d3c2a8818fb4b95d0b188072e27ecc1e18a748afa6 max-plus-one-zeros.layout.hex +zeros-long repeated-byte-v1 00 1048576 1048576 keep:blob:v1:blake3-256:1048576:a5e90495f5a61a299dc276d89da91cd60add2b62e997f48cc14bd821cdb6724a keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb 4 352 c82f881f3af274b35c04c693228cc9078eb88044d60e62aef60dfa42a9310582 keep:layout:v1:flat-chunks-v1:blake3-256:352:ecc098f68f7c6691936bced4b8ec31aa55de34f78224ce3cffb035f0f97c7513 4b4545503a4c41594f55543a49440000000100010000000000000160ecc098f68f7c6691936bced4b8ec31aa55de34f78224ce3cffb035f0f97c7513 zeros-long.layout.hex diff --git a/conformance/layout/v1/max-plus-one-zeros.layout.hex b/conformance/layout/v1/max-plus-one-zeros.layout.hex new file mode 100644 index 0000000..73f0237 --- /dev/null +++ b/conformance/layout/v1/max-plus-one-zeros.layout.hex @@ -0,0 +1 @@ +4b4545503a4c41594f55543a504c414e00010001000000000090002c000000000000010800000002010100014b4545503a424c4f423a4944000000000001010000000000040001dbe1c952ccd18c317f9c7a83301da48e251730d5b9279ecfb101cd80da9b6f07000101aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb000000000000000000000000000000040000c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b0000000000040000000000019b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff629e23e3a56d449e4f19919270602d29f2eb8a614f4a022ea42d1c0243d409eb42 diff --git a/conformance/layout/v1/mutations.tsv b/conformance/layout/v1/mutations.tsv new file mode 100644 index 0000000..0f6d3ca --- /dev/null +++ b/conformance/layout/v1/mutations.tsv @@ -0,0 +1,39 @@ +keep.flat-chunk-layout-mutations/v1 +case base_case operation offset span_length parameter checksum_posture decision_phase expected_outcome requirement +wrong-record-magic empty xor-v1 0 1 01 recompute-v1 decoding layout.wrong-magic KEEP-LAYOUT-001 +unsupported-format-version empty replace-v1 16 2 0002 recompute-v1 decoding layout.unsupported-format-version KEEP-LAYOUT-001 +unsupported-layout-codec empty replace-v1 18 2 0002 recompute-v1 decoding layout.unsupported-codec KEEP-LAYOUT-001 +unknown-mandatory-flag empty replace-v1 20 4 00000001 recompute-v1 decoding layout.unknown-flags KEEP-LAYOUT-010 +wrong-header-length empty replace-v1 24 2 0091 recompute-v1 decoding layout.wrong-header-length KEEP-LAYOUT-001 +wrong-entry-length empty replace-v1 26 2 002d recompute-v1 decoding layout.wrong-entry-length KEEP-LAYOUT-001 +wrong-record-length one-zero replace-v1 28 8 00000000000000db recompute-v1 decoding layout.record-length-mismatch KEEP-LAYOUT-011 +entry-count-length-mismatch one-zero replace-v1 36 4 00000002 recompute-v1 decoding layout.entry-count-length-mismatch KEEP-LAYOUT-011 +unsupported-checksum-algorithm empty replace-v1 40 1 02 recompute-v1 decoding layout.unsupported-checksum-algorithm KEEP-LAYOUT-012 +unsupported-chunk-hash-algorithm one-zero replace-v1 41 1 02 recompute-v1 decoding layout.unsupported-chunk-hash-algorithm KEEP-LAYOUT-005 +unsupported-chunk-identity-version one-zero replace-v1 42 2 0002 recompute-v1 decoding layout.unsupported-chunk-identity-version KEEP-LAYOUT-005 +wrong-blob-id-magic one-zero xor-v1 44 1 01 recompute-v1 validation layout.blob-id-wrong-magic KEEP-LAYOUT-003 +unsupported-blob-id-version one-zero replace-v1 60 2 0002 recompute-v1 validation layout.blob-id-unsupported-version KEEP-LAYOUT-003 +unsupported-blob-id-algorithm one-zero replace-v1 62 1 02 recompute-v1 validation layout.blob-id-unsupported-algorithm KEEP-LAYOUT-003 +blob-logical-length-mismatch one-zero replace-v1 63 8 0000000000000000 recompute-v1 validation layout.empty-blob-has-entries KEEP-LAYOUT-007 +blob-digest-substitution one-zero xor-v1 71 1 01 recompute-v1 verification layout.reconstruction-mismatch KEEP-LAYOUT-014 +unsupported-profile-identity-version one-zero replace-v1 103 2 0002 recompute-v1 admission layout.unsupported-storage-profile-version KEEP-LAYOUT-004 +unsupported-profile-hash-algorithm one-zero replace-v1 105 1 02 recompute-v1 admission layout.unsupported-storage-profile-algorithm KEEP-LAYOUT-004 +unregistered-profile-digest one-zero xor-v1 106 1 01 recompute-v1 admission layout.unsupported-storage-profile KEEP-LAYOUT-004 +nonzero-reserved-byte empty replace-v1 138 1 01 recompute-v1 decoding layout.nonzero-reserved KEEP-LAYOUT-010 +first-offset-not-zero one-zero replace-v1 144 8 0000000000000001 recompute-v1 validation layout.first-offset-not-zero KEEP-LAYOUT-006 +zero-chunk-length one-zero replace-v1 152 4 00000000 recompute-v1 validation layout.zero-chunk-length KEEP-LAYOUT-005 +final-chunk-over-profile-maximum one-zero replace-v1 152 4 00040001 recompute-v1 validation layout.profile-length-out-of-bounds KEEP-LAYOUT-005 +chunk-digest-substitution one-zero xor-v1 156 1 01 recompute-v1 verification layout.chunk-mismatch KEEP-LAYOUT-014 +profile-boundary-mismatch max-plus-one-zeros replace-v1 144 88 00000000000000000003ffff8f47b6632dfd6f015c4aede5d2b0fa50cbe5aad2b341b12c4b02074f4555d9cd000000000003ffff00000002e5c15b25f3fa2cd56fdccd7725224d897100420231e94e33ef1771c644d73be7 recompute-v1 verification layout.profile-boundary-mismatch KEEP-LAYOUT-016 +record-checksum-bit-flip one-zero xor-v1 188 1 01 preserve-v1 validation layout.checksum-mismatch KEEP-LAYOUT-012 +truncated-record one-zero delete-v1 219 1 - preserve-v1 decoding layout.truncated-record KEEP-LAYOUT-011 +trailing-byte one-zero insert-v1 220 0 00 preserve-v1 decoding layout.trailing-data KEEP-LAYOUT-011 +inserted-duplicate-flags-field one-zero insert-v1 24 0 00000000 preserve-v1 decoding layout.wrong-header-length KEEP-LAYOUT-011 +entry-order-swap zeros-long swap-v1 188 44 232 recompute-v1 validation layout.gap KEEP-LAYOUT-006 +entry-gap max-plus-one-zeros replace-v1 188 8 0000000000040001 recompute-v1 validation layout.gap KEEP-LAYOUT-006 +entry-overlap max-plus-one-zeros replace-v1 188 8 000000000003ffff recompute-v1 validation layout.overlap KEEP-LAYOUT-006 +final-aggregate-mismatch max-plus-one-zeros replace-v1 196 4 00000002 recompute-v1 validation layout.aggregate-length-mismatch KEEP-LAYOUT-007 +empty-blob-with-entry one-zero replace-v1 44 59 4b4545503a424c4f423a4944000000000001010000000000000000c0074a279c09f9d019dc10e4c821f79f1450cfb8541ab4627132ab9f3c75e33f recompute-v1 validation layout.empty-blob-has-entries KEEP-LAYOUT-008 +nonempty-blob-without-entry empty replace-v1 44 59 4b4545503a424c4f423a49440000000000010100000000000000011cfb8fa9e917aba15a1f592095f377ff180755fe1212b0d7d2ec750bd128b606 recompute-v1 validation layout.nonempty-blob-has-no-entries KEEP-LAYOUT-008 +entry-count-limit-exceeded empty replace-v1 36 4 00100001 recompute-v1 decoding layout.entry-count-limit-exceeded KEEP-LAYOUT-009 +record-length-limit-exceeded empty replace-v1 28 8 0000000002c000b1 recompute-v1 decoding layout.record-length-limit-exceeded KEEP-LAYOUT-009 diff --git a/conformance/layout/v1/one-zero.layout.hex b/conformance/layout/v1/one-zero.layout.hex new file mode 100644 index 0000000..ac73748 --- /dev/null +++ b/conformance/layout/v1/one-zero.layout.hex @@ -0,0 +1 @@ +4b4545503a4c41594f55543a504c414e00010001000000000090002c00000000000000dc00000001010100014b4545503a424c4f423a49440000000000010100000000000000011cfb8fa9e917aba15a1f592095f377ff180755fe1212b0d7d2ec750bd128b606000101aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb0000000000000000000000000000000000019b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff62093e7ab79a7e454629ab504b1736e4e6b655a64f1ba6780cb97d3e074384db7c diff --git a/conformance/layout/v1/zeros-long.layout.hex b/conformance/layout/v1/zeros-long.layout.hex new file mode 100644 index 0000000..277f145 --- /dev/null +++ b/conformance/layout/v1/zeros-long.layout.hex @@ -0,0 +1 @@ +4b4545503a4c41594f55543a504c414e00010001000000000090002c000000000000016000000004010100014b4545503a424c4f423a4944000000000001010000000000100000a5e90495f5a61a299dc276d89da91cd60add2b62e997f48cc14bd821cdb6724a000101aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb000000000000000000000000000000040000c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b000000000004000000040000c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b000000000008000000040000c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b00000000000c000000040000c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87bc82f881f3af274b35c04c693228cc9078eb88044d60e62aef60dfa42a9310582 diff --git a/docs/formats/README.md b/docs/formats/README.md new file mode 100644 index 0000000..aecdfaf --- /dev/null +++ b/docs/formats/README.md @@ -0,0 +1,14 @@ +# Durable Formats + +Keep treats every durable format as a versioned protocol. A format is not +admitted merely because one Rust type can serialize and deserialize it. + +## Format registry + +| Format | Coordinate | Status | Evidence | +| --- | --- | --- | --- | +| [Flat Chunk Layout v1](flat-chunk-layout-v1/README.md) | `keep.flat-chunks/v1` | Specified; implementation belongs to issue #10 | [Golden corpus](../../conformance/layout/v1/README.md) | + +The registry records protocol specifications, including formats whose +implementation is still planned. Each format page states its exact proof +boundary and nonclaims. diff --git a/docs/formats/flat-chunk-layout-v1/README.md b/docs/formats/flat-chunk-layout-v1/README.md new file mode 100644 index 0000000..d8c3309 --- /dev/null +++ b/docs/formats/flat-chunk-layout-v1/README.md @@ -0,0 +1,427 @@ +# Flat Chunk Layout Version 1 + +- Status: Frozen specification; no production encoder or decoder exists +- Format coordinate: `keep.flat-chunks/v1` +- Format version: `1` +- Layout codec: `1` +- Related issue: [#9](https://github.com/flyingrobots/keep/issues/9) +- Implementation issue: [#10](https://github.com/flyingrobots/keep/issues/10) +- Depends on: + [ADR-0001](../../adr/0001-exact-logical-byte-identity.md), + [ADR-0002](../../adr/0002-separate-identity-from-physical-storage.md), + and [ADR-0003](../../adr/0003-deterministic-content-defined-chunking-profiles.md) + +This page specifies the canonical durable plan that maps one logical +`BlobId` to an ordered, bounded sequence of exact `ChunkId` values. It is a +language-independent binary protocol. + +Keep does not yet expose a layout API, encoder, decoder, ingestion path, or +stored reconstruction path. The checked-in fixtures prove exact design bytes; +they do not prove production behavior. Issue #10 owns that implementation and +its parser, corruption, property, and fuzz evidence. + +## Core law + +A valid version-1 flat layout commits to: + +- one target `BlobId`, including its exact logical length; +- one registered `StorageProfileId`; +- one ordered sequence of version-1 `ChunkId` values; +- one explicit logical offset and exact positive length per entry; and +- no physical segment, path, object key, catalog generation, or retention + fact. + +Concatenating the exact verified bytes named by the entries in order MUST +produce the exact bytes named by the target `BlobId`, or reconstruction MUST +refuse. Structural layout validation cannot prove that content claim without +the chunk bytes; verified reconstruction performs that final comparison. +Verified reconstruction MUST also reproduce the declared spans by replaying +the bound storage profile over those bytes, or it MUST refuse the profile +claim. + +Repeated `ChunkId` values are valid when equal bytes occur more than once. +Entry order and logical offsets belong to the layout. They do not become part +of `ChunkId`. + +## Canonical `LayoutId` + +Version 1 assigns layout codec `1` to `keep.flat-chunks/v1`. For exact +canonical plan bytes `P`, including the 32-byte record checksum: + +```text +layout_digest_v1(P) = BLAKE3-256( + ASCII("KEEP:LAYOUT:ID\0\0") + || u16be(1) + || u16be(1) + || P + || u64be(length(P)) +) +``` + +The first integer is the layout identity-envelope version. The second is the +layout codec. This is the envelope accepted by ADR-0002. The plan length is +checked before hashing and cannot exceed `MAX_LAYOUT_RECORD_LENGTH`. + +Identity-envelope version 1 fixes BLAKE3-256. The `blake3-256` text token is +an exact coordinate component, not algorithm negotiation. A different layout +identity hash requires a new identity-envelope version. + +A validated `LayoutId` contains: + +- identity-envelope version `1`; +- layout codec `1`; +- exact plan length; +- the 32-byte `layout_digest_v1` result. + +### Text coordinate + +The only canonical text form is: + +```text +keep:layout:v1:flat-chunks-v1:blake3-256:: +``` + +The fixed tokens are exact lowercase ASCII. `plan_length` is canonical +unsigned decimal with no sign, whitespace, separators, or leading zeroes +except the value `0` itself. Codec 1 accepts only lengths in the inclusive +range 176 through 46,137,520 for which +`(plan_length - 176) % 44 == 0`. The digest is exactly 64 lowercase +hexadecimal characters. A canonical coordinate is at most 114 ASCII bytes. +The parser accepts at most 128 input bytes so it can distinguish malformed +fields, including an overflowing decimal length, and refuses a longer input +before token parsing. Unknown versions, codecs, algorithms, impossible or +overlong lengths, uppercase hex, and trailing data are refused. The checked-in +[invalid-text corpus](../../../conformance/layout/v1/invalid-layout-id-text.tsv) +fixes exact refusal classes for every text field and canonicality edge. + +### Binary coordinate + +The canonical binary `LayoutId` coordinate is exactly 60 bytes: + +| Offset | Size | Field | Version-1 value or encoding | +| ---: | ---: | --- | --- | +| 0 | 16 | `identity_magic` | ASCII `KEEP:LAYOUT:ID` followed by two zero bytes | +| 16 | 2 | `identity_version` | unsigned big-endian `1` | +| 18 | 2 | `layout_codec` | unsigned big-endian `1` | +| 20 | 8 | `plan_length` | unsigned big-endian canonical record length | +| 28 | 32 | `digest` | raw BLAKE3-256 output | + +Parsing either coordinate proves only canonical supported shape. It does not +prove possession, checksum validity, structural validity, or reconstruction +of the named plan. Binary parsing applies the same codec-1 plan-length range +and congruence law as text parsing. The checked-in +[invalid-binary corpus](../../../conformance/layout/v1/invalid-layout-id-binary.tsv) +provides exact byte mutations and refusal classes. + +## Canonical plan record + +The canonical plan `P` is: + +```text +fixed_header || entries || record_checksum +``` + +All integers are unsigned big-endian fixed-width values. There are no +variable-width integers, serializer defaults, map keys, tags, or optional +field encodings. The record is positional, so duplicate fields are +unrepresentable. Inserting a second encoding of any field changes the byte +length and is refused as noncanonical framing. + +The exact record length is: + +```text +record_length = 144 + entry_count * 44 + 32 +``` + +Every operation in this expression uses checked arithmetic before allocation +or cursor movement. + +### Fixed header + +The header is exactly 144 bytes: + + + +| Offset | Size | Field | Version-1 value or rule | +| ---: | ---: | --- | --- | +| 0 | 16 | `magic` | ASCII `KEEP:LAYOUT:PLAN` | +| 16 | 2 | `format_version` | unsigned big-endian `1` | +| 18 | 2 | `layout_codec` | unsigned big-endian `1` | +| 20 | 4 | `flags` | zero; every nonzero bit is mandatory-to-understand and unsupported | +| 24 | 2 | `header_length` | unsigned big-endian `144` | +| 26 | 2 | `entry_length` | unsigned big-endian `44` | +| 28 | 8 | `record_length` | exact total plan length, including checksum | +| 36 | 4 | `entry_count` | number of 44-byte entries | +| 40 | 1 | `record_checksum_algorithm` | `1`, meaning BLAKE3-256 | +| 41 | 1 | `chunk_hash_algorithm` | `1`, meaning BLAKE3-256 | +| 42 | 2 | `chunk_identity_version` | unsigned big-endian `1` | +| 44 | 59 | `target_blob_id` | exact canonical ADR-0001 binary `BlobId` | +| 103 | 2 | `storage_profile_identity_version` | unsigned big-endian `1` | +| 105 | 1 | `storage_profile_hash_algorithm` | `1`, meaning BLAKE3-256 | +| 106 | 32 | `storage_profile_digest` | raw `StorageProfileId` digest | +| 138 | 6 | `reserved` | all zero | + + + +The profile coordinate is the typed +`keep:storage-profile:v1:blake3-256:` identity specified by ADR-0003. +Version-1 admission recognizes only registered profile identities. The +profile record itself is not duplicated inside each layout. + +The header binds every entry to version-1 `ChunkId` and BLAKE3-256 once. +Each entry therefore carries the exact positive `ChunkLength` and digest +that constitute a version-1 `ChunkId` without repeating a 16-byte magic or +algorithm coordinate per chunk. + +### Entry + +Each entry is exactly 44 bytes: + +| Relative offset | Size | Field | Encoding | +| ---: | ---: | --- | --- | +| 0 | 8 | `logical_offset` | unsigned big-endian absolute blob offset | +| 8 | 4 | `chunk_length` | unsigned big-endian positive length | +| 12 | 32 | `chunk_digest` | raw version-1 `ChunkId` digest | + +Entry `i` begins at: + +```text +144 + i * 44 +``` + +That offset is computed with checked arithmetic. No entry contains a physical +location. A future catalog resolves a logical identity to physical evidence +under its own generation and verification rules. + +### Record checksum + +Let `R` be the exact header and entries, excluding the checksum. The final +32 bytes are: + +```text +layout_record_checksum_v1(R) = BLAKE3-256( + ASCII("KEEP:LAYOUT:SUM\0") + || u16be(1) + || u8(1) + || R + || u64be(length(R)) +) +``` + +The checksum detects accidental corruption when no expected `LayoutId` is +available. It is not a MAC, signature, authority statement, retention proof, +or substitute for comparing the complete `LayoutId`. When an expected +`LayoutId` is available, Keep verifies both the record checksum and the +identity. + +## Bounds + +Version 1 defines: + +| Bound | Value | +| --- | ---: | +| `LAYOUT_HEADER_LENGTH` | 144 bytes | +| `LAYOUT_ENTRY_LENGTH` | 44 bytes | +| `LAYOUT_CHECKSUM_LENGTH` | 32 bytes | +| `MAX_CANONICAL_LAYOUT_ID_TEXT_LENGTH` | 114 bytes | +| `MAX_LAYOUT_ID_TEXT_INPUT_LENGTH` | 128 bytes | +| `LAYOUT_ID_BINARY_LENGTH` | 60 bytes | +| `MAX_LAYOUT_DEPTH` | 1 record | +| `MAX_LAYOUT_ENTRY_COUNT` | 1,048,576 (`2^20`) | +| `MAX_LAYOUT_RECORD_LENGTH` | 46,137,520 bytes | + +`MAX_LAYOUT_RECORD_LENGTH` is exactly: + +```text +144 + 1,048,576 * 44 + 32 +``` + +With the registered profile's 262,144-byte hard chunk maximum, a flat +version-1 plan can describe at most 256 GiB of logical bytes. Larger blobs +require a future hierarchical codec with explicit bounds; they MUST NOT bypass +the entry limit or reinterpret codec `1`. + +Depth is exactly one record. Entries can name only chunks; they cannot name +another layout, record, collection, or indirect node. A decoder therefore +performs no recursion and rejects every attempt to introduce child-layout +bytes as noncanonical entry or trailing data. + +A decoder MUST: + +1. read at most the fixed 144-byte header before trusting `entry_count`; +2. reject `entry_count > MAX_LAYOUT_ENTRY_COUNT`; +3. calculate entry bytes and expected record length with checked arithmetic; +4. reject declared or calculated lengths above `MAX_LAYOUT_RECORD_LENGTH`; +5. require the declared length, calculated length, and actual input length to + be equal before allocating an entry collection; and +6. stream entries and checksum calculation when materialization is + unnecessary. + +The wire bound does not authorize one allocation of the maximum size. Public +APIs MUST document their configured admission cap and allocation behavior. +An implementation may apply a lower configured cap, but that policy cannot +change the canonical bytes or identity of a plan it does admit. + +## Structural laws + +Validation walks entries once in encoded order with checked `u64` arithmetic. + +For an empty target blob: + +- `entry_count` MUST be zero; and +- the record MUST contain only its header and checksum. + +For a nonempty target blob: + +- `entry_count` MUST be positive; +- the first `logical_offset` MUST be zero; +- every `chunk_length` MUST be positive; +- every entry offset MUST equal the checked exclusive end of its predecessor; +- a lower offset is an overlap; +- a higher offset is a gap; +- the sequence MUST be strictly ordered by offset; +- each nonfinal length MUST be between the admitted profile's minimum and + maximum, inclusive; +- the final length MUST be between one and the profile maximum, inclusive; + and +- the checked final exclusive end MUST equal the logical length embedded in + `target_blob_id`. + +The structural checks prove contiguity and aggregate length. They do not prove +that chunk digests match bytes, that content-derived boundaries are natural +for the profile, or that concatenated bytes match the target `BlobId`. +Verified reconstruction proves all three content-dependent claims. + +## Parse, validate, admit, verify + +Implementations MUST keep these states distinct: + +1. **Decoded fields** are bounded raw integers and byte arrays. They carry no + domain trust and cannot construct public validated types. +2. **Validated layout** has exact framing, supported version and codec, + canonical fixed widths, zero flags and reserved bytes, a valid checksum, + canonical nested identity encodings, checked entry arithmetic, contiguous + spans, and exact aggregate length. +3. **Admitted layout** is validated and uses locally supported + `BlobId`, `ChunkId`, and registered `StorageProfileId` coordinates within + the configured resource cap. +4. **Verified reconstruction** has loaded every named chunk through a + verification boundary, compared each observed `ChunkId`, replayed the + registered profile's boundary detector over the exact reconstructed byte + stream, compared its emitted spans with the declared entries, and compared + the complete observed `BlobId` with the target. + +Parsing, structural validation, or admission alone MUST NOT be reported as +content verification. + +## Deterministic refusal order + +A version-1 decoder validates in this order: + +1. minimum header availability; +2. magic, format version, codec, flags, fixed lengths, algorithms, and + reserved bytes; +3. entry-count and record-length bounds using checked arithmetic; +4. exact declared, calculated, and actual length equality; +5. record checksum; +6. canonical nested identity coordinates and registered profile admission; +7. empty/nonempty cardinality; +8. positive entry lengths, profile bounds, encoded-order offset continuity, + and checked aggregate length; and +9. expected `LayoutId`, when supplied. + +The first failed law determines the typed boundary error. Implementations MUST +not allocate from an unbounded count, continue after ambiguous framing, or +silently canonicalize malformed bytes. Entry validation processes entries in +encoded order and stops at the first failed entry law; a later offset cannot +replace an earlier gap or overlap with another error. + +## Requirement ledger + + + +| ID | Requirement | Design evidence | Implementation status | +| --- | --- | --- | --- | +| `KEEP-LAYOUT-001` | Exact magic, version, codec, and big-endian fixed-width grammar | Header table and golden records | Planned in #10 | +| `KEEP-LAYOUT-002` | `LayoutId` uses the ADR-0002 domain and exact plan length | Identity grammar, fixture coordinates, and text and binary refusal tables | Planned in #10 | +| `KEEP-LAYOUT-003` | Target `BlobId` and logical length are inseparable | Embedded canonical 59-byte coordinate | Planned in #10 | +| `KEEP-LAYOUT-004` | One registered `StorageProfileId` governs all entries | Header profile coordinate | Planned in #10 | +| `KEEP-LAYOUT-005` | Entry `ChunkId` kind, version, algorithm, length, and digest are typed | Header and entry grammar | Planned in #10 | +| `KEEP-LAYOUT-006` | Offsets are strictly ordered, contiguous, and gap/overlap free | Structural laws and mutation ledger | Planned in #10 | +| `KEEP-LAYOUT-007` | Entry and aggregate arithmetic is checked | Bounds and structural laws | Planned in #10 | +| `KEEP-LAYOUT-008` | Empty and nonempty layouts have exact cardinality | Empty and one-chunk golden records | Planned in #10 | +| `KEEP-LAYOUT-009` | Depth, counts, and record allocation are bounded before allocation | Depth 1, `2^20` entries, and 46,137,520-byte limit | Planned in #10 | +| `KEEP-LAYOUT-010` | Nonzero flags, reserved bytes, and unknown mandatory coordinates are refused | Mutation ledger | Planned in #10 | +| `KEEP-LAYOUT-011` | Trailing, truncated, duplicated, and noncanonical framing is refused | Mutation ledger | Planned in #10 | +| `KEEP-LAYOUT-012` | Record checksum is typed, domain-separated, and checked | Checksum grammar and golden records | Planned in #10 | +| `KEEP-LAYOUT-013` | Physical locations never participate | Header and entry grammar | Specified | +| `KEEP-LAYOUT-014` | Decoded, validated, admitted, and verified states remain distinct | State model | Planned in #10 | +| `KEEP-LAYOUT-015` | Flat v1 never silently becomes hierarchical | Compatibility section and rationale | Specified | +| `KEEP-LAYOUT-016` | Verified reconstruction reproduces the declared spans under the bound storage profile | Verification state and profile-boundary mutation | Planned in #10 | + + + +## Compatibility and migration + +The canonical record, checksum preimage, `LayoutId` preimage, coordinate +forms, bounds, and refusal rules are compatibility commitments. + +Changing any of these requires a new layout codec: + +- field order, width, or endianness; +- checksum domain or algorithm; +- chunk identity version or algorithm; +- offset or length semantics; +- entry-count or record-length protocol limits; +- implicit versus explicit offsets; +- profile binding; +- hierarchy or indirection; or +- canonical text or binary identity grammar. + +A new codec may coexist beside codec `1`. Migration writes and verifies a new +layout with a new `LayoutId`; it never rewrites or reinterprets an existing +codec-1 identity. `BlobId` remains stable when reconstructed logical bytes are +unchanged. + +Hierarchical layouts are explicitly deferred. Codec `1` contains no child +layout references, recursion, depth field, or extension point. Unknown codecs +and nonzero flags are refused rather than guessed. + +## Security and privacy + +Lengths, chunk boundaries, repeated chunk identities, and the selected storage +profile are visible metadata. Version 1 provides no confidentiality. + +BLAKE3 checksums and identities detect accidental or adversarial byte changes +when compared with independently obtained expected values. They do not +authenticate a writer. A malicious source that controls a complete record can +recompute its checksum and `LayoutId`; admission and application authority are +separate concerns. + +The fixed header and hard count and length limits prevent attacker-controlled +allocation from preceding validation. Every offset, multiplication, addition, +and host-size conversion is checked. A decoder refuses ambiguous or +unsupported state and never returns partially authenticated bytes as verified. + +## Golden and mutation evidence + +The implementation-independent +[layout corpus](../../../conformance/layout/v1/README.md) contains: + +- an empty plan; +- a one-byte, one-chunk plan; +- a two-chunk plan that exercises the registered profile's hard maximum and + final runt; +- a four-chunk plan that proves repeated `ChunkId` values remain lawful at + distinct offsets; +- exact canonical record hex, checksums, `BlobId`, and `LayoutId` coordinates; + and +- text and binary `LayoutId` refusal tables plus a plan mutation ledger + covering every fixed header field, entry field, checksum, truncation, + trailing bytes, duplicate-field insertion, ordering, gap, overlap, + aggregate mismatch, allocation limit, and later content or storage-profile + verification failure. + +The reasons for the flat grammar, explicit offsets, checksum, bounds, and +hierarchy posture are recorded in the +[colocated rationale](rationale.md). diff --git a/docs/formats/flat-chunk-layout-v1/rationale.md b/docs/formats/flat-chunk-layout-v1/rationale.md new file mode 100644 index 0000000..a70f31e --- /dev/null +++ b/docs/formats/flat-chunk-layout-v1/rationale.md @@ -0,0 +1,160 @@ +# Flat Chunk Layout Version 1 Rationale + +## Decision + +Keep assigns layout codec `1`, named `keep.flat-chunks/v1`, to a fixed-width +big-endian flat sequence of explicitly offset version-1 `ChunkId` values. +The record embeds the complete canonical binary target `BlobId`, binds one +registered `StorageProfileId`, carries a domain-separated checksum, and +limits the entry count to `2^20`. + +This decision governs the canonical record, `LayoutId`, validation stages, +resource bounds, migration posture, and first durable layout compatibility +surface. It does not implement an encoder, decoder, ingestion path, storage +backend, or verified reader. + +## Why a flat first codec + +A flat ordered plan has one bounded traversal, one offset domain, and no +recursive trust transition. That makes gaps, overlaps, aggregate mismatch, +allocation, and range planning directly auditable. + +The `2^20` entry limit bounds the largest record at 46,137,520 bytes and the +registered profile's largest describable blob at 256 GiB. That is a deliberate +version-1 protocol limit, not a claim that every caller should materialize a +46 MiB plan. + +Issue #12 will measure real workloads. A hierarchical codec requires its own +depth, fanout, cycle, aggregate, and allocation laws and therefore receives a +new codec coordinate rather than an inactive flag in v1. + +## Why explicit offsets + +Offsets are derivable from preceding lengths, but encoding them creates a +cross-field invariant that detects gaps, overlaps, reordering, and incorrect +range planning before chunk bytes are loaded. + +The redundancy is canonical: exactly one offset is lawful at each entry. +Physical locations remain absent. The offset is a logical coordinate within +the target blob. + +## Why fixed-width binary + +Fixed-width big-endian integers have one encoding, make bounds inspectable +before allocation, and require no serializer configuration. A positional +record cannot contain duplicate map keys or alternate field order. + +The format still rejects inserted duplicate field bytes, trailing bytes, and +changed fixed lengths. A serializer-owned Rust struct, arbitrary Serde output, +JSON, CBOR, and host-native layout are not durable protocol. + +## Why bind identities by kind + +The full canonical binary `BlobId` appears in the header because it already +has an admitted boundary encoding and includes the logical length. + +The profile coordinate uses its identity version, hash algorithm, and digest +instead of duplicating the 96-byte profile record in every layout. Admission +resolves the identity through the registered profile set. + +The header binds every entry to version-1 BLAKE3-256 `ChunkId`. An entry then +stores the exact length and digest that form that typed value. Repeating the +chunk magic, version, and algorithm in every entry would add bytes without +adding a new validation boundary. + +Verified reconstruction replays the registered profile's deterministic +boundary detector over the reconstructed stream and compares its emitted +spans with the entries. Chunk and blob digest agreement alone cannot prove +that the declared profile produced the plan. + +Any future identity kind, version, or algorithm requires a new layout codec. +No version-1 field is reinterpreted. + +## Why include a record checksum + +`LayoutId` verifies a plan when an independently obtained expected identity is +available. Recovery and inspection may encounter an isolated candidate record +before a catalog supplies that expectation. + +The record checksum provides a typed corruption check in that state. Its +domain differs from `LayoutId`, and it covers the header and entries but not +itself. It is not authentication and cannot make an untrusted record +authoritative. + +## Why the checksum is inside `LayoutId` + +The exact canonical plan includes its checksum. This prevents two records +with different checksum bytes from sharing one `LayoutId`, keeps the durable +record self-contained, and makes checksum-algorithm evolution an explicit +codec change. + +## Alternatives rejected + +### Encode only ordered lengths and digests + +Rejected because gaps, overlaps, and entry reordering would be +unrepresentable rather than independently checkable. Explicit offsets create +the required continuity law and support later exact range planning. + +### Embed physical segment coordinates + +Rejected because compaction, copying, tier movement, and catalog recovery move +locations without moving logical content or its reconstruction plan. +ADR-0002 assigns physical coordinates to generation-checked catalogs. + +### Embed the complete storage-profile record + +Rejected because the immutable registered profile already has a typed +identity. Repeating the record increases every layout and creates two copies +whose disagreement would need another precedence rule. + +### Use CBOR, JSON, or arbitrary Serde output + +Rejected because serializer defaults, map order, duplicate fields, integer +width choices, and library upgrades create a larger canonicality surface. +Keep reserves JSON and CBOR for boundaries with named canonical profiles; this +format needs neither. + +### Use variable-width integers + +Rejected because fixed-width fields are small relative to chunk digests and +have a one-byte representation. Varints add overlong encodings and parsing +branches without meaningful layout-size savings. + +### Omit the checksum and rely only on `LayoutId` + +Rejected because recovery may need to classify an isolated record before an +expected identity is available. The checksum supplies corruption evidence +without claiming catalog membership or writer authority. + +### Make version 1 hierarchical + +Rejected because recursive layouts require explicit depth, fanout, cycle, +aggregate, partial-read, and allocation laws. Adding unused child fields or +flags would create dormant ambiguity. A future hierarchy receives a new codec +and fixtures. + +### Remove the entry-count limit and stream indefinitely + +Rejected because an unbounded durable record is an unbounded validation and +indexing obligation even when decoding streams. The hard limit makes refusal +portable and reviewable; configured policies may be lower. + +### Use one chunk for the empty blob + +Rejected because `ChunkId` names only nonempty bytes and FastCDC emits zero +chunks for empty input. The unique empty layout therefore contains zero +entries and the empty `BlobId`. + +## Consequences + +- A layout is independently reproducible from exact typed inputs. +- Rechunking can move `LayoutId` without moving `BlobId`. +- Range planning can validate logical continuity before storage lookup. +- Verified reconstruction refuses a content-correct plan that falsely claims + a storage profile whose deterministic boundaries it does not reproduce. +- Large plans remain bounded but may still warrant streaming APIs. +- Blobs above the flat codec's capacity are precisely unsupported rather than + partially represented. +- Format evolution creates a new codec and migration path instead of changing + version-1 meaning. diff --git a/xtask/tests/layout_format_contract.rs b/xtask/tests/layout_format_contract.rs new file mode 100644 index 0000000..4b3614f --- /dev/null +++ b/xtask/tests/layout_format_contract.rs @@ -0,0 +1,123 @@ +//! Written-contract regression evidence for the flat chunk layout format. + +const SPECIFICATION: &str = include_str!("../../docs/formats/flat-chunk-layout-v1/README.md"); +const RATIONALE: &str = include_str!("../../docs/formats/flat-chunk-layout-v1/rationale.md"); +const CONFORMANCE_GUIDE: &str = include_str!("../../conformance/layout/v1/README.md"); +const INVALID_LAYOUT_ID_BINARY: &str = + include_str!("../../conformance/layout/v1/invalid-layout-id-binary.tsv"); +const INVALID_LAYOUT_ID_TEXT: &str = + include_str!("../../conformance/layout/v1/invalid-layout-id-text.tsv"); +const MUTATIONS: &str = include_str!("../../conformance/layout/v1/mutations.tsv"); + +#[test] +fn verified_reconstruction_proves_the_bound_storage_profile() { + assert!(SPECIFICATION.contains("replayed the")); + assert!(SPECIFICATION.contains("registered profile's boundary detector")); + assert!( + MUTATIONS + .lines() + .any(|line| line.starts_with("profile-boundary-mismatch\t")) + ); +} + +#[test] +fn mutation_outcomes_follow_first_failure_precedence() { + assert_eq!( + expected_mutation_outcome("inserted-duplicate-flags-field"), + Some("layout.wrong-header-length") + ); + assert_eq!( + expected_mutation_outcome("entry-order-swap"), + Some("layout.gap") + ); + assert_eq!( + expected_mutation_outcome("blob-logical-length-mismatch"), + Some("layout.empty-blob-has-entries") + ); +} + +#[test] +fn format_prose_uses_unambiguous_compound_phrases() { + assert!(!CONFORMANCE_GUIDE.contains("same exact")); + assert!(!RATIONALE.contains("one byte representation")); + assert!(!SPECIFICATION.contains("future bounded hierarchical codec")); +} + +#[test] +fn layout_id_coordinates_have_field_complete_refusal_vectors() { + assert!(INVALID_LAYOUT_ID_BINARY.starts_with( + "keep.flat-chunk-layout-id.invalid-binary/v1\n\ + case\tbase_case\toperation\toffset\tspan_length\tparameter\t\ + expected_outcome\n" + )); + assert!(INVALID_LAYOUT_ID_TEXT.starts_with( + "keep.flat-chunk-layout-id.invalid-text/v1\n\ + case\tinput_hex\texpected_outcome\n" + )); + for case_name in [ + "wrong-identity-magic", + "unsupported-identity-version", + "unsupported-layout-codec", + "plan-length-out-of-bounds", + "plan-length-not-congruent", + "plan-length-mismatch", + "digest-mismatch", + "truncated-coordinate", + "trailing-byte", + ] { + assert!(contains_case(INVALID_LAYOUT_ID_BINARY, case_name)); + } + for case_name in [ + "input-too-long", + "empty-input", + "wrong-scheme", + "wrong-kind", + "malformed-version", + "unsupported-version", + "unsupported-codec", + "unsupported-algorithm", + "leading-zero-plan-length", + "signed-plan-length", + "plan-length-overflow", + "plan-length-out-of-bounds", + "plan-length-not-congruent", + "short-digest", + "uppercase-digest", + "nonhex-digest", + "trailing-field", + "leading-space", + "trailing-newline", + ] { + assert!(contains_case(INVALID_LAYOUT_ID_TEXT, case_name)); + } + assert!( + INVALID_LAYOUT_ID_BINARY + .lines() + .skip(2) + .all(|line| line.split('\t').count() == 7) + ); + assert!( + INVALID_LAYOUT_ID_TEXT + .lines() + .skip(2) + .all(|line| line.split('\t').count() == 3) + ); +} + +fn contains_case(table: &str, case_name: &str) -> bool { + table.lines().skip(2).any(|line| { + line.split('\t') + .next() + .is_some_and(|field| field == case_name) + }) +} + +fn expected_mutation_outcome(case_name: &str) -> Option<&'static str> { + MUTATIONS.lines().skip(2).find_map(|line| { + let mut fields = line.split('\t'); + if fields.next()? != case_name { + return None; + } + fields.nth(7) + }) +}