Specify canonical flat chunk layouts

CHANGELOG.md

@@ -54,6 +54,12 @@ after its public API and format compatibility policies are established.
 
 ### Added
 
+- The canonical `keep.flat-chunks/v1` durable layout specification, typed
+  `LayoutId` grammar, checked flat-plan bounds, domain-separated checksum,
+  exact golden records, field-complete `LayoutId` refusal tables and
+  cardinality-before-aggregate first-failure plan mutation ledger, and
+  verified storage-profile boundary replay law. Production encoding, decoding,
+  admission, ingestion, and reconstruction remain assigned to issue #10.
 - Canonical version-1 `ChunkId` calculation in a domain distinct from
   `BlobId`, with independent golden vectors.
 - A constant-memory `FastCdc` detector for `fastcdc-64k-v1` that preserves

README.md

@@ -29,6 +29,13 @@ for that profile and a domain-separated `ChunkId`. The detector consumes
 arbitrary borrowed feed partitions, emits identified spans without retaining
 candidate bytes, and flushes a final runt only when the caller declares EOF.
 
+The next M2 design slice freezes the canonical `keep.flat-chunks/v1` layout
+record and `LayoutId`: one bounded, ordered sequence of exact chunk identities
+and logical offsets bound to one `BlobId` and one `StorageProfileId`. Its
+language-neutral golden and mutation corpus exists, but the encoder, decoder,
+ingestion path, and reconstruction path remain assigned to the next
+implementation slice.
+
 Keep does **not** expose ingestion, layouts, or physical storage yet.
 Durability, retention, recovery, verification of stored structures, and
 garbage collection remain planned work. Calculating a `BlobId` or `ChunkId`
@@ -110,7 +117,10 @@ implemented proof boundary and explicit nonclaims. The separate
 [CDC profile corpus](conformance/cdc-profile/v1/README.md) freezes M2 boundary
 semantics, and the
 [chunk identity invariant](docs/invariants/chunk-identity/README.md) defines
-the implemented `ChunkId` and `FastCdc` proof boundary.
+the implemented `ChunkId` and `FastCdc` proof boundary. The
+[Flat Chunk Layout v1 specification](docs/formats/flat-chunk-layout-v1/README.md)
+and [layout corpus](conformance/layout/v1/README.md) freeze the next format
+boundary without claiming that its implementation exists.
 
 ## Contributing
 

conformance/layout/v1/ORIGIN.md

@@ -0,0 +1,62 @@
+# Flat Chunk Layout Corpus Origin
+
+## Provenance
+
+- Created: 2026-07-27
+- Governing specification:
+  `docs/formats/flat-chunk-layout-v1/README.md`
+- Layout identity envelope:
+  `docs/adr/0002-separate-identity-from-physical-storage.md`
+- Blob witnesses:
+  `conformance/golden-file-worldline/v1/identities.tsv`
+- Chunk witnesses: `conformance/chunk-id/v1/identities.tsv`
+- Storage-profile witness: `conformance/cdc-profile/v1/profile.tsv`
+- Independent digest executable: `b3sum` 1.8.5
+- Byte-assembly runtime: Node.js 24.18.0 `Buffer` fixed-width big-endian
+  operations
+
+## Construction
+
+The fixtures were constructed before any production layout encoder or decoder
+existed.
+
+The one-off construction process:
+
+1. assembled the 144-byte header from literal magic bytes and fixed-width
+   unsigned big-endian integers;
+2. copied existing canonical binary `BlobId` witnesses or independently
+   calculated the typed ADR-0001 preimage for the deterministic source;
+3. copied the accepted `StorageProfileId` digest;
+4. copied independently checked version-1 `ChunkId` lengths and digests;
+5. assembled each 44-byte entry in logical order;
+6. sent the exact typed checksum preimage to the external `b3sum` executable;
+7. appended the resulting raw 32-byte checksum;
+8. sent the exact ADR-0002 `LayoutId` preimage to `b3sum`; and
+9. rendered the final record bytes as lowercase hex for review.
+
+Node.js assembled bytes but did not supply a BLAKE3 implementation. `b3sum`
+was the only digest oracle. No Keep production type, serializer, encoder,
+decoder, or chunk detector generated expected layout bytes.
+
+The `profile-boundary-mismatch` mutation's replacement `ChunkId` digests were
+independently calculated from 262,143 and 2 zero bytes through the same typed
+version-1 preimage and external `b3sum` oracle.
+
+## Review controls
+
+The checked-in tables expose every source recipe, identity, entry coordinate,
+record length, checksum, and layout identity needed for an independent
+implementation to reproduce the fixtures.
+
+Issue #10 MUST add an independent checker that:
+
+- decodes the hex fixtures without using the production layout decoder;
+- reconstructs every field at its documented offset;
+- recomputes both checksum and `LayoutId` through an independently admitted
+  BLAKE3 capability;
+- executes every text and binary `LayoutId` refusal vector;
+- applies every mutation in `mutations.tsv`; and
+- cross-checks production encoding only after the independent values pass.
+
+Regenerating a fixture because production output differs is forbidden. Resolve
+the disagreement against the specification and independent digest oracle.

conformance/layout/v1/README.md

@@ -0,0 +1,99 @@
+# Flat Chunk Layout Corpus v1
+
+This directory is the implementation-independent design corpus for
+`keep.flat-chunks/v1`. It freezes canonical record bytes, `LayoutId`
+coordinates, and mutation expectations without supplying a production encoder
+or decoder.
+
+Issue [#10](https://github.com/flyingrobots/keep/issues/10) owns the
+implementation, independent checker, corruption tests, property tests, and
+continuous fuzz target. Until that issue lands, these fixtures are protocol
+evidence, not evidence that Keep can ingest, decode, admit, or reconstruct a
+layout.
+
+The normative format is the
+[Flat Chunk Layout v1 specification](../../../docs/formats/flat-chunk-layout-v1/README.md).
+
+## Files
+
+- `layouts.tsv` binds each case to source bytes, `BlobId`, record length,
+  checksum, `LayoutId`, and exact record fixture.
+- `entries.tsv` lists every entry's index, logical offset, exact length, and
+  version-1 `ChunkId` digest.
+- `*.layout.hex` contains one lowercase hexadecimal encoding of the complete
+  canonical binary record followed by exactly one LF. Hex is fixture transport;
+  decoded bytes are the durable record.
+- `invalid-layout-id-text.tsv` supplies hexadecimal input bytes and exact
+  refusal classes for malformed or unsupported text coordinates.
+- `invalid-layout-id-binary.tsv` supplies exact byte mutations and refusal
+  classes for malformed or mismatched binary coordinates.
+- `mutations.tsv` defines structural and verification mutations for issue #10.
+- `ORIGIN.md` records independent construction and review provenance.
+
+All TSV files are UTF-8, tab-delimited, and terminated by exactly one final LF.
+Fields use canonical unsigned decimal unless their name ends in `_hex`. `-`
+means inapplicable; it is not an empty field.
+
+## Golden cases
+
+The four records exercise distinct laws:
+
+- `empty` binds the empty `BlobId` and contains zero entries;
+- `one-zero` binds one byte to one one-byte `ChunkId`;
+- `max-plus-one-zeros` binds 262,145 zero bytes to one hard-maximum chunk and
+  one final one-byte runt; and
+- `zeros-long` binds four repeated hard-maximum chunks, proving that repeated
+  `ChunkId` values are lawful at distinct logical offsets.
+
+The chunk digests are existing independent witnesses from
+`conformance/chunk-id/v1/identities.tsv`. The registered profile identity is
+the accepted ADR-0003 witness.
+
+## Mutation protocol
+
+`mutations.tsv` uses these operations:
+
+- `replace-v1` replaces exactly `span_length` bytes with the same number of
+  decoded `parameter` hexadecimal bytes;
+- `xor-v1` XORs exactly `span_length` bytes with equal-length decoded
+  `parameter` hexadecimal bytes;
+- `insert-v1` inserts decoded `parameter` bytes before `offset`;
+- `delete-v1` removes exactly `span_length` bytes and requires parameter `-`;
+  and
+- `swap-v1` swaps the span at `offset` with the equal-length span whose
+  canonical decimal offset is `parameter`.
+
+`recompute-v1` recalculates the record checksum after a same-length mutation so
+the targeted semantic law, rather than checksum corruption, decides the
+result. `preserve-v1` leaves every existing checksum byte unchanged.
+
+Expected outcomes follow the specification's first-failure order. Inserting a
+duplicate flags-width field at offset 24 shifts the canonical header and makes
+`header_length` the first invalid field. Swapping the middle entries of
+`zeros-long` creates a gap at the first swapped position before the later
+offset decrease is observed.
+
+Mutations of opaque identity digests are not all structural errors. A
+rechecksummed `BlobId` digest or `ChunkId` digest can form another structurally
+valid layout. Those cases remain admitted until verified content proves
+`layout.reconstruction-mismatch` or `layout.chunk-mismatch`. The ledger names
+that later phase explicitly.
+
+The `profile-boundary-mismatch` mutation replaces the natural 262,144-byte
+hard-maximum boundary for 262,145 zero bytes with structurally valid
+262,143-byte and 2-byte chunks. Both replacement `ChunkId` values and the
+target `BlobId` name the exact source bytes. Verification MUST refuse the plan
+because replaying `fastcdc-64k-v1` emits the original boundary.
+
+## `LayoutId` coordinate mutations
+
+Issue #10 applies every row in `invalid-layout-id-binary.tsv` to the named
+case's `layout_id_binary_hex` value. Operations have semantics identical to the
+plan mutation protocol. Wider replacements deliberately distinguish an
+out-of-bounds plan length, an in-bounds but incongruent length, and a different
+valid record length that mismatches the named record.
+
+`invalid-layout-id-text.tsv` stores raw input as hexadecimal so empty input,
+whitespace, and other noncanonical bytes remain unambiguous in a
+line-oriented table. The production codecs MUST assert the exact typed
+outcomes in both files rather than only generic failure.

conformance/layout/v1/empty.layout.hex

@@ -0,0 +1 @@
+4b4545503a4c41594f55543a504c414e00010001000000000090002c00000000000000b000000000010100014b4545503a424c4f423a4944000000000001010000000000000000c0074a279c09f9d019dc10e4c821f79f1450cfb8541ab4627132ab9f3c75e33f000101aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb000000000000625119aa215bafe19f952bb3f0114147a8ef5097779113abd97fb2eb8bd0f615

conformance/layout/v1/entries.tsv

@@ -0,0 +1,9 @@
+keep.flat-chunk-layout-entries/v1
+case	entry_index	logical_offset	chunk_length	chunk_digest_hex
+one-zero	0	0	1	9b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff62
+max-plus-one-zeros	0	0	262144	c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b
+max-plus-one-zeros	1	262144	1	9b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff62
+zeros-long	0	0	262144	c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b
+zeros-long	1	262144	262144	c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b
+zeros-long	2	524288	262144	c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b
+zeros-long	3	786432	262144	c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b

conformance/layout/v1/invalid-layout-id-binary.tsv

@@ -0,0 +1,11 @@
+keep.flat-chunk-layout-id.invalid-binary/v1
+case	base_case	operation	offset	span_length	parameter	expected_outcome
+wrong-identity-magic	empty	xor-v1	0	1	01	layout-id.wrong-magic
+unsupported-identity-version	empty	replace-v1	16	2	0002	layout-id.unsupported-version
+unsupported-layout-codec	empty	replace-v1	18	2	0002	layout-id.unsupported-codec
+plan-length-out-of-bounds	empty	replace-v1	20	8	0000000002c000b1	layout-id.plan-length-out-of-bounds
+plan-length-not-congruent	empty	replace-v1	20	8	00000000000000b1	layout-id.plan-length-not-congruent
+plan-length-mismatch	empty	replace-v1	20	8	00000000000000dc	layout-id.plan-length-mismatch
+digest-mismatch	empty	xor-v1	28	1	01	layout-id.mismatch
+truncated-coordinate	empty	delete-v1	59	1	-	layout-id.wrong-length
+trailing-byte	empty	insert-v1	60	0	00	layout-id.wrong-length

conformance/layout/v1/invalid-layout-id-text.tsv

@@ -0,0 +1,21 @@
+keep.flat-chunk-layout-id.invalid-text/v1
+case	input_hex	expected_outcome
+input-too-long	616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161616161	layout-id.input-too-long
+empty-input		layout-id.malformed-structure
+wrong-scheme	6b6570743a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.wrong-scheme
+wrong-kind	6b6565703a626c6f623a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.wrong-kind
+malformed-version	6b6565703a6c61796f75743a313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.malformed-version
+unsupported-version	6b6565703a6c61796f75743a76323a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.unsupported-version
+unsupported-codec	6b6565703a6c61796f75743a76313a747265652d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.unsupported-codec
+unsupported-algorithm	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a7368613235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.unsupported-algorithm
+leading-zero-plan-length	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a303137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.noncanonical-plan-length
+signed-plan-length	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a2b3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.noncanonical-plan-length
+plan-length-overflow	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a31383434363734343037333730393535313631363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.plan-length-overflow
+plan-length-out-of-bounds	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137353a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.plan-length-out-of-bounds
+plan-length-not-congruent	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137373a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.plan-length-not-congruent
+short-digest	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a353339353136636335326665373433336533613938346134653261343837373637363637336262363264363230336138386466653163653235336332633066	layout-id.invalid-digest-length
+uppercase-digest	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306646	layout-id.noncanonical-digest-case
+nonhex-digest	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306667	layout-id.invalid-digest-alphabet
+trailing-field	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a353339353136636335326665373433336533613938346134653261343837373637363637336262363264363230336138386466653163653235336332633066383a6578747261	layout-id.trailing-data
+leading-space	206b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a35333935313663633532666537343333653361393834613465326134383737363736363733626236326436323033613838646665316365323533633263306638	layout-id.wrong-scheme
+trailing-newline	6b6565703a6c61796f75743a76313a666c61742d6368756e6b732d76313a626c616b65332d3235363a3137363a353339353136636335326665373433336533613938346134653261343837373637363637336262363264363230336138386466653163653235336332633066380a	layout-id.invalid-digest-length

conformance/layout/v1/layouts.tsv

@@ -0,0 +1,6 @@
+keep.flat-chunk-layouts/v1
+case	source_recipe	source_parameter	source_count	logical_length	blob_id	storage_profile_id	entry_count	record_length	record_checksum_hex	layout_id	layout_id_binary_hex	record_fixture
+empty	empty-v1	-	0	0	keep:blob:v1:blake3-256:0:c0074a279c09f9d019dc10e4c821f79f1450cfb8541ab4627132ab9f3c75e33f	keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb	0	176	625119aa215bafe19f952bb3f0114147a8ef5097779113abd97fb2eb8bd0f615	keep:layout:v1:flat-chunks-v1:blake3-256:176:539516cc52fe7433e3a984a4e2a4877676673bb62d6203a88dfe1ce253c2c0f8	4b4545503a4c41594f55543a494400000001000100000000000000b0539516cc52fe7433e3a984a4e2a4877676673bb62d6203a88dfe1ce253c2c0f8	empty.layout.hex
+one-zero	repeated-byte-v1	00	1	1	keep:blob:v1:blake3-256:1:1cfb8fa9e917aba15a1f592095f377ff180755fe1212b0d7d2ec750bd128b606	keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb	1	220	093e7ab79a7e454629ab504b1736e4e6b655a64f1ba6780cb97d3e074384db7c	keep:layout:v1:flat-chunks-v1:blake3-256:220:887da23f1a7483359a78fc9a7fde80030ec2c4690603803f0ab7d0edb56575b8	4b4545503a4c41594f55543a494400000001000100000000000000dc887da23f1a7483359a78fc9a7fde80030ec2c4690603803f0ab7d0edb56575b8	one-zero.layout.hex
+max-plus-one-zeros	repeated-byte-v1	00	262145	262145	keep:blob:v1:blake3-256:262145:dbe1c952ccd18c317f9c7a83301da48e251730d5b9279ecfb101cd80da9b6f07	keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb	2	264	9e23e3a56d449e4f19919270602d29f2eb8a614f4a022ea42d1c0243d409eb42	keep:layout:v1:flat-chunks-v1:blake3-256:264:2b281c5e381edfad1dc513d3c2a8818fb4b95d0b188072e27ecc1e18a748afa6	4b4545503a4c41594f55543a494400000001000100000000000001082b281c5e381edfad1dc513d3c2a8818fb4b95d0b188072e27ecc1e18a748afa6	max-plus-one-zeros.layout.hex
+zeros-long	repeated-byte-v1	00	1048576	1048576	keep:blob:v1:blake3-256:1048576:a5e90495f5a61a299dc276d89da91cd60add2b62e997f48cc14bd821cdb6724a	keep:storage-profile:v1:blake3-256:aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb	4	352	c82f881f3af274b35c04c693228cc9078eb88044d60e62aef60dfa42a9310582	keep:layout:v1:flat-chunks-v1:blake3-256:352:ecc098f68f7c6691936bced4b8ec31aa55de34f78224ce3cffb035f0f97c7513	4b4545503a4c41594f55543a49440000000100010000000000000160ecc098f68f7c6691936bced4b8ec31aa55de34f78224ce3cffb035f0f97c7513	zeros-long.layout.hex

conformance/layout/v1/max-plus-one-zeros.layout.hex

@@ -0,0 +1 @@
+4b4545503a4c41594f55543a504c414e00010001000000000090002c000000000000010800000002010100014b4545503a424c4f423a4944000000000001010000000000040001dbe1c952ccd18c317f9c7a83301da48e251730d5b9279ecfb101cd80da9b6f07000101aafa6f05bdc8894306abd41ec6f2b3b76cde995f2598fa3fd547d81fbe1a34eb000000000000000000000000000000040000c7794a07d017e900d37c3a874f1dd9fd4783a3ead2fb176755ae42a4ce65f87b0000000000040000000000019b9c9a42912a0efdcd41e83ea024d72f10f2627d239e4eb240dd53f39ce0ff629e23e3a56d449e4f19919270602d29f2eb8a614f4a022ea42d1c0243d409eb42