core(project-boundary): finish persisted schema/version verdict and retire observation-only shadow authority

[qnbs]

Context

The living docs/native/CORE-MIGRATION-LEDGER.md marks the Project state-shape compatibility adapter as an in-progress prerequisite before native Core/Qt admission:

• features/project/coreBoundaryAdapter.ts already translates Redux EntityState/array shapes to the renderer-neutral Rust project model and preserves ID/order invariants;
• parity fixtures and an observation-only desktop shadow caller exist;
• the current Rust verdict is explicitly partial because unknown fields are not rejected/fully classified;
• the boundary envelope currently synthesizes schemaVersion rather than consuming one persisted as authoritative data;
• there is deliberately no production authority switch yet.

No dedicated open issue currently owns this remainder.

Goal

Turn the current proof into an explicit, versioned Project/Core compatibility contract that can safely support later authority migration and Qt, without importing Redux Toolkit shape into Rust Core or discarding forward-compatible project data.

Required decisions

1. Persisted version authority

Decide where the compatibility/schema version comes from and how it is persisted.

Do not let a synthesized current version make older/unknown persisted data appear already migrated.

Define:

• version field/owner;
• absent-version legacy semantics;
• supported older/current/future versions;
• upgrade/migration admission;
• downgrade behavior;
• error/recovery result when a version is unsupported.

2. Unknown-field policy

The current Rust model does not provide a complete verdict for fields outside its modeled subset.

Choose an explicit policy per boundary/evolution stage:

PRESERVE_OPAQUE
ALLOW_FORWARD_COMPATIBLE
REJECT_UNKNOWN
MODEL_AND_VALIDATE
OUT_OF_SCOPE_BUT_MUST_NOT_BE_DROPPED

The default must never be silent data loss.

If Core does not understand a field that TypeScript currently persists, an observation/round-trip path must either preserve it or decline authority rather than deserialize/re-serialize a narrower object and erase it.

3. Authority-switch admission

Define the gates that move the current shadow comparison from:

OBSERVATION_ONLY

to any production decision/write authority.

At minimum require:

• complete field inventory for the admitted lifecycle;
• compatibility/version verdict;
• no-loss round-trip proof;
• TS/Rust accept/reject parity where both remain active;
• rollback/fallback behavior;
• representative historical fixtures;
• no Redux-specific types leaking into Core.

Entity-state invariants

Preserve the already-established contract for characters/worlds:

• stable IDs preserved exactly, including all IDs accepted by the authoritative import/persistence contract;
• declared order preserved;
• duplicates fail closed;
• missing ids/entity correspondence fails closed;
• orphan entities fail closed or are handled by an explicitly versioned migration;
• prototype-named IDs must not be lost through ordinary object maps (coordinate with local-first: preserve prototype-named entity IDs when rebuilding ProjectData from Yjs #548 where Local-First uses the same data contract).

Fixture/evidence matrix

Add/maintain fixtures for:

• current full project;
• representative historical projects;
• missing legacy version;
• supported old version;
• future/unsupported version;
• extra unknown fields at top level and nested modeled boundaries;
• array and EntityState character/world forms;
• prototype/empty/whitespace IDs according to the established compatibility contract;
• duplicate/missing/orphan entity references;
• optional fields;
• large representative project;
• round-trip no-loss comparison.

Use golden fixtures in both TS and Rust where transition parity matters.

Relationship to other work

• security(core/R-15): renderer-neutral encrypted desktop storage, durable migration & identity binding #445 owns future renderer-neutral protected storage/envelope security, not this project-domain compatibility verdict.
• local-first: preserve prototype-named entity IDs when rebuilding ProjectData from Yjs #548 owns the separate Local-First/Yjs reconstruction bug.
• onboarding: project-bootstrap repair effect has three pre-existing edge-case gaps #531 concerns bootstrap metadata intent, not Core schema authority.
• Qt must consume this admitted Core boundary; QML must not recreate its own project schema rules.

Acceptance criteria

• Persisted project schema/version authority is explicit and no longer synthesized in a way that can hide compatibility state.
• Unknown-field behavior is explicitly defined and cannot silently discard persisted user data.
• Full admitted project-field inventory exists for any authority switch.
• Historical/current/future-version fixtures exercise TS/Rust compatibility.
• Array/EntityState conversion remains lossless for valid data and fail-closed for malformed correspondence.
• Shadow verdict remains observation-only until all authority-switch gates pass.
• A rollback/fallback path exists for any future Core authority transition.
• Qt can use the same project contract without Redux Toolkit coupling.
• CORE-MIGRATION-LEDGER.md row 9 is updated whenever the authority state or compatibility policy changes.

Non-goals

• moving Redux reducers into Rust;
• forcing strict unknown-field rejection when preservation/forward compatibility is the safer admitted policy;
• implementing R-15 encryption here;
• redesigning import UX;
• beginning Qt vertical slices before the binding roadmap gates permit it.

[qnbs]

Queue / execution status — 2026-09-05

Priority: P0 — immediate P0-B prerequisite before renderer-neutral R-15 implementation may advance.

This issue is not itself the R-15 implementation issue. Its purpose is to close the remaining Project/Core authority-boundary ambiguity: persisted schema/version authority, unknown-field preservation/no-loss behavior, historical/current/future fixtures, and a truthful admission verdict for any later Core authority switch.

Required sequencing:

#553 terminal + post-main green → re-read live Core Migration Ledger / native roadmap → only if all implementation-readiness gates are genuinely satisfied may #445 advance into its next admitted R-15 slice.

Closing #553 must not mechanically flip S5_IMPLEMENTATION_READY or production authority. Update the ledger only from evidence. Unknown fields must never be silently dropped merely to obtain TS/Rust parity, and shadow/observation-only behavior remains non-authoritative until its stated no-loss/admission gates pass.

[qnbs]

PR #617 review carry-forward / implementation observation ledger — 2026-09-05

PR #617 remains a design/contract-only PR. This comment preserves implementation obligations surfaced during its review so they cannot disappear when review threads are resolved or become stale. It does not expand #553 beyond its existing compatibility/no-loss/authority-switch scope.

Already verified VALID / IMPLEMENTATION_REQUIRED

The #553 implementation must explicitly reconcile and test all of the following before any authority switch:

1. Post-header typed-parse failure: a document whose raw/header parse yields a valid CURRENT schemaVersion but whose owned/current typed fields fail validation must transition to preserve-first, non-editable MALFORMED recovery — never remain admitted as CURRENT and never reach autosave/write authority.
2. Migration commit fencing: LEGACY_TO_V1 and every numbered migration must apply the same atomic source-generation check + commit fence required for ordinary raw-carrier write-back. A write landing after verification but before replacement must make the migration fail closed rather than overwrite the newer generation.
3. schemaVersion validation/parity: invalid present version markers need deterministic cross-renderer classification. The implementation must not rely on TS/Rust parser quirks or coercion.
4. Nested field/path inventory: MODEL_AND_VALIDATE is path/domain-granular. A containing object/collection is not switch-ready merely because Rust has a struct for it; nested TS-persisted fields and historical accepted values must be reconciled or preserved opaquely.
5. Large-project fixture must become measurable: implementation/test admission must choose concrete representative scale/resource bounds and a fail-closed behavior when supported parser/memory bounds are exceeded; an arbitrary “large” fixture is not sufficient evidence.

New exact-head b6ca45a3bf2e17c74351f8fcbf5dfc678e6166a6 observations

These are currently unresolved on PR #617. If Claude fixes them in the design PR, this list becomes evidence/redundancy; if #617 merges without them, they remain mandatory #553 implementation/reconciliation items:

6. Absent version vs. malformed must remain disjoint: missing schemaVersion is LEGACY_UNVERSIONED and enters LEGACY_TO_V1; only an unparseable envelope or a present invalid version marker is MALFORMED.
7. Entity create/delete semantics in no-loss write-back: ordinary owned edits may intentionally add or delete an owned entity/subtree. The no-loss verifier must distinguish the declared owned delta from accidental loss, allow deletion of the intended entity including its opaque children, preserve opaque fields on surviving entities, and test create/delete/reorder cases by stable ID.
8. Duplicate schemaVersion keys: classification must be deterministic across TS/Rust. Prefer fail-closed rejection of duplicate version keys rather than parser-dependent last-key-wins behavior; add cross-renderer fixtures.
9. Every authoritative writer must use the canonical raw-carrier contract: the overlay/owned-path verification/generation fence cannot be Core-only. TypeScript/React-PWA remains the current writer and a permanent coexisting renderer, so TS save/autosave/flush must also preserve opaque canonical payload data and participate in the same write-back invariants. Add a TS autosave/write fixture carrying opaque fields.

Existing third-wave contract corrections already landed in b6ca45a3

For provenance: the current PR head already records the required portable-boundary stripping/re-derivation of filesystem trust metadata, source→destination verification for migration rename/replacement manifests, snapshot-create→restore canonical-payload coverage, schema bumps for additive values on REJECT_UNKNOWN discriminants, lossless numeric precision requirements, and historical-domain compatibility before any path becomes MODEL_AND_VALIDATE authority.

Gate reminder: #553 must not close, and R-15/#445 must not advance, merely because #617 merges. Required sequence remains #617 terminal + post-main green → #553 implementation/fixtures/parity/no-loss gates terminal + post-main green → re-read live ledger/roadmap → recompute R-15 readiness from evidence.

[qnbs]

PR #618 Slice A observation — duplicate-key detection must honor decoded JSON property names

Live review of PR #618 head 571adb0819c8cdc065d0c706b7c5654c03b6b17f found one additional compatibility/parity edge that is not covered by the current literal-text duplicate-key scanner.

Both the TS and Rust implementations currently detect duplicate top-level schemaVersion keys by scanning the raw source for the literal token "schemaVersion". JSON property names are semantically decoded strings, however, so a payload such as:

{"schemaVersion":1,"\u0073chemaVersion":2}

contains two semantically identical top-level property names after JSON string decoding, while the current scanner can count only the literal spelling. The normal parsed object then collapses them to one key, so this can bypass the admitted contract rule that any duplicate schemaVersion key always classifies MALFORMED deterministically across implementations.

Required disposition before Slice A is treated terminal: either fix PR #618 with a duplicate-key detector/parser that compares decoded top-level property names, or explicitly carry this as a mandatory #553 implementation correction before any ingress wiring/authority admission. Add mirrored TS/Rust fixtures for escaped-equivalent duplicate keys (at least one \uXXXX spelling) and retain the existing nested-key negative control.

Classification: VALID / IN-SCOPE / CONTRACT-COMPLIANCE + PARITY, not a CodeScene style issue.

This comment is observational/carry-forward only; it does not authorize unrelated scope expansion.

[qnbs]

PR #618 Slice A — Codex carry-forward / live triage

Current Slice A review surfaced three substantive compatibility/parity obligations that remain within #553's existing admitted scope. If Claude fixes them in #618, this comment becomes supporting evidence; if any survives the PR, it remains mandatory before #553 can be terminal:

1. Decoded duplicate-key identity: duplicate schemaVersion rejection must compare decoded JSON property names, not raw source spelling. Escaped-equivalent keys such as "\u0073chemaVersion" / "schema\u0056ersion" are semantically schemaVersion and must classify MALFORMED in both TS and Rust. Keep mirrored positive/negative parity fixtures.
2. LEGACY_TO_V1 verify-before-stamp: an arbitrary unversioned object must not be stampable into an apparently CURRENT v1 record without the contract's required legacy-shape / V1-conformance verification. Either validate before stamping or require an input type/token that can only be produced by successful validation; preserve the later no-loss + fenced durable-commit gates.
3. Header classification must not depend on unrelated nested depth/shape: Rust currently full-deserializes into serde_json::Value before classifying. A deeply nested but valid FUTURE document can hit serde_json recursion limits and become MALFORMED while TS returns FUTURE. Classification must extract/validate the top-level version header without requiring full payload deserialization; add a cross-renderer deep-nesting FUTURE fixture.

Review-noise classifications for provenance:

• Graphify raw-report comment on the original head = STALE / ALREADY FIXED; current head a7a89fe5… reverts graphify-out/GRAPH_REPORT.md out of the PR.
• Codex signature finding on 571adb0819c8cdc065d0c706b7c5654c03b6b17f = FALSE POSITIVE / PROVIDER ARTIFACT; GitHub REST reports verification.verified=true, reason=valid, SSH signature, verified_at=2026-09-05T21:35:21Z, and the repository Verified Signatures CI job passed on that head.

No roadmap expansion. These are Slice-A/#553 correctness and parity details only.

[qnbs]

PR #618 Slice A — post-push carry-forward: header scanner must still reject syntactically invalid JSON

Live head inspected: 8347d92edbed831c7d55e2ca4824ec702c7cfc2e.

The three previously tracked Slice-A findings are materially addressed on this head: decoded duplicate-key identity, verify-before-stamp, and avoiding full Rust serde_json::Value traversal of unrelated deeply nested FUTURE data.

One new contract/parity gap remains in the replacement Rust header scanner and must not be lost if PR #618 moves quickly:

VALID / IN-SCOPE / CONTRACT-COMPLIANCE — header-only must not mean syntax-permissive

scan_raw_header / skip_json_value skip unrelated values by byte/depth scanning and do not validate the full JSON grammar of those skipped values or object separators. Therefore malformed documents can be classified as CURRENT/FUTURE in Rust while TypeScript's JSON.parse returns MALFORMED.

Representative cases to fixture on both sides:

• missing comma: {"schemaVersion":2 "x":3}
• invalid escape in an unrelated value: {"schemaVersion":2,"x":"\\q"}
• malformed nested value/separator variants that remain balanced by braces/brackets

Contract requirement remains: envelope unparseable as JSON at all => MALFORMED; FUTURE tolerance applies to unknown shape/semantics, not invalid JSON syntax. Permanent TS/Rust classification parity therefore requires the Rust header-only path to validate JSON syntax while avoiding recursive materialization of unrelated nested values.

Implementation options are open, but do not regress to full recursive serde_json::Value materialization that reintroduces the deep-FUTURE recursion-limit bug. Prefer a streaming/duplicate-aware/header visitor or a small syntax-validating tokenizer that can skip nested values without allocating the full tree.

This should be treated as a #618 merge blocker if still present on the final exact head; otherwise carry forward until fixed before any ingress wiring/authority admission.

[qnbs]

PR #618 follow-up — LEGACY_TO_V1 verification is improved but still incomplete

Current head 8347d92edbed831c7d55e2ca4824ec702c7cfc2e changes stampLegacyToV1 to return null unless a minimal shape check passes. This fixes the worst {}/missing-container case, but it is not yet equivalent to the contract's required LEGACY_TO_V1 "verify conformance to PROJECT_SCHEMA_V1" step.

Observed gaps on the current head:

• ProjectData.logline is required and logline is MODEL_AND_VALIDATE, but the helper does not validate it.
• author, when present, is also MODEL_AND_VALIDATE, but the helper does not validate its type.
• characters/worlds are only checked as array-or-EntityState containers; nested entity IDs/content/domain validity are not verified, so e.g. structurally invalid entity payloads can still be stamped.
• The helper itself documents this as only "minimal structural verification".

For Slice A, either make the verifier actually use the canonical/historically-compatible V1 validation domain for all currently owned paths, or change the API so stampLegacyToV1 can only accept a validated token/type produced by that canonical verification. Do not let a shallow container check become the migration admission gate consumed by Slice B.

Treat as an in-scope #618 correctness blocker unless the PR deliberately downgrades stampLegacyToV1 to a lower-level non-admission primitive and leaves the real verify gate explicitly for the next slice.

[qnbs]

PR #618 carry-forward / Slice-A numeric-domain observation (current head 8347d92): the admitted #617 contract explicitly requires exact large-integer preservation / fail-closed behavior outside a jointly lossless numeric domain. Current TS classifier accepts any non-negative Number.isInteger value, while Rust parses the captured numeric span through serde_json::Value::as_f64() and casts to u64. This can diverge or silently round beyond JS's safe-integer range. Before Slice A is terminal, define a single jointly lossless schemaVersion numeric domain (practically the JS safe-integer domain unless another exact strategy is proven), reject outside it as MALFORMED, and add mirrored TS/Rust boundary fixtures around 2^53-1 / 2^53 and representative very-large literals. Do not treat this as a generic raw-payload opaque-number policy; this is specifically the schemaVersion discriminant used for admission.

[qnbs]

PR #618 terminal reconciliation / next #553 gate — 2026-09-06

PR #618 (feat(project): add persisted schema-version classification (#553 Slice A)) is terminal and post-main truthful-green.

Verified terminal state

• final PR head: 8cabb60d56f7dc9c0eb380b4d28994946d9db125
• merge/main commit: 5511d37ecd874c3240369d2da44360874d750c10
• exact-head PR CI/CD #2771 (33999616474): SUCCESS
• exact-head PR CodeQL #2619 (33999616477): SUCCESS
• post-merge main CI/CD #2772 (34000685271): SUCCESS
• post-merge main CodeQL #2620 (34000685270): SUCCESS
• all PR inline review threads reconciled/resolved; repeated nonexistent/unsigned squash-SHA findings were provider artifacts, while real signatures/gates were verified.

Slice A obligations now closed by evidence

• fresh production PROJECT_SCHEMA_V1 and disjoint legacy/current/future/older/malformed classification in TS + Rust;
• decoded-key duplicate schemaVersion rejection, including escaped-equivalent keys;
• full JSON syntax rejection on Rust without whole-document recursive materialization;
• deep valid FUTURE payloads remain classifiable without serde recursion-limit failure;
• exact JSON whitespace parity; malformed separators/escapes/numbers/trailing content fail closed;
• TS own-property presence check (Object.hasOwn) prevents prototype-chain pollution from fabricating schemaVersion;
• bounded common numeric admission domain through 2^53-1, with raw-token Rust parsing used to match V8/f64 rounding behavior;
• the unsafe partial LEGACY_TO_V1 stamping helper was removed rather than being mistaken for full V1 conformance verification;
• adversarial mirrored TS/Rust fixture coverage landed; Rust tests were split out so the production module remains under the repository file-size guideline.

One remaining Slice-A contract semantic to correct before ingress/authority use

The admitted contract says the accepted schemaVersion representation is a non-negative integer JSON number, explicitly treating a fractional number such as 1.5 as invalid/MALFORMED. The final runtime implementations currently validate the parsed IEEE-754 value. Therefore a raw fractional literal such as 9007199254740991.4 can round to integer 9007199254740991 in both V8 and Rust f64 parsing and be admitted as FUTURE. This is cross-renderer-parity-correct but not literally the admitted raw JSON-number grammar.

Required disposition before any real ingress/write-authority wiring: validate the raw schemaVersion token's mathematical integer-ness (while still accepting equivalent integer-valued spellings such as 1.0/1e0 if the contract intends them as integer JSON numbers), or amend the contract deliberately if parsed-value semantics are desired. Do not silently let IEEE rounding redefine the admitted grammar.

Classification: VALID / CONTRACT-COMPLIANCE / IMPLEMENTATION_REQUIRED BEFORE INGRESS, not a reason to roll back #618.

Remaining #553 implementation work (not closed by #618)

• canonical raw-carrier + MODEL_AND_VALIDATE projection for the owned V1 paths (title, logline, present author, characters, worlds, manuscript) with historical TS/Rust domain reconciliation;
• real LEGACY_TO_V1: recognize -> validate -> stamp -> no-loss verify -> fenced durable commit; no partial heuristic;
• CURRENT header success + typed/projection failure -> preserve-first noneditable MALFORMED recovery, never autosave/write authority;
• universal ingress admission and canonical durable egress for storage/import/snapshot/recovery paths;
• every authoritative TypeScript writer/autosave/flush path must preserve opaque raw-carrier fields and participate in generation fencing;
• stable-ID owned create/delete/reorder overlay semantics with opaque survivor preservation;
• migration source-generation/commit fencing and destructive-migration backup requirements;
• backend-local trust metadata stripping/re-derivation at portable boundaries;
• complete parity/no-loss/historical/snapshot/autosave/concurrency/numeric-precision/large-project fixture matrix;
• no authority switch until all admitted gates pass.

Living-ledger truth drift to reconcile in the next #553 slice

docs/native/CORE-MIGRATION-LEDGER.md row 9 still says IMPLEMENTATION_STARTED = NO and still describes the persisted-version work as not implemented/synthesized. After #618 this is no longer current truth. Update row 9 narrowly to record that Slice A classification/parity is implemented while ingress, canonical payload, migration/writeback, and authority-switch gates remain incomplete/observation-only.

Gate remains: #553 stays OPEN. Do not mechanically mark R-15/#445 implementation-ready. Next work must continue the smallest causal #553 implementation slice; after #553 itself becomes terminal + post-main green, re-read the live ledger/roadmap and recompute R-15 readiness from evidence.