graph/cycle_sink: the concrete Lance-backed WalSink — storage-proven cognitive-cycle sink (§I.6) - #911
Conversation
…ive plan Six-agent read-only investigation + unshallowed git history (5 grafted roots -> 4162 commits) established that lance-graph holds TWO MedCare lineages, not one incomplete runtime: - LIVE (consumer-pull): MedCare-rs medcare-bridge -> vendored lance-graph-ogar -> MedcareBridge = UnifiedBridge<HealthcarePort> -> OGAR canonical Health codebook 0x0901..0x090C. Bridge migration COMPLETED at ddb6c84 (2026-06-21); deprecated alias only. Contract codebook mirror verified in sync slot-for-slot; the 7-alias-vs-12-slot gap is intentional (harvest mints, no OGIT entity). - DEAD (host-side scaffold): modules/medcare/manifest.yaml -> CallcenterSupervisor -> MedcareConsumerActor, frozen since birth 2026-05-13. Manifest compile-time parsed but runtime-orphaned (one caller: a test); entity codes / action_capabilities / message_type discarded pre-codegen; medcare_policy nonexistent; StubConsumerActor hard-coded; Dispatch rejected before any child; MedCareActor / MedCareMessage exist in NEITHER repository. New ACTIVE plan medcare-consumer-pull-thinking-proof-v1: prove one real medical thought over the live consumer-pull path (HealthcarePort classids -> cognitive-shader-driver + real MailboxSoA -> owner_adapter cast -> the #879 sparse cycle loop), falsifiers F1-F4 including the currently-absent Healthcare fail-closed unknown-actor test. ogar-obo (MONDO/HPO/Uberon/PATO, verified real, zero consumer edges) is an optional slice. Older MedCare plans classified (completed / dormant- decision-required / stale-but-unmarked); open decisions carried, not blocked on (OQ-2 retention 2190/3650, Ueberweisung/Anamnese canon gap, dead-lineage retire-vs-revive, .grok lineage, NoopAuditSink default). Documentation-only: no Rust, no tests, no manifests changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp
…t to deterministic regeneration; re-home the proof to the live consumer Round-2 grain-of-salt realignment (operator-ruled), post-#879-merge. 1) Pre-commit failure contract. The authoritative rule is deterministic regeneration, not retained-batch retry: sealed Vn + unchanged Kanban task + deterministic computation = the same provisional intent on the next sweep Commit fails before Vn+1 exists -> publish nothing, mutate no owner, advance no watermark, discard provisional slots / held moves / planning results, rerun the unchanged task from Vn. - SealFailure{casts} reclassified: OPTIONAL retry cache / implementation convenience only — never the correctness mechanism, never a provisional-planning ledger; dropping it is always sound. - recover_fleet doc-pinned as COMMITTED-HISTORY recovery ONLY (Vn+1 exists, application/restart interrupted); explicitly separated from ordinary pre-commit write failure — no shared state. - HeldIntent doc-pinned as within-success scheduling convenience, discarded on a failed seal, regenerated by the next thought pass. - NEW authoritative falsifier pre_commit_failure_discards_everything_and_regenerates_from_vn: derive cycle C deterministically from Vn, inject commit failure, DROP the SealFailure cache, assert no version/phase/watermark change, rerun the unchanged task from Vn, assert the same SEMANTIC sparse cycle regenerates, allow commit, assert exactly one Vn+1 and one advance per represented owner. Object identity of the first heap batch deliberately not asserted. - The prior byte-identical-retry test demoted to an optional-cache probe (secondary, convenience path). 20 cycle_driver tests green; clippy + fmt clean. Latency figures in review prose are operator-provided measurements, not workspace-reproduced benchmarks; their values are not restated. 2) Proof re-homed (medcare-consumer-pull-thinking-proof-v1 section 4): primary home = MedCare-rs (the live composition root); lance-graph contributes only genuinely-missing GENERIC seams — no MedCare-shaped host adapter here (that would rebuild the dead lineage in miniature). Hard requirement: the proof must invoke the existing cognitive-shader-driver + MailboxSoA operational unit; shade_owner only as the driver's existing downstream gate, never a substitute. Trace-and-report obligation added; F1 strengthened to a discriminating driver outcome. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp
…cognitive-cycle sink
Closes the gap persist_sink deliberately left ('this module builds NO
concrete Lance sink'): LanceCycleSink implements
lance_graph_planner::persist_sink::WalSink over the official Lance 9
insert path (Dataset::write / Dataset::append — the same InsertBuilder
transaction machinery every Lance writer uses). No bespoke ledger, no
acknowledgement protocol, no parallel replay system: Lance's own
manifest/version chain IS the WAL.
The §I.6 invariant made physical: one detached cycle batch → ONE
official Lance commit → exactly one real DatasetVersion base+1, with
the epistemic fence in both halves — pre-commit head==base (a stale
base is refused with nothing written; an empty store's head is
DatasetVersion(0)) and post-commit published==base+1 (Lance
auto-resolves append-append conflicts, so a foreign interleaved writer
surfaces as a loud timeline anomaly instead of silently shifting the
cycle↔version identity reads derive). The per-cycle frame row seals
the cycle↔version mapping inside the same atomic commit, so the
coarse timeline survives restart with zero sidecar state. Order is a
write-side property: landings are stored in the already-deinterlaced
freeze order and scanned back with scan_in_order(true) — never sorted
on read.
Domain-0x09 witness contract (module doc): the patient SoA is the
ONLY place patient reasoning is written to Lance, so the store is
witness-focused and maximally rich — payload carries the 512-byte
canonical EpisodicWitness node whose edges point INTO the immutable
domain-0x03 ontology address space; the cycle takes ontology
immutability for granted for its representation window via
base_version, and the sealed versioning is a reflection of the
thinking.
Gated on the default-on 'planner' feature (the trait lives in the
optional planner dep). 6 tokio tests, every guarantee proven against
a REOPENED dataset (fresh sink instance + fresh Dataset::open): seal
survives restart; stale base fenced writing nothing; sequential
cycles chain V1→V2→V3 with strictly-after filtering; a zero-landing
cycle advances the timeline only; an empty store reads empty
(DatasetNotFound is a state, not an error); move-nullability +
payload byte-exact round-trip. Board hygiene (LATEST_STATE +
PR_ARC_INVENTORY) in this same commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp # Conflicts: # .claude/board/LATEST_STATE.md
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_4b5e21b3-bb5e-4a3d-9322-d5b1c43725d3) |
📝 WalkthroughWalkthroughThis PR adds a Lance-backed ChangesCycle persistence and recovery
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e081ba32b8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/board/EPIPHANIES.md:
- Around line 871-879: Move the entire `2026-08-02 —
E-TWO-MEDCARE-LINEAGES-THE-LIVE-ONE-PULLS-THE-DEAD-ONE-HOSTS-1` entry to the
topmost newest-first ledger position, ahead of all older entries and the other
existing `2026-08-02` item. Preserve the exact entry content and retain the
relative order of every other entry.
In @.claude/board/INTEGRATION_PLANS.md:
- Around line 147-169: Move the complete 2026-08-02
medcare-consumer-pull-thinking-proof v1 entry to the head of the ledger in
INTEGRATION_PLANS.md, before the existing 2026-08-02 entry. Preserve the entry’s
content exactly and do not edit any older entries.
In @.claude/board/LATEST_STATE.md:
- Around line 6-8: Define and document reconciliation for durable post-commit
version mismatches in commit_cycle(): do not treat PersistError::Write as
blindly retryable after rows may have been appended; serialize commits or return
an explicit ambiguous/committed result, and provide a rule to detect and
reconcile an existing frame-row cycle-to-version mapping before retrying the
same SealFailure.casts. Update the `#879` entries at .claude/board/LATEST_STATE.md
lines 6-8 and 150-152 and .claude/board/PR_ARC_INVENTORY.md lines 38-40; all
three sites require the reconciliation behavior to be recorded.
In @.claude/plans/medcare-consumer-pull-thinking-proof-v1.md:
- Line 29: Update the three Markdown diagram code fences in the document to
specify the text language identifier, including the fences near the referenced
sections, so each opening fence uses text and satisfies markdownlint MD040.
In `@crates/lance-graph/src/graph/cycle_sink.rs`:
- Around line 204-227: Enforce the canonical 512-byte payload size in the
landing-write loop around payload.append_value, rejecting any s.payload whose
length is not exactly 512 bytes before persistence. Update existing
short-payload fixtures to 512 bytes and add a test verifying invalid landing
payloads are rejected while preserving the canonical node row stride and 16-byte
edge reservation.
- Around line 369-408: Prevent interleaved cycle commits from appending under a
shifted DatasetVersion: update the cycle sink commit path at
crates/lance-graph/src/graph/cycle_sink.rs:369-408 to use an expected-version
conditional append, or return a distinct committed-anomaly result containing the
actual published version. Update
crates/lance-graph-supervisor/src/cycle_driver.rs:26-50 and :1016-1117 so this
result is routed through committed-history handling rather than classified as an
uncommitted CycleError::Seal; preserve the actual published version when
reporting the anomaly.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 01554bd8-d079-4c0f-86d7-aa25f719acf6
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (8)
.claude/board/EPIPHANIES.md.claude/board/INTEGRATION_PLANS.md.claude/board/LATEST_STATE.md.claude/board/PR_ARC_INVENTORY.md.claude/plans/medcare-consumer-pull-thinking-proof-v1.mdcrates/lance-graph-supervisor/src/cycle_driver.rscrates/lance-graph/src/graph/cycle_sink.rscrates/lance-graph/src/graph/mod.rs
| ## 2026-08-02 — E-TWO-MEDCARE-LINEAGES-THE-LIVE-ONE-PULLS-THE-DEAD-ONE-HOSTS-1 — MedCare is integrated through consumer-pull; the host-side actor/manifest scaffold froze on its birth day | ||
|
|
||
| **Status:** FINDING (six-agent read-only investigation + unshallowed git history, HEAD `71d1db1`). **Confidence:** High — every claim carries a commit sha or file:line; contradicting prose ledgers were checked against code. Plan: `.claude/plans/medcare-consumer-pull-thinking-proof-v1.md`. | ||
|
|
||
| **The finding.** lance-graph does NOT contain "one incomplete MedCare runtime" — it contains **two lineages**. The LIVE one is consumer-pull: MedCare-rs `medcare-bridge` → vendored `lance-graph-ogar` → `MedcareBridge = UnifiedBridge<HealthcarePort>` (collapse commit `ddb6c840`, 2026-06-21; deprecated alias since `10e717d4`) → OGAR canonical Health codebook (`0x0901–0x090C`; the contract mirror verified in sync slot-for-slot; the 7-alias-vs-12-slot gap is INTENTIONAL per OGAR `ports.rs:598-606`). The DEAD one is the host-side scaffold born in the sprint-7 wave (2026-05-13, `927788e9`/`3f67aed6`) and **never functionally touched again**: `modules/medcare/manifest.yaml` is compile-time parsed (`contract/build.rs`) into `MANIFEST_METADATA` whose ONLY caller is a test; its richest fields (entity codes 100–105, all five `action_capabilities`, `actor.message_type`) are parsed then DISCARDED pre-codegen; `rbac_policy: medcare_policy` names a nonexistent symbol; `CallcenterSupervisor::spawn_consumer_actor` unconditionally spawns `StubConsumerActor` (`supervisor.rs:369`); `DispatchToG` returns `DispatchNotImplemented` before any child actor for every non-Health envelope; `MedcareConsumerActor` has ZERO constructors repo-wide; and the manifest's declared `MedCareActor`/`MedCareMessage` exist in NEITHER repository (verified against the MedCare-rs sibling directly). | ||
|
|
||
| **Load-bearing distinctions this pins:** (a) TWO same-named `UnifiedBridge` types — ogar's `<P: PortSpec>` resolver (NO authorize method) vs callcenter's `<B: NamespaceBridge>` authorize/audit wrapper — never composed, and currently UNCOMPOSABLE (callcenter has zero dep on lance-graph-ogar); classify composition as later hardening, never as a shipped edge. (b) `lance-graph-ogar` is workspace-EXCLUDED — a green parent-workspace build compiles none of it (`--manifest-path` required; 83 tests green there). (c) The audit default is `NoopAuditSink` + `SuperDomain::Unknown`/salt 0 with swallowed emit errors, and `MEDCARE_AUDIT_SALT` is read fail-open then discarded — for a HIPAA-regime domain, "configured" vs "recording" is indistinguishable (P0 candidate). (d) Retention 2190d (`StepDomain::Medcare`, tested) vs 3650d (manifest, dead) is the DOCUMENTED open question OQ-2 (HIPAA vs BMV-Ä §57), not an accident. (e) `Anamnese`/`Ueberweisung` are first-class consumer concepts (97/81 refs) with no port alias / no canonical id at all — a real canon gap, not naming noise. (f) A third, unreconciled `.grok/` lineage (Zone-3 via spear, 2026-05-08) contradicts both. | ||
|
|
||
| **The lesson (generalizes):** when a repo hosts a consumer-shaped scaffold AND the consumer separately pulls the repo's libraries, the scaffold can freeze at birth while every ledger keeps narrating it as "the integration" — dormancy is invisible in prose because infra work *around* the frozen files (branch names, "first-thought loop" commits) keeps the term alive. The falsifier that cut through: per-file `git log` dormancy (81 days = age of the file) + constructor-count (zero) + the two-repo actor-name check. Also: the clone was SHALLOW (5 grafted roots, 11-day window) — `git fetch --unshallow` converted a "between 06-23 and 06-26 per prose" estimate into the exact commit `ddb6c840` two days earlier. **Archaeology on a shallow clone is prose archaeology; unshallow first.** |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
file=".claude/board/EPIPHANIES.md"
rg -n '^## [0-9]{4}-[0-9]{2}-[0-9]{2} — ' "$file"
git diff --unified=3 -- "$file"Repository: AdaWorldAPI/lance-graph
Length of output: 50380
Move the MedCare finding to the newest-first ledgers position.
:2026-08-02 — E-TWO-MEDCARE-LINEAGES… is currently after older entries and before another existing 2026-08-02 item. This board must keep new entries prepended and preserve prior order.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/board/EPIPHANIES.md around lines 871 - 879, Move the entire
`2026-08-02 — E-TWO-MEDCARE-LINEAGES-THE-LIVE-ONE-PULLS-THE-DEAD-ONE-HOSTS-1`
entry to the topmost newest-first ledger position, ahead of all older entries
and the other existing `2026-08-02` item. Preserve the exact entry content and
retain the relative order of every other entry.
Sources: Coding guidelines, Learnings
| let published = match self.open_if_exists().await? { | ||
| None => { | ||
| // Empty store: sealed head is DatasetVersion(0) by convention. | ||
| if base.0 != 0 { | ||
| return Err(WriteFailed(format!( | ||
| "stale base {base:?}: sealed head is DatasetVersion(0) (empty store)" | ||
| ))); | ||
| } | ||
| let reader = RecordBatchIterator::new(vec![Ok(record_batch)], schema); | ||
| let params = WriteParams { | ||
| mode: WriteMode::Create, | ||
| ..Default::default() | ||
| }; | ||
| let ds = Dataset::write(reader, &self.dataset_path, Some(params)) | ||
| .await | ||
| .map_err(|e| WriteFailed(format!("create commit: {e}")))?; | ||
| ds.version().version | ||
| } | ||
| Some(mut ds) => { | ||
| let head = ds.version().version; | ||
| if head != base.0 { | ||
| return Err(WriteFailed(format!( | ||
| "stale base {base:?}: sealed head is DatasetVersion({head})" | ||
| ))); | ||
| } | ||
| let reader = RecordBatchIterator::new(vec![Ok(record_batch)], schema); | ||
| ds.append(reader, None) | ||
| .await | ||
| .map_err(|e| WriteFailed(format!("append commit: {e}")))?; | ||
| ds.version().version | ||
| } | ||
| }; | ||
| if published != base.0 + 1 { | ||
| // The data IS committed at `published` — but the one-writer §I.6 | ||
| // timeline is broken (a foreign writer interleaved). Surface loudly; | ||
| // never let a shifted identity pass as a sealed cycle. | ||
| return Err(WriteFailed(format!( | ||
| "timeline anomaly: committed at DatasetVersion({published}), expected {}", | ||
| base.0 + 1 | ||
| ))); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🌐 Web query:
For Lance 9.0.0 Rust, which Dataset write or append API can make an append conditional on an expected dataset version, so a concurrent writer cannot turn a cycle based on Vn into a successful commit at Vn+2?
💡 Result:
In Lance 9.0.0 (Rust), you can implement a conditional append by using the lower-level Dataset::commit function, which allows you to explicitly provide a read_version parameter [1]. The standard high-level append method (e.g., Dataset::append) generally handles the current dataset version automatically [2], which may not provide the strict conditional control you need to prevent a concurrent writer from evolving the state (e.g., from Vn to Vn+2) before your commit completes. To ensure your write is conditional on a specific expected version (Vn), follow these steps: 1. Read the current version (Vn) of your dataset [3]. 2. Prepare your operation (e.g., an Append operation) with your data [4][5]. 3. Use the Dataset::commit method to finalize the transaction, passing Vn as the read_version argument [1][3]. The signature for Dataset::commit is: pub async fn commit( dest: impl Into<WriteDestination<'_>>, operation: Operation, read_version: Option, store_params: Option, commit_handler: Option<Arc>, session: Arc, enable_v2_manifest_paths: bool) -> Result<>; By explicitly providing read_version = Some(Vn), the commit process enforces optimistic concurrency control [4]. If a concurrent writer has already committed a newer version (Vn+1) to the underlying storage, the Lance commit mechanism will detect the conflict and fail the operation [4][6], preventing your transaction from creating a cycle or incorrectly applying changes based on an stale manifest [4][5]. If you are using the high-level InsertBuilder or append path, you are typically operating within a more abstracted flow that may mask this manual version management [2]. For maximum control over concurrency and version coupling, calling commit directly is the recommended approach [1].
Citations:
- 1: https://docs.rs/lance/latest/lance/dataset/struct.Dataset.html
- 2: https://github.com/lancedb/lance/blob/119f87b3/rust/lance/src/dataset.rs
- 3: https://lance.org/guide/distributed_write/
- 4: http://lance.org/format/table/transaction/
- 5: https://github.com/lancedb/lance/blob/119f87b3/rust/lance/src/dataset/transaction.rs
- 6: Optimistic concurrency control lance-format/lance#951
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate relevant files =="
git ls-files | rg 'crates/(lance-graph/src/graph/cycle_sink\.rs|lance-graph-supervisor/src/cycle_driver\.rs)$' || true
echo
echo "== cycle_sink outline =="
ast-grep outline crates/lance-graph/src/graph/cycle_sink.rs --view compact || true
echo
echo "== relevant cycle_sink lines =="
sed -n '330,430p' crates/lance-graph/src/graph/cycle_sink.rs
echo
echo "== cycle_driver relevant imports/types/calls =="
rg -n "run_cycle|write|WriteFailed|Determine|DatasetVersion|sealed|SealFailure|recover" crates/lance-graph-supervisor/src/cycle_driver.rs | head -n 200Repository: AdaWorldAPI/lance-graph
Length of output: 18330
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== cycle_sink constructor and append path outline =="
ast-grep outline crates/lance-graph/src/graph/*cycle_sink*.rs --view signatures
echo
echo "== cycle_sink full file (focused around methods) =="
wc -l crates/lance-graph/src/graph/cycle_sink.rs
sed -n '1,180p' crates/lance-graph/src/graph/cycle_sink.rs
echo
echo "== cycle_driver run_cycle and run_cycle tests =="
sed -n '318,358p' crates/lance-graph-supervisor/src/cycle_driver.rs
sed -n '486,530p' crates/lance-graph-supervisor/src/cycle_driver.rs
sed -n '984,1120p' crates/lance-graph-supervisor/src/cycle_driver.rsRepository: AdaWorldAPI/lance-graph
Length of output: 19532
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== run_cycle error variants and callers =="
rg -n "enum CycleError|CycleError\::|seal_cycle\(|recover_fleet\(|recover_and_apply\(" crates/lance-graph-supervisor -g '*.rs' | head -n 200
echo
echo "== run_circular/fleet lifecycle callers =="
for f in $(git ls-files crates/lance-graph-supervisor/src | rg '\.rs$'); do
echo "--- $f"
rg -n "run_cycle|seal_cycle|recover_fleet|CycleError" "$f" || true
done
echo
echo "== persist_sink traits =="
rg -n "struct|enum|trait|persist_cycle|commit_cycle|WriteFailed|DetachedCycleBatch" crates/*/src crates/lance-graph-supervisor/src -g '*.rs' | head -n 240
echo
echo "== Lance dependency version =="
rg -n 'name = "lance"\n|lance ="|lance = "' -g 'Cargo.toml' -C 3 .Repository: AdaWorldAPI/lance-graph
Length of output: 28182
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== cycle_driver error enum =="
sed -n '136,180p' crates/lance-graph-supervisor/src/cycle_driver.rs
sed -n '467,518p' crates/lance-graph-supervisor/src/cycle_driver.rs
echo
echo "== persist_sink definitions/usages =="
rg -n "pub (trait WalSink|enum WriteFailed|struct DetachedCycleBatch|struct PersistError|fn persist_cycle|fn commit_cycle)|WriteFailed|persist_cycle|commit_cycle" crates/lance-graph-planner crates/lance-graph src crates/lance-graph-supervisor -g '*.rs'
echo
echo "== Lance dependency declarations =="
rg --multiline -- '^\s*name = "lance"\n|^\s*name = "lance-graph"\n|^\s*name = "lance-graph-contract"\n|^\s*name = "lance-graph-supervisor"\n' Cargo.lock -g 'Cargo.lock' -C 2 || true
echo
echo "== deterministic verifier for race window and row sealing =="
python3 - <<'PY'
from dataclasses import dataclass
from enum import auto, Enum
class State(Enum):
base_read = auto()
foreign_commit_n_plus_1 = auto()
append_starts_after_foreign_commit = auto()
append_commits_unconditionally = auto()
post_check_mismatch = auto()
`@dataclass`(equivalent=True)
class Run:
state: State
trace = []
for state in State:
run = Run(state)
trace.append(run)
if state is State.base_read:
base = 0
head = 0
elif state is State.foreign_commit_n_plus_1:
head = base + 1
elif state is State.append_starts_after_foreign_commit:
base_for_append = 0
head_for_append = base + 1
elif state is State.append_commits_unconditionally:
published = head_for_append + 1
elif state is State.post_check_mismatch:
expected = base + 1
mismatch_publish = published
row_base_version = base
row_derived_version = row_base_version + 1
print({
"race_trace": [run.state.name for run in trace],
"published_after_race": mismatch_publish,
"row_derives": row_derived_version,
"expected": expected,
})
PYRepository: AdaWorldAPI/lance-graph
Length of output: 21275
Fail the cycle commit before appending an interleaved version.
A foreign writer can publish DatasetVersion(base.0 + 1) after this code reads the current head. ds.append() then commits under the new manifest and can return DatasetVersion(base.0 + 2), but the rows are still emitted with sealed_version() == base + 1. run_cycle then classifies this as CycleError::Seal and treats it as uncommitted regeneration. Make the append expected-version conditional, or return a separate committed-anomaly result including the actual published version and route it to committed-history handling.
📍 Affects 2 files
crates/lance-graph/src/graph/cycle_sink.rs#L369-L408(this comment)crates/lance-graph-supervisor/src/cycle_driver.rs#L26-L50crates/lance-graph-supervisor/src/cycle_driver.rs#L1016-L1117
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/lance-graph/src/graph/cycle_sink.rs` around lines 369 - 408, Prevent
interleaved cycle commits from appending under a shifted DatasetVersion: update
the cycle sink commit path at crates/lance-graph/src/graph/cycle_sink.rs:369-408
to use an expected-version conditional append, or return a distinct
committed-anomaly result containing the actual published version. Update
crates/lance-graph-supervisor/src/cycle_driver.rs:26-50 and :1016-1117 so this
result is routed through committed-history handling rather than classified as an
uncommitted CycleError::Seal; preserve the actual published version when
reporting the anomaly.
…alesced image, 512-byte witness gate, projected reads Addresses the #911 review findings (Codex P1x2/P2, CodeRabbit): - Fence made effective even post-publication: Lance has no expected- version conditional append (the rebase runs even on a single-attempt Append commit; strict mode is Overwrite-only — measured in lance-9.0.0/src/io/commit.rs), so a foreign interleaved writer can land the batch at base+2. Detecting that now triggers a compensating official Dataset::delete scoped to exactly this cycle's rows (the cycle id is an unsealed identity, so the predicate is exact) BEFORE the retryable WriteFailed returns — 'write failed' is true at the visible head, the driver's regenerate-from-Vn contract stays sound, and no rows survive under a shifted sealed_version identity. The one manual-reconciliation corner (the compensating delete itself fails) names the orphaned version explicitly. - The coalesced image is now DURABLE: kind=2 rows (row -> final payload after the per-row fold) persist in the same atomic commit as the landings, and LanceCycleSink::scan_image(cycle) reads a sealed cycle's coherent end-state (projected row+payload under the kind+cycle predicate) while scan_sealed keeps the per-cast history. - Witness payloads are gated to exactly EPISODIC_WITNESS_BYTES = 512 (the canonical key(16)|edges(16)|value(480) node stride) before anything durable happens; malformed rows refuse the whole cycle. - versions() projects only cycle+base_version under kind=0 and scan_sealed pushes kind=1 into the scan, so the coarse-timeline lookup never materializes a witness payload. - Board entries updated to the corrected contract; MD040 text fences in the medcare plan doc; INTEGRATION_PLANS 2026-08-02 entry moved ahead of its same-date sibling (EPIPHANIES placement already coherent). 8 reopened-dataset tests green (2 new: malformed-payload refusal with nothing written; durable per-cycle image with later-stream-position- wins and intact cast history). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_54530d94-a5ed-414a-9b5a-c142b85054ee) |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/lance-graph/src/graph/cycle_sink.rs`:
- Around line 517-547: The compensating delete in the post-publication branch
advances the physical dataset version, so treating the append as an uncommitted
retryable failure leaves sealed history inconsistent. Update the `ds.delete`
compensation and subsequent commit handling to use an expected-version
conditional, or record the actual published version as committed history; never
regenerate from the recovered sealed predecessor after the append was externally
visible.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3351047e-9880-410c-addc-641bc5c60ae0
📒 Files selected for processing (5)
.claude/board/INTEGRATION_PLANS.md.claude/board/LATEST_STATE.md.claude/board/PR_ARC_INVENTORY.md.claude/plans/medcare-consumer-pull-thinking-proof-v1.mdcrates/lance-graph/src/graph/cycle_sink.rs
🚧 Files skipped from review as they are similar to previous changes (4)
- .claude/board/PR_ARC_INVENTORY.md
- .claude/plans/medcare-consumer-pull-thinking-proof-v1.md
- .claude/board/LATEST_STATE.md
- .claude/board/INTEGRATION_PLANS.md
| let published = ds.version().version; | ||
| if published != base.0 + 1 { | ||
| // A foreign writer interleaved between the fence check and | ||
| // the commit; Lance's append rebase landed this batch at a | ||
| // shifted version. Make the fence effective retroactively: | ||
| // remove exactly this cycle's just-appended rows, then | ||
| // report the (now-true) retryable failure. The cycle id is | ||
| // an unsealed identity at this point — no earlier sealed | ||
| // rows can carry it — so the predicate is exact. | ||
| let compensate = ds | ||
| .delete(&format!( | ||
| "cycle = {} AND base_version = {}", | ||
| batch.frame.cycle.0, base.0 | ||
| )) | ||
| .await; | ||
| return Err(match compensate { | ||
| Ok(_) => WriteFailed(format!( | ||
| "fenced post-publication: a foreign writer moved the head past \ | ||
| {base:?} (batch landed at DatasetVersion({published})); the \ | ||
| cycle's rows were deleted again — nothing of cycle {} is \ | ||
| visible; regenerate from the current sealed head", | ||
| batch.frame.cycle.0 | ||
| )), | ||
| Err(e) => WriteFailed(format!( | ||
| "TIMELINE ANOMALY, MANUAL RECONCILIATION REQUIRED: cycle {} \ | ||
| committed at DatasetVersion({published}) (expected {}), and the \ | ||
| compensating delete failed: {e}", | ||
| batch.frame.cycle.0, | ||
| base.0 + 1 | ||
| )), | ||
| }); |
There was a problem hiding this comment.
Do not treat a published append as an uncommitted failure.
The compensating Dataset::delete publishes another dataset version. If the append lands at base + 2, the delete advances the physical head again. versions() still reports only frame-derived versions, so a subsequent commit using the recovered sealed predecessor fails the head != base.0 check at Line 508.
Use an expected-version commit, or return the actual published version as committed history. Do not regenerate from a cycle that was externally visible.
For Lance Rust version 9.0.0, does `Dataset::delete` create a new dataset version? Which official API supports a write conditional on an expected dataset version or read version?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/lance-graph/src/graph/cycle_sink.rs` around lines 517 - 547, The
compensating delete in the post-publication branch advances the physical dataset
version, so treating the append as an uncommitted retryable failure leaves
sealed history inconsistent. Update the `ds.delete` compensation and subsequent
commit handling to use an expected-version conditional, or record the actual
published version as committed history; never regenerate from the recovered
sealed predecessor after the append was externally visible.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ba4f06cd5c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let compensate = ds | ||
| .delete(&format!( | ||
| "cycle = {} AND base_version = {}", | ||
| batch.frame.cycle.0, base.0 | ||
| )) |
There was a problem hiding this comment.
Do not delete the competing writer's cycle
When two writers derive the same cycle ID from the same sealed base, the first writer's successful rows and this writer's rebased rows both satisfy this predicate. The compensating delete therefore removes the winning writer's already-acknowledged frame, landings, and image as well, even though that writer may already have applied its transitions; after restart, its durable recovery history is gone. Fresh evidence beyond the earlier publication finding is that the new compensation path identifies rows only by the shared (cycle, base_version) rather than by the exact appended fragments or transaction.
Useful? React with 👍 / 👎.
| for s in &batch.landings { | ||
| if s.payload.len() != EPISODIC_WITNESS_BYTES { | ||
| return Err(WriteFailed(format!( | ||
| "landing payload for row {} is {} bytes, expected the canonical {EPISODIC_WITNESS_BYTES}", | ||
| s.row, | ||
| s.payload.len() | ||
| ))); |
There was a problem hiding this comment.
Preserve a valid payload when restaging held moves
When an owner produces multiple intent moves, restage_held explicitly stages each deferred move with Vec::new() (cycle_driver.rs:300-307), and the following cycle carries that empty payload into this check. The concrete sink then rejects the cycle because every landing must be exactly 512 bytes, so the documented held-intent progression can never commit through LanceCycleSink; restaging must retain or synthesize a valid row payload, or intent-only landings must be represented without violating the payload invariant.
Useful? React with 👍 / 👎.
Phase A: artifact-backed commits + the sole owned Lance writer (supersedes the #911 cycle contract)
…t deprecated (#879/#911/#912/#913) Recorded in the RP-SEAL plan header + the pin-ruling EPIPHANIES entry; the workflow script was corrected in place (source map + the two cell briefs naming rustynum); the in-flight independent pass could not be force-stopped in this harness build, so the ruling binds consolidation as a hard filter. Deltalake removal ratified same exchange. Co-Authored-By: Claude <noreply@anthropic.com>
…e.dev debug=0 (#962) * ogar_codebook: sync the ConceptDomain wire-mirror -- Ontology, Blocks, and the C-band The contract's mirror of OGAR's ConceptDomain ended at Geo (0x0F) while OGAR carries Ontology (0x03, populated by the DisMech 0x0333 mints), Blocks (0x17), and the C-band JavaRuntime/Analytics/BinaryLifting (0xC0/0xC1/0xC4 -- the altitude ruling, OGAR #276+#277; 0xC0 is Panama FFM alone, Valhalla being a property of the C0 vocabulary rather than an addressable concept). Both sides' docs demand they update together; this is the catch-up, found by the lance-graph-java session and verified independently by the ruff/R2IL session at db488f5, with ownership of the sync explicitly handed here so the ruff arc's PR3 rebases trivially. The real finding is WHY the drift guard never fired: domains_agree + assert_codebook_parity only walk ids that carry concept rows, so a reserved-EMPTY domain added to one enum but not the other is invisible to a content walk. Proven live -- the first disable-run (dropping the new BinaryLifting pair from domains_agree) stayed GREEN. Repaired with reserved_empty_domains_agree_across_the_mirror: one id per new domain, the populated 0x0333, the deliberate 0xC2-0xC3 gap pinned like OGAR's own 0x10-0x16, the band edges, and the 0x0C/0xC0 digit-swap two-sided. Both disable-runs (bridge pair dropped; contract arm dropped) now go red on exactly that test; the contract's own domain_routes_on_high_byte independently catches the arm removal. Gates: lance-graph-contract 1162/1162 + doctests, clippy --all-targets clean; lance-graph-ogar (workspace-excluded, tested via manifest-path) 64/64 incl. assert_codebook_parity green -- content parity holds, this was domain-level drift only. The crate's 11 pre-existing clippy warnings are measured identical with this diff stashed and left untouched. Board: EPIPHANIES E-OGAR-CODEBOOK-MIRROR-DOMAIN-DRIFT-SYNCED-1 prepended in the same commit, per the board-hygiene rule. * lotus Phase 0/1: frontier audit + F-ORD-REAL pre-registered falsifier Research charter deliverables 1-2 (no fix included, by design): - docs/lotus/LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology across the write path, persistence capability, placement/comma prior art, and frontier visibility, every statement graded VERIFIED / INFERENCE / HYPOTHESIS / BLOCKER. Headline findings: content_hash folds arrival-minted stream_position values into batch_hash (persist_sink.rs:414), contradicting DetachedCycleBatch's own order-independence doc; the seal is O(batch bytes) x3 passes with the batch resident up to 3x at seal; SweepSlot's caller-supplied semantic order-key contract vs collect_casts' arrival mint; the lance crate source is absent from this sandbox (prepared-artifact capability audit BLOCKED). Section 6 answers the permeability question: the cycle does not become permeable, it becomes thin — trailing-publication pipelining (Regime A) needs no epistemic weakening; rung-qualified frontier visibility (temporal.rs EpistemicMode ladder) stratifies rather than reopens the retired race; texts stay linear, tiles get derived placement, resolved per class. - docs/lotus/F-ORD-REAL-FALSIFIER.md — the defect mechanism in four verified steps + the test design (perturb the process that creates the key: permute cast() call order, never post-mint slots). - cycle_driver.rs tests: f_ord_real_defect_pin_... (GREEN, two-sided — anti-vacuity proves the perturbation reaches the key mint; semantic set + image pinned arrival-independent; batch_hash pinned arrival-DEPENDENT; fails loudly when a fix lands) and f_ord_real_publication_identity_... (#[ignore]d RED falsifier — the desired property, red under --ignored on the real chain). Gates: cargo test -p lance-graph-supervisor --features cycle-driver 28 passed / 1 ignored (+ suites green); RED verified red under --ignored; fmt clean; clippy adds zero new warnings (8 pre-existing recover_fleet lints only visible under this non-default feature). Board: EPIPHANIES E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1; STATUS_BOARD lotus-seal-fractal-commit-frontier section (D-LOTUS-1..9). Co-Authored-By: Claude <noreply@anthropic.com> * board: #961 arc entry + LATEST_STATE + D-LOTUS-1/2 Shipped flips Co-Authored-By: Claude <noreply@anthropic.com> * operator pin ruling: DF 54.1 only — remove broken delta feature; RP-SEAL charter + boards Ruling (E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1): lance 9 / lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, always, across AdaWorldAPI forks, usually via [patch -> upstream repository git]. Measured before acting: the only DF-53 source was deltalake 0.32.4 (^53.1.0) behind the already-non-default, already-broken delta feature; registry check shows no DF-54 deltalake exists. Removed: delta feature, deltalake + url optional deps, DeltaTableReader (module docs + Cargo.toml carry dated removal notes; DataSourceFormat::Delta stays as a catalog tag). Post-removal Cargo.lock: exactly ONE datafusion = 54.1.0, zero deltalake entries. cargo check -p lance-graph green; fmt clean; full suite running as the PR gate. Docker pins surveyed: root + avx512 = Rust 1.97.1 + protobuf-compiler, no delta references — removal is docker-safe (stale flag noted: crates/symbiont/Dockerfile rust:1.95). Also: RP-SEAL research charter committed as .claude/plans/erasure-seals-compaction-research-v1.md (15-researcher program, independent pass dispatched as background workflow wf_ca974718-1b4; adversaries strongest-tier, builders/scouts grindwork tier); INTEGRATION_PLANS + STATUS_BOARD entries; D-LOTUS-6 BLOCKER lifted (operator-sanctioned upstream-git source consult; exact v9.0.0 tag on disk matching the lock checksum) with the audit carrying the dated lift note; CLAUDE.md 'BOTH MAJORS ARE REQUIRED' note superseded in place. Co-Authored-By: Claude <noreply@anthropic.com> * scope pivot (operator): rustynum struck — everything ndarray; symbiont deprecated (#879/#911/#912/#913) Recorded in the RP-SEAL plan header + the pin-ruling EPIPHANIES entry; the workflow script was corrected in place (source map + the two cell briefs naming rustynum); the in-flight independent pass could not be force-stopped in this harness build, so the ruling binds consolidation as a hard filter. Deltalake removal ratified same exchange. Co-Authored-By: Claude <noreply@anthropic.com> * correct the delta-removal notes: delta-rs MAIN is already DF 54 + arrow 58 Operator-pointed, verified from delta-io/delta-rs main Cargo.toml (datafusion = 54.0.0, arrow = 58): only the crates.io releases top out at DF 53. Restoration is available now via a git-pin on upstream + a reader refactor to the current builder API — as its own deliberate PR if a consumer needs Delta. The removal itself stands on the need ruling ('we don't need deltalake'), not on availability; notes in Cargo.toml, CLAUDE.md, and the EPIPHANIES entry corrected accordingly. Co-Authored-By: Claude <noreply@anthropic.com> * profile.dev debug=0 — smaller/faster builds (operator ruling) Full-debuginfo test binaries grew target/ to 17 GB and SIGBUS'd the linker on a full disk this session; debug=0 shrinks the lance/datafusion-stack test binaries ~an order of magnitude. Matches the gate already running with CARGO_PROFILE_DEV_DEBUG=0 (same resolved profile, cache reuse). line-tables-only noted as the fallback if line-numbered backtraces are ever needed. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
* ogar_codebook: sync the ConceptDomain wire-mirror -- Ontology, Blocks, and the C-band The contract's mirror of OGAR's ConceptDomain ended at Geo (0x0F) while OGAR carries Ontology (0x03, populated by the DisMech 0x0333 mints), Blocks (0x17), and the C-band JavaRuntime/Analytics/BinaryLifting (0xC0/0xC1/0xC4 -- the altitude ruling, OGAR #276+#277; 0xC0 is Panama FFM alone, Valhalla being a property of the C0 vocabulary rather than an addressable concept). Both sides' docs demand they update together; this is the catch-up, found by the lance-graph-java session and verified independently by the ruff/R2IL session at db488f5, with ownership of the sync explicitly handed here so the ruff arc's PR3 rebases trivially. The real finding is WHY the drift guard never fired: domains_agree + assert_codebook_parity only walk ids that carry concept rows, so a reserved-EMPTY domain added to one enum but not the other is invisible to a content walk. Proven live -- the first disable-run (dropping the new BinaryLifting pair from domains_agree) stayed GREEN. Repaired with reserved_empty_domains_agree_across_the_mirror: one id per new domain, the populated 0x0333, the deliberate 0xC2-0xC3 gap pinned like OGAR's own 0x10-0x16, the band edges, and the 0x0C/0xC0 digit-swap two-sided. Both disable-runs (bridge pair dropped; contract arm dropped) now go red on exactly that test; the contract's own domain_routes_on_high_byte independently catches the arm removal. Gates: lance-graph-contract 1162/1162 + doctests, clippy --all-targets clean; lance-graph-ogar (workspace-excluded, tested via manifest-path) 64/64 incl. assert_codebook_parity green -- content parity holds, this was domain-level drift only. The crate's 11 pre-existing clippy warnings are measured identical with this diff stashed and left untouched. Board: EPIPHANIES E-OGAR-CODEBOOK-MIRROR-DOMAIN-DRIFT-SYNCED-1 prepended in the same commit, per the board-hygiene rule. * lotus Phase 0/1: frontier audit + F-ORD-REAL pre-registered falsifier Research charter deliverables 1-2 (no fix included, by design): - docs/lotus/LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology across the write path, persistence capability, placement/comma prior art, and frontier visibility, every statement graded VERIFIED / INFERENCE / HYPOTHESIS / BLOCKER. Headline findings: content_hash folds arrival-minted stream_position values into batch_hash (persist_sink.rs:414), contradicting DetachedCycleBatch's own order-independence doc; the seal is O(batch bytes) x3 passes with the batch resident up to 3x at seal; SweepSlot's caller-supplied semantic order-key contract vs collect_casts' arrival mint; the lance crate source is absent from this sandbox (prepared-artifact capability audit BLOCKED). Section 6 answers the permeability question: the cycle does not become permeable, it becomes thin — trailing-publication pipelining (Regime A) needs no epistemic weakening; rung-qualified frontier visibility (temporal.rs EpistemicMode ladder) stratifies rather than reopens the retired race; texts stay linear, tiles get derived placement, resolved per class. - docs/lotus/F-ORD-REAL-FALSIFIER.md — the defect mechanism in four verified steps + the test design (perturb the process that creates the key: permute cast() call order, never post-mint slots). - cycle_driver.rs tests: f_ord_real_defect_pin_... (GREEN, two-sided — anti-vacuity proves the perturbation reaches the key mint; semantic set + image pinned arrival-independent; batch_hash pinned arrival-DEPENDENT; fails loudly when a fix lands) and f_ord_real_publication_identity_... (#[ignore]d RED falsifier — the desired property, red under --ignored on the real chain). Gates: cargo test -p lance-graph-supervisor --features cycle-driver 28 passed / 1 ignored (+ suites green); RED verified red under --ignored; fmt clean; clippy adds zero new warnings (8 pre-existing recover_fleet lints only visible under this non-default feature). Board: EPIPHANIES E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1; STATUS_BOARD lotus-seal-fractal-commit-frontier section (D-LOTUS-1..9). Co-Authored-By: Claude <noreply@anthropic.com> * board: #961 arc entry + LATEST_STATE + D-LOTUS-1/2 Shipped flips Co-Authored-By: Claude <noreply@anthropic.com> * operator pin ruling: DF 54.1 only — remove broken delta feature; RP-SEAL charter + boards Ruling (E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1): lance 9 / lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, always, across AdaWorldAPI forks, usually via [patch -> upstream repository git]. Measured before acting: the only DF-53 source was deltalake 0.32.4 (^53.1.0) behind the already-non-default, already-broken delta feature; registry check shows no DF-54 deltalake exists. Removed: delta feature, deltalake + url optional deps, DeltaTableReader (module docs + Cargo.toml carry dated removal notes; DataSourceFormat::Delta stays as a catalog tag). Post-removal Cargo.lock: exactly ONE datafusion = 54.1.0, zero deltalake entries. cargo check -p lance-graph green; fmt clean; full suite running as the PR gate. Docker pins surveyed: root + avx512 = Rust 1.97.1 + protobuf-compiler, no delta references — removal is docker-safe (stale flag noted: crates/symbiont/Dockerfile rust:1.95). Also: RP-SEAL research charter committed as .claude/plans/erasure-seals-compaction-research-v1.md (15-researcher program, independent pass dispatched as background workflow wf_ca974718-1b4; adversaries strongest-tier, builders/scouts grindwork tier); INTEGRATION_PLANS + STATUS_BOARD entries; D-LOTUS-6 BLOCKER lifted (operator-sanctioned upstream-git source consult; exact v9.0.0 tag on disk matching the lock checksum) with the audit carrying the dated lift note; CLAUDE.md 'BOTH MAJORS ARE REQUIRED' note superseded in place. Co-Authored-By: Claude <noreply@anthropic.com> * scope pivot (operator): rustynum struck — everything ndarray; symbiont deprecated (#879/#911/#912/#913) Recorded in the RP-SEAL plan header + the pin-ruling EPIPHANIES entry; the workflow script was corrected in place (source map + the two cell briefs naming rustynum); the in-flight independent pass could not be force-stopped in this harness build, so the ruling binds consolidation as a hard filter. Deltalake removal ratified same exchange. Co-Authored-By: Claude <noreply@anthropic.com> * correct the delta-removal notes: delta-rs MAIN is already DF 54 + arrow 58 Operator-pointed, verified from delta-io/delta-rs main Cargo.toml (datafusion = 54.0.0, arrow = 58): only the crates.io releases top out at DF 53. Restoration is available now via a git-pin on upstream + a reader refactor to the current builder API — as its own deliberate PR if a consumer needs Delta. The removal itself stands on the need ruling ('we don't need deltalake'), not on availability; notes in Cargo.toml, CLAUDE.md, and the EPIPHANIES entry corrected accordingly. Co-Authored-By: Claude <noreply@anthropic.com> * profile.dev debug=0 — smaller/faster builds (operator ruling) Full-debuginfo test binaries grew target/ to 17 GB and SIGBUS'd the linker on a full disk this session; debug=0 shrinks the lance/datafusion-stack test binaries ~an order of magnitude. Matches the gate already running with CARGO_PROFILE_DEV_DEBUG=0 (same resolved profile, cache reuse). line-tables-only noted as the fallback if line-numbered backtraces are ever needed. Co-Authored-By: Claude <noreply@anthropic.com> * drift cleanup: rustynum/symbiont no-go applied to live surfaces - lance-graph-cognitive: rustynum_accel was a name-only shim (zero rustynum dependency); renamed simd_accel (module + file + 11 call sites), rerouted through the sanctioned ndarray::simd re-export instead of ndarray::hpc::bitwise. Default compile green; the wip feature's pre-existing not-yet-compiling state has zero errors naming the rename. - docs/lotus audit: symbiont::domino::morton4 candidate row struck (census history only). - CLAUDE.md: symbiont removed from the binding-consumer lists via dated annotations. - EPIPHANIES: cleanup recorded in the pin-ruling entry. Audit also confirmed NO redo needed: A1's research report is clean (0 citations), the Java/Panama arc never touched either (ndarray::simd enforced), board history stays append-only. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
… core, compaction = optional economics (#964) * ogar_codebook: sync the ConceptDomain wire-mirror -- Ontology, Blocks, and the C-band The contract's mirror of OGAR's ConceptDomain ended at Geo (0x0F) while OGAR carries Ontology (0x03, populated by the DisMech 0x0333 mints), Blocks (0x17), and the C-band JavaRuntime/Analytics/BinaryLifting (0xC0/0xC1/0xC4 -- the altitude ruling, OGAR #276+#277; 0xC0 is Panama FFM alone, Valhalla being a property of the C0 vocabulary rather than an addressable concept). Both sides' docs demand they update together; this is the catch-up, found by the lance-graph-java session and verified independently by the ruff/R2IL session at db488f5, with ownership of the sync explicitly handed here so the ruff arc's PR3 rebases trivially. The real finding is WHY the drift guard never fired: domains_agree + assert_codebook_parity only walk ids that carry concept rows, so a reserved-EMPTY domain added to one enum but not the other is invisible to a content walk. Proven live -- the first disable-run (dropping the new BinaryLifting pair from domains_agree) stayed GREEN. Repaired with reserved_empty_domains_agree_across_the_mirror: one id per new domain, the populated 0x0333, the deliberate 0xC2-0xC3 gap pinned like OGAR's own 0x10-0x16, the band edges, and the 0x0C/0xC0 digit-swap two-sided. Both disable-runs (bridge pair dropped; contract arm dropped) now go red on exactly that test; the contract's own domain_routes_on_high_byte independently catches the arm removal. Gates: lance-graph-contract 1162/1162 + doctests, clippy --all-targets clean; lance-graph-ogar (workspace-excluded, tested via manifest-path) 64/64 incl. assert_codebook_parity green -- content parity holds, this was domain-level drift only. The crate's 11 pre-existing clippy warnings are measured identical with this diff stashed and left untouched. Board: EPIPHANIES E-OGAR-CODEBOOK-MIRROR-DOMAIN-DRIFT-SYNCED-1 prepended in the same commit, per the board-hygiene rule. * lotus Phase 0/1: frontier audit + F-ORD-REAL pre-registered falsifier Research charter deliverables 1-2 (no fix included, by design): - docs/lotus/LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology across the write path, persistence capability, placement/comma prior art, and frontier visibility, every statement graded VERIFIED / INFERENCE / HYPOTHESIS / BLOCKER. Headline findings: content_hash folds arrival-minted stream_position values into batch_hash (persist_sink.rs:414), contradicting DetachedCycleBatch's own order-independence doc; the seal is O(batch bytes) x3 passes with the batch resident up to 3x at seal; SweepSlot's caller-supplied semantic order-key contract vs collect_casts' arrival mint; the lance crate source is absent from this sandbox (prepared-artifact capability audit BLOCKED). Section 6 answers the permeability question: the cycle does not become permeable, it becomes thin — trailing-publication pipelining (Regime A) needs no epistemic weakening; rung-qualified frontier visibility (temporal.rs EpistemicMode ladder) stratifies rather than reopens the retired race; texts stay linear, tiles get derived placement, resolved per class. - docs/lotus/F-ORD-REAL-FALSIFIER.md — the defect mechanism in four verified steps + the test design (perturb the process that creates the key: permute cast() call order, never post-mint slots). - cycle_driver.rs tests: f_ord_real_defect_pin_... (GREEN, two-sided — anti-vacuity proves the perturbation reaches the key mint; semantic set + image pinned arrival-independent; batch_hash pinned arrival-DEPENDENT; fails loudly when a fix lands) and f_ord_real_publication_identity_... (#[ignore]d RED falsifier — the desired property, red under --ignored on the real chain). Gates: cargo test -p lance-graph-supervisor --features cycle-driver 28 passed / 1 ignored (+ suites green); RED verified red under --ignored; fmt clean; clippy adds zero new warnings (8 pre-existing recover_fleet lints only visible under this non-default feature). Board: EPIPHANIES E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1; STATUS_BOARD lotus-seal-fractal-commit-frontier section (D-LOTUS-1..9). Co-Authored-By: Claude <noreply@anthropic.com> * board: #961 arc entry + LATEST_STATE + D-LOTUS-1/2 Shipped flips Co-Authored-By: Claude <noreply@anthropic.com> * operator pin ruling: DF 54.1 only — remove broken delta feature; RP-SEAL charter + boards Ruling (E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1): lance 9 / lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, always, across AdaWorldAPI forks, usually via [patch -> upstream repository git]. Measured before acting: the only DF-53 source was deltalake 0.32.4 (^53.1.0) behind the already-non-default, already-broken delta feature; registry check shows no DF-54 deltalake exists. Removed: delta feature, deltalake + url optional deps, DeltaTableReader (module docs + Cargo.toml carry dated removal notes; DataSourceFormat::Delta stays as a catalog tag). Post-removal Cargo.lock: exactly ONE datafusion = 54.1.0, zero deltalake entries. cargo check -p lance-graph green; fmt clean; full suite running as the PR gate. Docker pins surveyed: root + avx512 = Rust 1.97.1 + protobuf-compiler, no delta references — removal is docker-safe (stale flag noted: crates/symbiont/Dockerfile rust:1.95). Also: RP-SEAL research charter committed as .claude/plans/erasure-seals-compaction-research-v1.md (15-researcher program, independent pass dispatched as background workflow wf_ca974718-1b4; adversaries strongest-tier, builders/scouts grindwork tier); INTEGRATION_PLANS + STATUS_BOARD entries; D-LOTUS-6 BLOCKER lifted (operator-sanctioned upstream-git source consult; exact v9.0.0 tag on disk matching the lock checksum) with the audit carrying the dated lift note; CLAUDE.md 'BOTH MAJORS ARE REQUIRED' note superseded in place. Co-Authored-By: Claude <noreply@anthropic.com> * scope pivot (operator): rustynum struck — everything ndarray; symbiont deprecated (#879/#911/#912/#913) Recorded in the RP-SEAL plan header + the pin-ruling EPIPHANIES entry; the workflow script was corrected in place (source map + the two cell briefs naming rustynum); the in-flight independent pass could not be force-stopped in this harness build, so the ruling binds consolidation as a hard filter. Deltalake removal ratified same exchange. Co-Authored-By: Claude <noreply@anthropic.com> * correct the delta-removal notes: delta-rs MAIN is already DF 54 + arrow 58 Operator-pointed, verified from delta-io/delta-rs main Cargo.toml (datafusion = 54.0.0, arrow = 58): only the crates.io releases top out at DF 53. Restoration is available now via a git-pin on upstream + a reader refactor to the current builder API — as its own deliberate PR if a consumer needs Delta. The removal itself stands on the need ruling ('we don't need deltalake'), not on availability; notes in Cargo.toml, CLAUDE.md, and the EPIPHANIES entry corrected accordingly. Co-Authored-By: Claude <noreply@anthropic.com> * profile.dev debug=0 — smaller/faster builds (operator ruling) Full-debuginfo test binaries grew target/ to 17 GB and SIGBUS'd the linker on a full disk this session; debug=0 shrinks the lance/datafusion-stack test binaries ~an order of magnitude. Matches the gate already running with CARGO_PROFILE_DEV_DEBUG=0 (same resolved profile, cache reuse). line-tables-only noted as the fallback if line-numbered backtraces are ever needed. Co-Authored-By: Claude <noreply@anthropic.com> * drift cleanup: rustynum/symbiont no-go applied to live surfaces - lance-graph-cognitive: rustynum_accel was a name-only shim (zero rustynum dependency); renamed simd_accel (module + file + 11 call sites), rerouted through the sanctioned ndarray::simd re-export instead of ndarray::hpc::bitwise. Default compile green; the wip feature's pre-existing not-yet-compiling state has zero errors naming the rename. - docs/lotus audit: symbiont::domino::morton4 candidate row struck (census history only). - CLAUDE.md: symbiont removed from the binding-consumer lists via dated annotations. - EPIPHANIES: cleanup recorded in the pin-ruling entry. Audit also confirmed NO redo needed: A1's research report is clean (0 citations), the Java/Panama arc never touched either (ndarray::simd enforced), board history stays append-only. Co-Authored-By: Claude <noreply@anthropic.com> * RP-SEAL pass 1 consolidated: evidence matrix M1-M15, tiering, Appendix H 15-researcher independent pass complete (15/15, 0 errors, no cross-talk). Consolidation per charter §12-§13: docs/lotus/RP-SEAL-CONSOLIDATION-PASS1.md - DELETED: phase/comma coefficient schedule (3 independent routes), the unconditional SFC default, 'compaction = one publication', the naive query+repair one-grouping hope - Found: b+1 coalescing amplification (65x at b=64) + non-amortizing FNV seal CPU (E2); 11 order leaks, L2 = replay order IS physical scan order (D2); five defective seal paths wearing strong names (C2/C3); Lance 9.0.0 apparatus complete and unused (5 cells concordant) - Survived: hash + row/column P+Q 64x64 seal baseline; compaction-time layout-key upstream seam; two novel candidates (joint locality question refined by the anti-synergy; STRICT/AWARE/RETRO reader-rung tiers) - Appendix H: 15 reports + orchestrator pre-pass under docs/lotus/rp-seal-v1/ - Scope-pivot filter PASSED (zero rustynum/symbiont-sourced findings) - Board: EPIPHANIES entry, AGENT_LOG entry (one-writer), STATUS_BOARD + plan header flips Co-Authored-By: Claude <noreply@anthropic.com> * §0 operator STORNO: canonical replay coordinates are core; compaction = optional economics PHYSICAL ITERATION ORDER MUST NOT DEFINE SEMANTIC REPLAY ORDER — fixed by canonical replay coordinates/order, never by physical reordering or compaction. Permanent temporal replay additionally requires a retention/tombstone policy under which historical knowledge states stay reconstructible indefinitely. Desired model: arbitrary placement + arbitrary fragmentation + historically addressable tombstones + canonical semantic replay order + current visibility as a projection + historical visibility as a QueryReference projection. Consolidation doc gains §0 with in-place ⊘-marked corrections; Tier 1 reordered (canonical coordinates, retention/tombstone, QueryReference projection fixes); Tier 2/3 layout/compaction items reclassified as the optional storage-economics track; F-NOCOMPACT promoted to the default operating description. EPIPHANIES ruling entry + STATUS_BOARD + plan header amended. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
What
Closes the gap
persist_sinkdeliberately left ("this module builds NO concrete Lance sink — compile+test green ≠ storage proven"):lance_graph::graph::cycle_sink::LanceCycleSinkis a concrete implementation oflance_graph_planner::persist_sink::WalSinkover the official Lance 9 insert path (Dataset::write/Dataset::append— the sameInsertBuildertransaction machinery every Lance writer uses). No bespoke ledger, no acknowledgement protocol, no parallel replay system: Lance's own manifest/version chain IS the WAL.The §I.6 invariant made physical
base(empty store ⇒ headDatasetVersion(0)), else refused with nothing written. Post-commit: the published version must be exactlybase + 1— Lance auto-resolves append-append conflicts, so a foreign interleaved writer surfaces as a loud timeline anomaly instead of silently shifting the cycle ↔ version identity reads derive.versions()andscan_sealed()read the reopened store, andsealed_version = base_version + 1is an identity the commit path verifies, never assumes.DetachedCycleBatch::freezeorder and scanned back withscan_in_order(true)— this sink never sorts on read.Domain-0x09 witness contract (module doc)
The patient SoA at classid domain
0x09is the only place patient reasoning is written to Lance, so the store's schema is witness-focused and maximally rich:payloadcarries the 512-byte canonical EpisodicWitness node (visited ontology addresses, executed crosswalk mappings, exact RO/ontology edge ids, supporting/contradicting/missing observations, NARS truth + confidence, differential branches), whose edges point INTO the immutable domain-0x03ontology address space. A cycle takes ontology immutability for granted for its representation window (base_versionnames the sealed predecessor it read) and never restates ontology content — the sealed versioning is a reflection of the thinking; downstream displays read the sealed version, never a live recomputation.Gates
Dataset::open, never an in-memory echo): seal survives restart; stale base fenced with nothing written (version chain, landings, timeline all checked untouched); sequential cycles chain V1→V2→V3 with strictly-after filtering; a zero-landing cycle advances the timeline only; an empty store reads empty (DatasetNotFoundis a state, not an error); move-nullability + 512-byte payload byte-exact round-trip. The real Lance version chain is cross-checked against the returnedDatasetVersionin-test.cargo clippy -p lance-graph --libclean;cargo fmtclean; gated on the default-onplannerfeature (the trait lives in the optional planner dep).The branch also carries the two pre-existing post-#879 follow-up commits that were already on the remote branch (pre-commit failure contract correction + MedCare plan consolidation), merged with current main (
lance-graph-supervisorre-checked green post-merge).Deferred
The MedCare consumer arc (production
drive_cohort_thoughtscaller, witness-seal of real observation + CURIE/ELK/RO nodes, views reading the sealed version) — next PR, in MedCare-rs.recover_and_applywiring against this sink in a production driver. Object-store URI smoke (only local paths exercised).🤖 Generated with Claude Code
https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp
Generated by Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation