From 799b89fe3c79a50785ed534c4d536dfe8f7526d9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 07:16:43 +0000 Subject: [PATCH 1/8] ogar_codebook: sync the ConceptDomain wire-mirror -- Ontology, Blocks, and the C-band The contract's mirror of OGAR's ConceptDomain ended at Geo (0x0F) while OGAR carries Ontology (0x03, populated by the DisMech 0x0333 mints), Blocks (0x17), and the C-band JavaRuntime/Analytics/BinaryLifting (0xC0/0xC1/0xC4 -- the altitude ruling, OGAR #276+#277; 0xC0 is Panama FFM alone, Valhalla being a property of the C0 vocabulary rather than an addressable concept). Both sides' docs demand they update together; this is the catch-up, found by the lance-graph-java session and verified independently by the ruff/R2IL session at db488f5, with ownership of the sync explicitly handed here so the ruff arc's PR3 rebases trivially. The real finding is WHY the drift guard never fired: domains_agree + assert_codebook_parity only walk ids that carry concept rows, so a reserved-EMPTY domain added to one enum but not the other is invisible to a content walk. Proven live -- the first disable-run (dropping the new BinaryLifting pair from domains_agree) stayed GREEN. Repaired with reserved_empty_domains_agree_across_the_mirror: one id per new domain, the populated 0x0333, the deliberate 0xC2-0xC3 gap pinned like OGAR's own 0x10-0x16, the band edges, and the 0x0C/0xC0 digit-swap two-sided. Both disable-runs (bridge pair dropped; contract arm dropped) now go red on exactly that test; the contract's own domain_routes_on_high_byte independently catches the arm removal. Gates: lance-graph-contract 1162/1162 + doctests, clippy --all-targets clean; lance-graph-ogar (workspace-excluded, tested via manifest-path) 64/64 incl. assert_codebook_parity green -- content parity holds, this was domain-level drift only. The crate's 11 pre-existing clippy warnings are measured identical with this diff stashed and left untouched. Board: EPIPHANIES E-OGAR-CODEBOOK-MIRROR-DOMAIN-DRIFT-SYNCED-1 prepended in the same commit, per the board-hygiene rule. --- .claude/board/EPIPHANIES.md | 36 +++++++ .../lance-graph-contract/src/ogar_codebook.rs | 93 +++++++++++++++++-- crates/lance-graph-ogar/src/lib.rs | 31 ++++++- 3 files changed, 151 insertions(+), 9 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 21d7d595..efe390d4 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,39 @@ +## 2026-08-18 — E-OGAR-CODEBOOK-MIRROR-DOMAIN-DRIFT-SYNCED-1 + +**Status:** FINDING + fix (cross-session: found by the lance-graph-java +session, verified independently by the ruff/R2IL session at `db488f5`, +ownership handed to lance-graph-java's session; this is that sync). + +**The drift.** `lance_graph_contract::ogar_codebook` documents itself as a +wire-compatible mirror of OGAR `ogar_vocab::ConceptDomain` under an +explicit both-sides-update-together drift guard — yet its enum ended at +`Geo` (0x0F) while OGAR carries `Ontology` (0x03, POPULATED since the +DisMech 0x0333 mints, OGAR #275), `Blocks` (0x17, since 2026-08-04), and +the C-band `JavaRuntime`/`Analytics`/`BinaryLifting` (0xC0/0xC1/0xC4, +OGAR #276+#277 — the altitude ruling: the domain byte is stratified by +layer; 0xC0 is **Panama FFM alone**, Valhalla being a property of the C0 +vocabulary, not an addressable concept). + +**Why the guard never fired — the real lesson.** `lance-graph-ogar`'s +`domains_agree` + `assert_codebook_parity` only walk ids that carry +CONCEPT ROWS. A reserved-EMPTY domain added to one enum but not the other +is invisible to a codebook-content walk — the exact class of drift a +DOMAIN mirror exists to catch. Proven live: the first disable-run +(dropping the new `BinaryLifting` pair from `domains_agree`) stayed GREEN +— the pairing was vacuously guarded until this PR added +`reserved_empty_domains_agree_across_the_mirror`, which pins one id per +new/reserved domain, the populated 0x0333, the deliberate 0xC2–0xC3 gap, +the band edges, and the 0x0C/0xC0 digit-swap two-sided. Both disable-runs +(bridge pair dropped; contract arm dropped) now go red on exactly that +test; the contract's own `domain_routes_on_high_byte` independently +catches the arm removal. + +**Scope, stated:** DOMAIN-level sync only. Codebook CONTENT parity was +re-run and is green (`assert_codebook_parity`); the R2IL container-concept +mints under 0xC4 arrive with the ruff arc's PR3 and will rebase trivially +on this. Pre-existing `lance-graph-ogar` clippy warnings (11, measured +identical with this diff stashed) left untouched. + ## 2026-08-17 — E-IDENTITY-QUAD-4X24-RATIFIED-PERMANENT-1 **Status:** RULING `[operator]` — explicit, in-session decision, not a diff --git a/crates/lance-graph-contract/src/ogar_codebook.rs b/crates/lance-graph-contract/src/ogar_codebook.rs index efae8e65..c317c106 100644 --- a/crates/lance-graph-contract/src/ogar_codebook.rs +++ b/crates/lance-graph-contract/src/ogar_codebook.rs @@ -48,6 +48,10 @@ pub enum ConceptDomain { ProjectMgmt, /// `0x02XX` — commerce / billing / ERP (Odoo ↔ OSB). Commerce, + /// `0x03XX` — Ontology (OBO reference vocabulary; plug-and-play, never + /// pulled into ERP consumers). Mirrors OGAR + /// `ogar_vocab::ConceptDomain::Ontology`. + Ontology, /// `0x04XX` — Weather / Atmosphere. Shared forecast and atmospheric cells. Weather, /// `0x07XX` — OSINT (open-source intelligence / Palantir-Gotham). @@ -88,8 +92,33 @@ pub enum ConceptDomain { /// (OGAR canon "256×256 centroid tile", D-BOTHCASC). Mirrors OGAR /// `ogar_vocab::ConceptDomain::Geo`; the parity tests pin `0x0F00 → Geo`. Geo, - /// Any high-byte slot not yet assigned a domain (`0x03XX`, `0x05XX`–`0x06XX`, - /// `0x10XX`+). + /// `0x17XX` — Blocks (the `ogar-loco` low-code substrate's domain: the + /// shared block/call opcode vocabulary; `0x1701`/`0x1702` are loco's node + /// shapes, `0x1717`+ per-frontend palettes). Mirrors OGAR + /// `ogar_vocab::ConceptDomain::Blocks` (reserved 2026-08-04). + Blocks, + /// `0xC0XX` — Java runtime (**Panama FFM alone** — downcalls, segments, + /// lanes, arenas, masks; the managed-runtime membrane over the SoA + /// substrate, the FLOOR of the C-band). Valhalla deliberately has no + /// domain slot: it is integrated as a *property* of the C0 vocabulary + /// (`value record`-ready descriptor types), never an addressable crossing + /// concept — canonical text in OGAR's `JavaRuntime` doc comment + /// (PR #276 + #277). Mirrors `ogar_vocab::ConceptDomain::JavaRuntime`. + JavaRuntime, + /// `0xC1XX` — Analytics (the analyst estate: addressable tabular units + + /// catalog ontology; one shared vocabulary under per-app render + /// prefixes). Mirrors `ogar_vocab::ConceptDomain::Analytics`. + Analytics, + /// `0xC4XX` — Binary lifting (normalized machine-code IR + artifact + /// ontology; Ghidra and r2sleigh are two consumers of one SLEIGH-derived + /// vocabulary; the R2IL container concepts mint here in the ruff PR3 + /// arc, replacing `PROVISIONAL_R2IL_VARNODE = 0x0000`). Mirrors + /// `ogar_vocab::ConceptDomain::BinaryLifting`. + BinaryLifting, + /// Any high-byte slot not yet assigned a domain (`0x05XX`–`0x06XX`, + /// `0x10XX`–`0x16XX`, `0x18XX`–`0xBFXX`, `0xC2XX`–`0xC3XX` — a + /// DELIBERATE gap, pinned like OGAR's own `0x10`–`0x16` — and + /// `0xC5XX`+). Unassigned, } @@ -103,6 +132,7 @@ pub fn canonical_concept_domain(id: u16) -> ConceptDomain { 0x00 => ConceptDomain::Reserved, 0x01 => ConceptDomain::ProjectMgmt, 0x02 => ConceptDomain::Commerce, + 0x03 => ConceptDomain::Ontology, 0x04 => ConceptDomain::Weather, 0x07 => ConceptDomain::Osint, 0x08 => ConceptDomain::Ocr, @@ -113,6 +143,10 @@ pub fn canonical_concept_domain(id: u16) -> ConceptDomain { 0x0D => ConceptDomain::HR, 0x0E => ConceptDomain::Genetics, 0x0F => ConceptDomain::Geo, + 0x17 => ConceptDomain::Blocks, + 0xC0 => ConceptDomain::JavaRuntime, + 0xC1 => ConceptDomain::Analytics, + 0xC4 => ConceptDomain::BinaryLifting, _ => ConceptDomain::Unassigned, } } @@ -671,7 +705,33 @@ mod tests { assert_eq!(canonical_concept_domain(0x0500), ConceptDomain::Unassigned); assert_eq!(canonical_concept_domain(0x0E00), ConceptDomain::Genetics); assert_eq!(canonical_concept_domain(0x0F00), ConceptDomain::Geo); + assert_eq!(canonical_concept_domain(0x0300), ConceptDomain::Ontology); + assert_eq!(canonical_concept_domain(0x0333), ConceptDomain::Ontology); + // Blocks (0x17), with its deliberate 0x10-0x16 gap pinned on both + // sides exactly as OGAR pins it. assert_eq!(canonical_concept_domain(0x1000), ConceptDomain::Unassigned); + assert_eq!(canonical_concept_domain(0x1600), ConceptDomain::Unassigned); + assert_eq!(canonical_concept_domain(0x1701), ConceptDomain::Blocks); + assert_eq!(canonical_concept_domain(0x17FF), ConceptDomain::Blocks); + assert_eq!(canonical_concept_domain(0x1800), ConceptDomain::Unassigned); + // The C-band (OGAR PR #276): strata above the Rust substrate. + assert_eq!(canonical_concept_domain(0xC000), ConceptDomain::JavaRuntime); + assert_eq!(canonical_concept_domain(0xC0FF), ConceptDomain::JavaRuntime); + assert_eq!(canonical_concept_domain(0xC100), ConceptDomain::Analytics); + assert_eq!( + canonical_concept_domain(0xC400), + ConceptDomain::BinaryLifting + ); + // The C2-C3 gap stays Unassigned BY INTENT (slots chosen + // deliberately, C4 = the blast radius), and the band's edges hold. + assert_eq!(canonical_concept_domain(0xC200), ConceptDomain::Unassigned); + assert_eq!(canonical_concept_domain(0xC300), ConceptDomain::Unassigned); + assert_eq!(canonical_concept_domain(0xBF00), ConceptDomain::Unassigned); + assert_eq!(canonical_concept_domain(0xC500), ConceptDomain::Unassigned); + // 0x0C Automation is NOT 0xC0 JavaRuntime -- digit swap, pinned + // two-sided (the same transposition pin OGAR carries). + assert_eq!(canonical_concept_domain(0x0C01), ConceptDomain::Automation); + assert_eq!(canonical_concept_domain(0xC001), ConceptDomain::JavaRuntime); } #[test] @@ -813,8 +873,16 @@ mod tests { let op = AppPrefix::OpenProject.render(0x0103); let rm = AppPrefix::Redmine.render(0x0103); - assert_ne!(classid_app_prefix(op), classid_app_prefix(rm), "render lenses differ"); - assert_eq!(classid_concept(op), classid_concept(rm), "concept is shared"); + assert_ne!( + classid_app_prefix(op), + classid_app_prefix(rm), + "render lenses differ" + ); + assert_eq!( + classid_concept(op), + classid_concept(rm), + "concept is shared" + ); assert_eq!( render_classid_for_concept(AppPrefix::Healthcare, "nope"), @@ -860,7 +928,11 @@ mod tests { 0x0000_0000, 0xFFFF_FFFF, ] { - assert_eq!(flip_classid(flip_classid(id)), id, "flip must be involutive"); + assert_eq!( + flip_classid(flip_classid(id)), + id, + "flip must be involutive" + ); } } @@ -882,7 +954,10 @@ mod tests { let legacy = compose_classid_with(ClassidOrder::CanonLow, concept, prefix); assert_eq!(legacy, ((prefix as u32) << 16) | (concept as u32)); - assert_eq!(split_classid_with(ClassidOrder::CanonLow, legacy), (concept, prefix)); + assert_eq!( + split_classid_with(ClassidOrder::CanonLow, legacy), + (concept, prefix) + ); assert_eq!(flip_classid(legacy), id); } } @@ -911,7 +986,11 @@ mod tests { split_classid_with(ClassidOrder::CanonHigh, fma).0, split_classid_with(ClassidOrder::CanonHigh, cpic).0, ]; - assert_eq!(canons, [0x0701, 0x0A01, 0x0E01], "canon halves stay distinct"); + assert_eq!( + canons, + [0x0701, 0x0A01, 0x0E01], + "canon halves stay distinct" + ); assert_eq!( [osint as u16, fma as u16, cpic as u16], [0x1000, 0x1000, 0x1000], diff --git a/crates/lance-graph-ogar/src/lib.rs b/crates/lance-graph-ogar/src/lib.rs index 55a53846..29b89c45 100644 --- a/crates/lance-graph-ogar/src/lib.rs +++ b/crates/lance-graph-ogar/src/lib.rs @@ -173,6 +173,11 @@ pub mod parity { | (O::HR, C::HR) | (O::Genetics, C::Genetics) | (O::Geo, C::Geo) + | (O::Ontology, C::Ontology) + | (O::Blocks, C::Blocks) + | (O::JavaRuntime, C::JavaRuntime) + | (O::Analytics, C::Analytics) + | (O::BinaryLifting, C::BinaryLifting) | (O::Unassigned, C::Unassigned) ) } @@ -212,6 +217,29 @@ pub mod parity { assert!(n >= 32, "expected ≥32 promoted concepts, got {n}"); } + #[test] + fn reserved_empty_domains_agree_across_the_mirror() { + // The codebook-parity walk above only visits ids that carry + // CONCEPT ROWS, so a reserved-EMPTY domain (Blocks, the C-band) + // added to one enum but not the other would slip past it — the + // exact drift this pairing exists to catch. Pin one id per + // reserved/new domain, an id from a POPULATED new domain + // (0x0333, the DisMech mints in Ontology), the deliberate + // C2-C3 gap, and the 0x0C/0xC0 digit-swap hazard two-sided. + for id in [ + 0x0300u16, 0x0333, // Ontology (populated: dismech) + 0x1701, 0x17FF, // Blocks + 0xC000, 0xC0FF, // JavaRuntime (Panama FFM alone) + 0xC100, // Analytics + 0xC400, // BinaryLifting + 0xC200, 0xC300, // the deliberate gap (both Unassigned) + 0xBF00, 0xC500, // band edges (both Unassigned) + 0x0C01, 0xC001, // Automation vs JavaRuntime, transposed + ] { + assert!(domains_agree(id), "domain drift at {id:#06x}"); + } + } + #[test] fn classid_low_u16_is_the_codebook_id() { use lance_graph_contract::NodeGuid; @@ -287,8 +315,7 @@ impl lance_graph_contract::hotplug::CapabilityAuthority for OgarAuthority { .into_iter() .map(|(name, id)| (name.to_string(), id)) .collect(); - if let Some(drift) = - lance_graph_contract::hotplug::verify_against_mirror(&concepts) + if let Some(drift) = lance_graph_contract::hotplug::verify_against_mirror(&concepts) { return Err(drift); } From 83601641e2e0f6737aabe66bc76ae9e6f924c1d8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 17:36:25 +0000 Subject: [PATCH 2/8] lotus Phase 0/1: frontier audit + F-ORD-REAL pre-registered falsifier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Research charter deliverables 1-2 (no fix included, by design): - docs/lotus/LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology across the write path, persistence capability, placement/comma prior art, and frontier visibility, every statement graded VERIFIED / INFERENCE / HYPOTHESIS / BLOCKER. Headline findings: content_hash folds arrival-minted stream_position values into batch_hash (persist_sink.rs:414), contradicting DetachedCycleBatch's own order-independence doc; the seal is O(batch bytes) x3 passes with the batch resident up to 3x at seal; SweepSlot's caller-supplied semantic order-key contract vs collect_casts' arrival mint; the lance crate source is absent from this sandbox (prepared-artifact capability audit BLOCKED). Section 6 answers the permeability question: the cycle does not become permeable, it becomes thin — trailing-publication pipelining (Regime A) needs no epistemic weakening; rung-qualified frontier visibility (temporal.rs EpistemicMode ladder) stratifies rather than reopens the retired race; texts stay linear, tiles get derived placement, resolved per class. - docs/lotus/F-ORD-REAL-FALSIFIER.md — the defect mechanism in four verified steps + the test design (perturb the process that creates the key: permute cast() call order, never post-mint slots). - cycle_driver.rs tests: f_ord_real_defect_pin_... (GREEN, two-sided — anti-vacuity proves the perturbation reaches the key mint; semantic set + image pinned arrival-independent; batch_hash pinned arrival-DEPENDENT; fails loudly when a fix lands) and f_ord_real_publication_identity_... (#[ignore]d RED falsifier — the desired property, red under --ignored on the real chain). Gates: cargo test -p lance-graph-supervisor --features cycle-driver 28 passed / 1 ignored (+ suites green); RED verified red under --ignored; fmt clean; clippy adds zero new warnings (8 pre-existing recover_fleet lints only visible under this non-default feature). Board: EPIPHANIES E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1; STATUS_BOARD lotus-seal-fractal-commit-frontier section (D-LOTUS-1..9). Co-Authored-By: Claude --- .claude/board/EPIPHANIES.md | 51 ++ .claude/board/STATUS_BOARD.md | 19 + .../src/cycle_driver.rs | 127 +++++ docs/lotus/F-ORD-REAL-FALSIFIER.md | 117 +++++ docs/lotus/LOTUS-FRONTIER-AUDIT.md | 457 ++++++++++++++++++ 5 files changed, 771 insertions(+) create mode 100644 docs/lotus/F-ORD-REAL-FALSIFIER.md create mode 100644 docs/lotus/LOTUS-FRONTIER-AUDIT.md diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index efe390d4..8f0cfcee 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,54 @@ +## 2026-08-18 — E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1 + +**Status:** FINDING (VERIFIED at file:line) + pre-registered falsifier landed +(LOTUS research charter Phase 0/1; docs: `docs/lotus/LOTUS-FRONTIER-AUDIT.md` ++ `docs/lotus/F-ORD-REAL-FALSIFIER.md`). + +**The defect chain.** `BatchWriter::cast` mints `CastId` at ARRIVAL +(`batch_writer.rs:132-138`) → `collect_casts` derives `stream_position = +position_base + cast.0` (`cycle_driver.rs:385`) → `freeze` stable-sorts by it +(`persist_sink.rs:378` — storage ORDER is safe) → **`content_hash` folds the +`stream_position` VALUES into `batch_hash`** (`persist_sink.rs:414`). The +publication identity — the durable `(cycle, batch_hash)` idempotency key — +therefore depends on producer completion order. The struct's own doc +(`persist_sink.rs:359-362`, "Identical completed sets yield identical hashes +regardless of worker completion order") is FALSE today. The row-keyed `image` +leg HOLDS (identity-derived `row_of(owner)`); the restart leg was already +answered (durable `position_base`, falsifier at `cycle_driver.rs:1460`). + +**Landed with this entry:** the GREEN two-sided defect pin +(`f_ord_real_defect_pin_arrival_order_leaks_into_batch_hash` — anti-vacuity: +perturbation provably reaches the key mint; semantic set + image asserted +arrival-INDEPENDENT; hash asserted arrival-DEPENDENT) and the `#[ignore]`d +RED falsifier (`f_ord_real_publication_identity_is_arrival_order_independent` +— red under `--ignored`, hashes 2604999916736672513 vs 4858955943411201665 on +the real chain). Fix mechanism deliberately unprescribed (four candidates in +the falsifier doc, per-workload per the linear-vs-tile split below). + +**Three adjacent verified tensions recorded in the audit, not fixed here:** +(1) `SweepSlot::stream_position`'s contract says "the caller's EXISTING +canonical (textual/stream) order key… NOT a new coordinate" +(`persist_sink.rs:168-183`) while `collect_casts` MINTS it — for linear/text +workloads the fix is carrying the true semantic position (then hashing it is +CORRECT); for tile workloads identity-derived placement; per-class resolution, +never one global rule (operator: "texts are organized linearly; GridLake +tiles would make it contradictory"). (2) The descriptor doctrine +(`batch_writer.rs:30-39`, "P is a DESCRIPTOR — never owned delta bytes") is +unrealized: the only production-shaped instantiation is `BatchWriter>` +→ batch resident up to 3× at seal (staging + image clone + retry clone). +(3) The seal is O(batch bytes) ×3 passes (repair sort + clone + byte-wise +FNV) — `order_cycle_stably`'s own doc already concedes the constructive +scatter alternative (`persist_sink.rs:126-129`). + +**Session BLOCKER, flagged loud:** the lance crate source is ABSENT from this +sandbox (zero `lance*` under the registry; family pinned `=9.0.0`) — the +prepared-artifact capability audit (fragments/Transaction/cleanup/blob) is +UNVERIFIABLE from source this session; Phase 6 of the charter is gated on a +`cargo fetch` or operator-sanctioned source consult. Also owed: the +`temporal.rs:396`/`:410` doc-drift (cites the RETIRED `DurableWitness` / +`DurableCoordinate` as production implementor) — correction folded into the +next phase PR. + ## 2026-08-18 — E-OGAR-CODEBOOK-MIRROR-DOMAIN-DRIFT-SYNCED-1 **Status:** FINDING + fix (cross-session: found by the lance-graph-java diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index b05e20fd..6368f397 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,22 @@ +## lotus-seal-fractal-commit-frontier — research arc (PRE-REGISTERED 2026-08-18) + +Charter: operator, 2026-08-18 (research + architecture; "first audit the +current code, build falsifiers, and try to kill the idea"). Docs live in +`docs/lotus/`. Verdict gate (A-E) before ANY implementation phase; no +metaphor becomes a type name without council approval. + +| D-id | Deliverable | Status | +|---|---|---| +| D-LOTUS-1 | LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology, 4 lenses, VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER graded; §6 answers the permeability/amortization question | **In PR** | +| D-LOTUS-2 | F-ORD-REAL-FALSIFIER.md + the pre-registered tests (GREEN defect pin + `#[ignore]`d RED falsifier, `cycle_driver.rs`) | **In PR** — RED lands BEFORE any fix, per charter | +| D-LOTUS-3 | LOTUS-SEAL-DESIGN.md (16⁴ tree, petal seal, H(parent)=H(children) — NEW design per audit §2.3) | Queued (post-verdict) | +| D-LOTUS-4 | COMMA-MODULATION-EXPERIMENT.md (petal-index-tier comma; locality-vs-resonance trade; "if it doesn't survive measurement, delete it") | Queued | +| D-LOTUS-5 | TEMPORAL-FRONTIER-SEAM.md (rung-qualified frontier visibility; hindsight-invariance falsifier; audit §6.4) | Queued | +| D-LOTUS-6 | PREPARED-ARTIFACT-PUBLICATION.md | **BLOCKED — lance crate source absent from sandbox** (audit §2.2); needs `cargo fetch` or sanctioned source consult | +| D-LOTUS-7 | BENCHMARK-PLAN.md (seal/append split; cycle-major vs tile-major k∈{1,4,16}; F-AMORT version-range scaling; F-MEM RAM high-water freeze-vs-petal; viewport partial hydration) | Queued | +| D-LOTUS-8 | PRE-REGISTERED-FALSIFIERS.md (the F-* catalogue) | Queued | +| D-LOTUS-9 | Final verdict A-E | Queued — gates all implementation | + ## weather-soa-bake-v1 — Zarr → NodeRow, the missing bake (PRE-REGISTERED 2026-08-13) Plan: `.claude/plans/weather-soa-bake-v1.md`. The arc's Python-over-Zarr phase diff --git a/crates/lance-graph-supervisor/src/cycle_driver.rs b/crates/lance-graph-supervisor/src/cycle_driver.rs index 167efbcb..925481ba 100644 --- a/crates/lance-graph-supervisor/src/cycle_driver.rs +++ b/crates/lance-graph-supervisor/src/cycle_driver.rs @@ -2488,4 +2488,131 @@ mod tests { 4 ); } + + // ── F-ORD-REAL (LOTUS Phase 1): publication identity vs producer arrival ──── + // + // The chain under test spans three functions: + // `BatchWriter::cast` mints `CastId` in ARRIVAL order (batch_writer.rs) + // -> `collect_casts` derives `stream_position = position_base + cast.0` + // -> `DetachedCycleBatch::freeze` stable-sorts by `stream_position` and + // then `content_hash` FOLDS THE `stream_position` VALUES THEMSELVES + // into `batch_hash` (persist_sink.rs, the + // `eat(&s.stream_position.to_le_bytes())` line). + // So even after the stable sort, the PUBLICATION IDENTITY of a cycle still + // depends on which producer finished first. `DetachedCycleBatch`'s own doc + // ("Identical completed sets yield identical hashes regardless of worker + // completion order") claims otherwise; these tests pin the actual + // behaviour. Design + evidence: docs/lotus/F-ORD-REAL-FALSIFIER.md. + // + // F-ORD sharpening (operator, 2026-08-18): "Do not test permutation after + // the order key already exists. Perturb the process that creates the key." + // These tests therefore permute the ORDER OF `cast()` CALLS — upstream of + // the CastId mint — never a `Vec` handed to `freeze` (freeze + // sorts, so a post-mint permutation passes vacuously). + + /// Owner-derived payload — a pure function of identity, so the SEMANTIC + /// completed set is identical no matter which arrival order produced it. + fn owner_payload(owner: MailboxId) -> Vec { + vec![owner as u8, (owner >> 8) as u8, 0xA5, (owner as u8) ^ 0x5A] + } + + /// Cast one identical semantic set (64 owners, no moves, owner-derived + /// payloads) in the given ARRIVAL order, then run the real + /// `collect_casts` -> `freeze` chain against one fixed frame. + fn freeze_with_arrival_order(arrival: &[MailboxId]) -> DetachedCycleBatch { + let mut writer: BatchWriter> = BatchWriter::new(); + for &owner in arrival { + writer.cast(owner, vec![], owner_payload(owner)); + } + let collected = collect_casts(&mut writer, CycleId(7), 0, u64::from); + assert!(collected.held.is_empty(), "no moves cast, nothing to hold"); + DetachedCycleBatch::freeze( + CycleFrame::new(CycleId(7), DatasetVersion(3)), + collected.slots, + ) + } + + /// The batch's SEMANTIC content — landings keyed by owner, stripped of the + /// arrival-minted coordinate: (owner, row, paired_move, payload). + fn semantic_set(b: &DetachedCycleBatch) -> Vec<(MailboxId, u64, Option, Vec)> { + let mut v: Vec<_> = b + .landings + .iter() + .map(|s| (s.owner, s.row, s.paired_move, s.payload.clone())) + .collect(); + v.sort_by_key(|e| e.0); + v + } + + /// The GREEN defect pin — two-sided: proves the perturbation reaches the + /// key mint, proves the semantic set and the row-keyed image are + /// arrival-INDEPENDENT (the legs that hold), and pins that `batch_hash` + /// is arrival-DEPENDENT (the defect). When the publication-identity fix + /// lands, the final assert FAILS — re-pin deliberately: delete this test + /// and un-ignore the RED falsifier below. + #[test] + fn f_ord_real_defect_pin_arrival_order_leaks_into_batch_hash() { + let forward: Vec = (0..64).collect(); + let reversed: Vec = (0..64).rev().collect(); + let b1 = freeze_with_arrival_order(&forward); + let b2 = freeze_with_arrival_order(&reversed); + + // Anti-vacuity: the perturbation genuinely reached the CastId mint — + // the SAME owner carries a DIFFERENT stream_position in the two runs + // (owner 0 arrives first in one and last in the other). Without this, + // "hashes differ" could be asserted vacuously of any two batches. + let pos_of = |b: &DetachedCycleBatch, owner: MailboxId| { + b.landings + .iter() + .find(|s| s.owner == owner) + .expect("owner landed") + .stream_position + }; + assert_ne!( + pos_of(&b1, 0), + pos_of(&b2, 0), + "perturbation must reach the key mint" + ); + + // The SEMANTIC completed set is identical by construction... + assert_eq!( + semantic_set(&b1), + semantic_set(&b2), + "same completed set in both arrival orders" + ); + // ...and the row-keyed image leg HOLDS (`row_of(owner)` discipline): + assert_eq!( + b1.image, b2.image, + "the coalesced image is arrival-independent" + ); + + // THE DEFECT, pinned: content_hash folds the stream_position VALUES, + // and those are minted from arrival. + assert_ne!( + b1.batch_hash, b2.batch_hash, + "pinned DEFECT: batch_hash currently depends on producer arrival \ + order, contradicting DetachedCycleBatch's own doc. If this just \ + failed, the fix landed — delete this pin and un-ignore \ + f_ord_real_publication_identity_is_arrival_order_independent." + ); + } + + /// F-ORD-REAL — the pre-registered RED falsifier (the DESIRED property). + /// Fails by design until the publication-identity fix lands; see it red + /// with `cargo test -p lance-graph-supervisor -- --ignored f_ord_real`. + #[test] + #[ignore = "F-ORD-REAL pre-registered RED falsifier: batch_hash is currently arrival-dependent (docs/lotus/F-ORD-REAL-FALSIFIER.md); un-ignore with the fix and delete the defect pin"] + fn f_ord_real_publication_identity_is_arrival_order_independent() { + let forward: Vec = (0..64).collect(); + let reversed: Vec = (0..64).rev().collect(); + let b1 = freeze_with_arrival_order(&forward); + let b2 = freeze_with_arrival_order(&reversed); + assert_eq!(semantic_set(&b1), semantic_set(&b2)); + assert_eq!(b1.image, b2.image); + assert_eq!( + b1.batch_hash, b2.batch_hash, + "identical completed sets must yield identical publication \ + identity regardless of producer arrival order" + ); + } } diff --git a/docs/lotus/F-ORD-REAL-FALSIFIER.md b/docs/lotus/F-ORD-REAL-FALSIFIER.md new file mode 100644 index 00000000..f889914f --- /dev/null +++ b/docs/lotus/F-ORD-REAL-FALSIFIER.md @@ -0,0 +1,117 @@ +# F-ORD-REAL — the arrival-order publication-identity falsifier (Phase 1) + +> Deliverable 2 of the LOTUS SEAL research charter (operator, 2026-08-18). +> Companion audit: `LOTUS-FRONTIER-AUDIT.md` §1 (the verified chain). The +> tests this document pre-registers land in the SAME PR, in +> `crates/lance-graph-supervisor/src/cycle_driver.rs`'s test module — the +> falsifier lands BEFORE any fix, per the charter's preference. + +## 1. The mechanism, in four verified steps + +1. `BatchWriter::cast` mints `CastId(next_id++)` at ARRIVAL + (`lance-graph-planner/src/batch_writer.rs:132-138`). +2. `collect_casts` derives `stream_position = position_base + cast.0` + (`lance-graph-supervisor/src/cycle_driver.rs:385`). +3. `DetachedCycleBatch::freeze` stable-sorts by `stream_position` + (`lance-graph-planner/src/persist_sink.rs:378`) — storage ORDER is safe. +4. `content_hash` folds the `stream_position` VALUES into `batch_hash` + (`persist_sink.rs:414`, `eat(&s.stream_position.to_le_bytes())`) — so the + publication identity (the durable idempotency key, the `(cycle, batch_hash)` + reconciliation key, the frame the store persists) depends on which producer + happened to finish first. + +**The smoking gun is a doc-contradiction:** `DetachedCycleBatch`'s own field +doc (`persist_sink.rs:359-362`) claims *"Identical completed sets yield +identical hashes regardless of worker completion order (the freeze +canonicalizes first)."* The freeze canonicalizes the ORDER; it does not +canonicalize the COORDINATES the hash then eats. The sentence is false today. + +**What is NOT implicated (already answered upstream):** +- The restart leg — `position_base` is a durable cursor + (`cycle_driver.rs:340-347`), pinned by + `restart_stable_stream_positions_survive_writer_reconstruction` (`:1460`). +- The row-keyed image — `image: row → last payload` with identity-derived + `row = row_of(owner)` is arrival-independent when rows are per-owner. The + defect pin asserts this GREEN leg explicitly, so the test documents which + leg holds, not just which breaks. + +## 2. Test design — perturb the process that creates the key + +Operator sharpening (A2, 2026-08-18): *"Do not test permutation after the +order key already exists. Perturb the process that creates the key."* + +Applied literally: the tests permute **the order of `cast()` calls** — the +upstream event that mints `CastId` — and then run the REAL +`collect_casts → freeze` chain. Two designs were rejected as vacuous or weak: + +- **Permuting `Vec` before `freeze`** — vacuous: `freeze` sorts by + the already-minted key, so any post-mint permutation passes regardless of + the defect. This is exactly the trap the sharpening names. +- **Real threads + jitter** — a nondeterministic way to produce the same + perturbation the deterministic form applies directly; it can pass by + scheduler luck. The deterministic permuted-arrival form is the stronger + falsifier and is what lands. (A threaded stress variant may join the Phase 5 + benchmark plan; it adds realism, not evidence.) + +Construction: 64 owners; payloads a pure function of owner identity (so the +SEMANTIC completed set is identical by construction across runs); no kanban +moves (the defect is fully expressed by positions; moves would only widen the +API surface under test); `row_of = identity`. Run once in forward arrival +order, once reversed; freeze both against the same `CycleFrame`. + +## 3. What lands, and what each half means + +**(a) The GREEN defect pin** — +`f_ord_real_defect_pin_arrival_order_leaks_into_batch_hash`: + +- anti-vacuity guard: the same owner carries a DIFFERENT `stream_position` in + the two runs — proof the perturbation reached the key mint (without this, + equal hashes would be vacuously assertable); +- asserts the semantic sets equal AND `image` equal (the leg that holds); +- asserts `batch_hash` DIFFERS (the defect, pinned two-sided). **When a fix + lands this assert fails**, forcing a deliberate re-pin: delete the pin and + un-ignore (b). A silent fix cannot slip past it. + +**(b) The RED falsifier** — +`f_ord_real_publication_identity_is_arrival_order_independent`, `#[ignore]`d +with a message citing this document. It asserts the DESIRED property (equal +semantic set ⇒ equal `batch_hash`) and fails today — run +`cargo test -p lance-graph-supervisor -- --ignored f_ord_real` to see it red. +The ignore attribute is what lets a red-by-design falsifier land on a green +main, per the charter's land-the-falsifier-first preference. + +## 4. Green criterion for any fix — mechanism deliberately unprescribed + +A fix is green when: identical completed semantic sets yield identical +`batch_hash` (test (b) un-ignored and green), with the restart falsifier, the +`(cycle, batch_hash)` reconciliation path, and `recover_and_apply`'s +per-owner watermark contract (`stream_position` monotonic per owner ACROSS +cycles — `persist_sink.rs:176-182`) all still green. + +Candidate mechanisms, listed NOT chosen (Phase 2+ design, council-gated): + +1. hash the canonical INDEX (position within the sorted batch) instead of the + `stream_position` value; +2. hash owner-keyed content only (owner, row, move, payload) and drop the + coordinate from the identity entirely; +3. identity-derived positions (the lotus placement: slot = f(identity), so + arrival never mints a coordinate at all) — the only candidate that ALSO + removes the seal's repair sort; see the audit §6.3; +4. carry the TRUE semantic order key through the cast (linear/textual + workloads only) — `SweepSlot`'s contract already says the key is + *"the caller's EXISTING canonical (textual/stream) order key… NOT a new + coordinate"* (`persist_sink.rs:168-183`); for a genuinely linear stream, + supplying the real witness position makes hashing it CORRECT, because the + coordinate becomes semantic identity rather than arrival residue. + +Candidates 3 and 4 are not rivals — they are the two halves of the workload +split (audit §6.5, "texts are linear; GridLake tiles would make it +contradictory"): linear-stream classes keep a supplied semantic key, tile +classes get derived placement, resolved per class like every other sanctioned +reading. Each candidate has a distinct blast radius on the watermark contract +— that analysis is the Phase 2 design document's job, not this falsifier's. +The falsifier is mechanism-agnostic by construction: it constrains WHAT must +become true, never HOW — with one nuance: under candidate 4 the test's +"identical semantic set" premise must key on the SUPPLIED semantic positions +(two arrival orders of the same text carry the same textual positions, so the +falsifier's assertion is unchanged in substance). diff --git a/docs/lotus/LOTUS-FRONTIER-AUDIT.md b/docs/lotus/LOTUS-FRONTIER-AUDIT.md new file mode 100644 index 00000000..ae714e85 --- /dev/null +++ b/docs/lotus/LOTUS-FRONTIER-AUDIT.md @@ -0,0 +1,457 @@ +# LOTUS-FRONTIER-AUDIT — Phase 0 archaeology (research arc, 2026-08-18) + +> Deliverable 1 of the LOTUS SEAL / FRACTAL COMMIT FRONTIER research charter +> (operator, 2026-08-18). Companion: `F-ORD-REAL-FALSIFIER.md` (deliverable 2, +> same directory). This document changes NO code; the paired PR lands only the +> pre-registered falsifier tests it describes. +> +> **Grading legend (charter):** +> - **VERIFIED** — read at file:line in this working tree this session. +> - **INFERENCE** — mechanism argued from VERIFIED parts; not yet measured. +> - **HYPOTHESIS** — the research claim itself; needs a falsifier to run. +> - **BLOCKER** — cannot be verified in this session; names what unblocks it. + +## §0 The question under investigation + +The charter investigates deterministic inverse-Morton 4×4 placement (a 16⁴ +tree: 65,536 leaves → 4,096 → 256 → 16 → root), local 16-way sealing (a petal +seals when all 16 children are RESOLVED — present or explicitly empty), +comma-style anti-resonance modulation, temporal/frontier visibility, and +pre-publication immutable chunk writes — as a way out of the +whole-batch-barrier trap without breaking cycle atomicity. + +The operator sharpened the underlying question mid-arc (2026-08-18): + +> *"Can temporal.rs amortize enough so that the cycles become permeable?"* +> — with the cost anchor *"a whole 64k batch costs 233ms write and seal"* +> (operator-measured, out-of-tree; the provenance discipline of the D-LGJ-W8 +> spec applies: OPERATOR-MEASURED, not re-derived here), the observation +> *"the Seal was potentially expensive too"*, the pipeline hypothesis +> *"if SoA are allowed to pull through with 4-16 cycle processing it can +> self-amortize for L1 cache cycles"*, and the mechanism bet *"the lotus or +> Pythagorean comma and Morton self-alignment sorting would try to make +> batches cheap enough."* + +§6 answers this against the evidence. Short form: **the cycle does not become +permeable; it becomes thin** — and the seal-cost evidence (§1.4) shows the +operator's mechanism bet attacks the same root as the F-ORD defect. + +## §1 The write path as it exists (VERIFIED) + +### §1.1 Arrival mints the coordinate + +`BatchWriter::cast` (`crates/lance-graph-planner/src/batch_writer.rs:132-138`) +mints `CastId(next_id++)` in ARRIVAL order. `CastId`'s derived `Ord` is +insertion order by design (`:84-90`). The module doc's description of the read +side is DECLARED-UNWIRED by its own header (`:12-20`; `cast()` has zero +production call sites — `TD-DOC-COMMENTS-CLAIM-UNWIRED-BEHAVIOUR`). + +### §1.2 The coordinate becomes the order key + +`collect_casts` (`crates/lance-graph-supervisor/src/cycle_driver.rs:357-393`) +derives `stream_position = position_base + cast.0` (`:385`). The RESTART leg is +already answered: `position_base` is a durable cursor +(`:340-347`; falsifier `restart_stable_stream_positions_survive_writer_reconstruction`, +`:1460`) — stream positions stay monotonic across writer rebuilds. What is NOT +answered is the within-cycle leg: two runs of the same semantic work with +different worker completion orders mint different `(owner → stream_position)` +assignments. + +### §1.3 The publication identity eats the arrival coordinate — THE DEFECT + +`DetachedCycleBatch::freeze` (`crates/lance-graph-planner/src/persist_sink.rs:377-390`) +stable-sorts by `stream_position` (`:378` via `order_cycle_stably`, `:133-135`) +— so far so good: completion order never becomes STORAGE order. But +`content_hash` (`:401-429`) then folds **the `stream_position` VALUES +themselves** into the batch hash (`:414`, +`eat(&s.stream_position.to_le_bytes())`). Since those values are minted from +arrival (§1.1-1.2), **`batch_hash` — the durable idempotency key and the +publication identity — depends on producer completion order even after the +sort.** + +The struct's own doc claims the opposite (`:359-362`: *"Identical completed +sets yield identical hashes regardless of worker completion order"*). That +sentence is FALSE today. `F-ORD-REAL-FALSIFIER.md` carries the mechanism and +the pre-registered test (landed alongside this audit: a GREEN two-sided defect +pin + an `#[ignore]`d RED falsifier asserting the desired property). + +What survives: the **row-keyed image leg holds** — `image` is +`row → last payload in stream order` (`:379-382`), `row = row_of(owner)` is +identity-derived, so with per-owner rows and per-owner payloads the coalesced +image is arrival-independent (the defect pin asserts this GREEN leg +explicitly). + +### §1.4 The seal is O(batch bytes) with a full copy and a byte-at-a-time hash (VERIFIED) + +The operator's *"the Seal was potentially expensive too"* is verifiable at +file:line — three full passes over the batch payload: + +1. **Sort** — `order_cycle_stably` is a general `sort_by_key` (`:133-135`); + its own doc already concedes the lotus-shaped alternative: *"when the key + is already a dense slot index into the cycle image, a stable scatter into + predetermined positions is cheaper than a general O(n log n) sort on the + 64k path"* (`:126-129`). +2. **Clone** — `freeze` clones every payload into the image + (`:381`, `image.insert(s.row, s.payload.clone())`); for 64K × 512 B that is + a 32 MB copy. A second full clone is held as the retry cache + (`cycle_driver.rs:420-425`: *"The one clone held until commit success is + the price of offering that cache"*). +3. **Hash** — `content_hash` is byte-at-a-time FNV-1a over every payload byte + (`:405-427`) — a serial, SIMD-hostile pass over the full 32 MB. + +INFERENCE: at 64K × 512 B these three passes make the seal itself +O(tens of ms) before Lance sees a byte, consistent with the operator's 233 ms +write+seal figure being dominated by more than the Lance append. Not measured +in-tree; `BENCHMARK-PLAN.md` (Phase 5 deliverable) owes the split +(seal-only vs append-only vs total). + +### §1.5 The gates that already exist (VERIFIED) + +- Artifact gate: zero non-empty payloads → `CommitOutcome::NoChange`, sink + never invoked, no version minted (`persist_sink.rs:618-630`). +- ≤1 move per owner per cycle, enforced pre-seal; extra moves HELD, re-staged + intent-only (empty payload — deliberately unable to trip the artifact gate) + (`cycle_driver.rs:337-410`). +- Failure semantics: on WAL failure nothing publishes, nothing mutates; + prescribed recovery is **deterministic regeneration from the unchanged Vn** + (`cycle_driver.rs:419-425`). This doctrine is load-bearing for §6's + pipeline answer. + +## §2 Persistence capability (VERIFIED + the session BLOCKER) + +### §2.1 What the store actually calls + +`LanceCycleWriter` (`crates/lance-graph/src/graph/cycle_sink.rs`) uses ONLY the +high-level Lance API: the sole `lance` import is +`use lance::dataset::{Dataset, WriteMode, WriteParams}` (`:89`), and the SOLE +mutation site is `raw_append` (`:706-728`) — `Dataset::write(…, Create)` on +first-ever commit, `ds.append(reader, None)` thereafter. One `RecordBatch` per +cycle carries all three row kinds (FRAME=0 / LANDING=1 / IMAGE=2; +`FixedSizeBinary(512)` payloads; `:99-144`) in ONE atomic commit (`:825-885`). +No `Fragment`, `FragmentCreateBuilder`, `Transaction`, `Operation::*`, blob, or +cleanup/GC API appears anywhere in the file (exhaustive grep). Supporting +machinery, all VERIFIED: `bootstrap()` mints a real version when it creates +(`:647-690`); `guard_schema` rejects drift loudly (`:358-397`); `WriterClaim` +is in-process-only exclusivity (`:221-274`); the `(cycle, batch_hash)` +Reconciled fast path (`:774-824`); `max_cycle` startup watermark (`:399-431`); +`checkpoint_bound` caps recovery below the latecomer fence +(`cycle_driver.rs:912-938`); `recover_and_apply` replays strictly above the +per-owner `stream_position` watermark in stored order, never re-sorting +(`persist_sink.rs:640-…`). + +### §2.2 BLOCKER — the lance crate source is absent from this sandbox + +`~/.cargo/registry/src/index.crates.io-…/` contains zero `lance*` directories; +no `.crate` archives cached; `vendor/` holds only `ractor`; `Cargo.lock` pins +the whole family `=9.0.0`. **Charter task 2(a)-(d) — fragment-level +write-without-commit (`write_fragments` / `Fragment::create`), a two-phase +prepared-fragments commit shape, orphan/cleanup APIs, and blob storage — is +UNVERIFIABLE from source this session.** Everything §2.1 states is about what +`cycle_sink.rs` CALLS, which is real but does not bound what lance 9.0.0 +OFFERS. Phase 6 (`PREPARED-ARTIFACT-PUBLICATION.md`) is **gated** on a +`cargo fetch` (network + disk budget permitting) or an operator-sanctioned +alternative source consult. Do not write Phase 6 from memory of lance APIs. + +### §2.3 The existing "merkle" is not a tree (VERIFIED) + +`MerkleRoot::from_fingerprint` (`crates/lance-graph/src/graph/spo/merkle.rs:18-38`) +is a flat XOR-fold rolling hash over one fingerprint's words — no parent/child +relation, no 16-slot fan-out anywhere in the crate (grep: zero matches for any +`H(parent)=H(children)` construct). `verify_lineage` is structural-only by its +own admission (*"Known gap: does not re-hash"*, `:143-157`). **Consequence: +the lotus petal hash — `H(parent) = H(canonical child slots 0..15)` — is NEW +design, not a reuse.** Nothing existing carries it; nothing existing conflicts +with it. + +## §3 Placement + comma prior art (census; VERIFIED unless noted) + +| Candidate | Where | Verdict for lotus | +|---|---|---| +| `FacetTier::morton()` | `lance-graph-contract` `facet.rs` (canon V3: "256 = 4⁴ hierarchical ancestry") | **Strongest leaf-identity candidate** — the 4-ary centroid-ancestry reading is exactly the 16⁴ tree's per-tier structure | +| `hhtl::NiblePath`, `FAN_OUT=16` | lance-graph core | Production 16-ary router; the doctrine already says *"HHTL is the deterministic PLACE (16-ary); helix is the RESIDUE"* | +| `CurveRuler` stride-4-over-17 | helix | **The shipped comma** — triple-falsified positive (DFT anti-aliasing ev7b; `comma_quorum` N_eff 11.00/12 vs 1.00 without; `comma_awareness`). D-QUANTGATE names it the mandatory quantized-layer phase generator | +| `basin_placement_learning` 75.8% | probe | **Overclaim flag (INFERENCE):** the measured 75.8% is a BINARY split, not 4-ary — `E-BASIN-IS-A-NODE` cites it wider than the measurement supports; re-grade before reuse | +| `symbiont::domino::morton4` | symbiont | Real 4×4 bijective Morton, but SIMD-lane-scoped; not a placement authority | +| thinking-engine `domino.rs` | thinking-engine | FALSE LEAD — name collision, unrelated mechanism | +| Feistel / XorShift mixers | — | ABSENT from the workspace; a comma design needing one imports new code | + +Cross-cutting census finding: Morton interleave is reinvented ≥10× across the +workspace with no canonical home in `ndarray/src` — consolidation onto +`FacetTier::morton()` (or an ndarray primitive it consumes) is a named seam +for the wave, not this PR. + +## §4 Frontier visibility — the adversarial findings (VERIFIED history, HYPOTHESIS design) + +1. **The space the frontier reopens was deliberately closed.** + `DurableWitness` / `DurableCoordinate` were RETIRED 2026-08-02 for epistemic + race safety — the ruling's shape: *an open cycle's output is never visible + as a `Vn` input*. Any design that lets a reader consume sealed-but- + unpublished petals is re-entering that space and must carry the retirement + reasoning, not route around it. +2. **F-VISIBILITY currently holds by STRUCTURAL ABSENCE** — pre-publication + state never touches Lance at all (§2.1: one append per cycle, nothing + else). Any prepared-petal write changes the mechanism that makes this + falsifier pass from "impossible" to "guarded" — a strictly weaker footing + that needs its own falsifier. +3. **Doc drift, correction owed:** `temporal.rs:396` + `:410` still cite the + retired `persist_sink::DurableWitness` / `DurableCoordinate` as "the + production implementor" — stale since the retirement. Fold the correction + into the Phase 2 PR (or its own doc-fix PR); flagged here so it is not + rediscovered. +4. **The supervisor and the core store have no Cargo edge** — + `lance-graph-supervisor` (collect/seal/recover) and + `lance-graph::cycle_sink` (the Lance writer) are composed only in prose; + no crate depends on the other. The lotus seam would be the first real + composition; that is a design event, not a refactor. +5. **The riskiest assumption in the whole charter (HYPOTHESIS):** a "sealed" + state short of the atomic commit either (a) becomes durable pre-publication + — threatening cycle-atomicity and reopening finding 1 — or (b) stays + in-memory — contradicting the restart-reuse motivation. The charter's + git-object pattern (content-addressed petal objects durable EARLY, one + root/manifest flip publishing LATE) is the only shape found that threads + this needle, and it is exactly the shape §2.2's BLOCKER prevents verifying + against lance 9.0.0 this session. + +## §5 Charter maxims, restated against the evidence + +- **ARRIVAL MAY FILL A SLOT, NEVER CREATE IT** — today arrival CREATES the + coordinate (§1.1-1.3). This is the F-ORD-REAL defect in maxim form. +- **UNRESOLVED IS NOT EMPTY** — no present/empty distinction exists anywhere + in the write path today; `SweepSlot` has no tri-state. New design. +- **SEALED MAY BE CONSUMED, OPEN MAY NOT** — today NOTHING pre-publication may + be consumed (§4.2); the maxim names a weakening that must be earned. +- **PREPARED IS NOT PUBLISHED** — no prepared tier exists (§2.1); gated on + §2.2. +- **CANONICALIZATION IS CONSTRUCTIVE, NOT A REPAIR SORT** — the repair sort is + live at `persist_sink.rs:378`, and its own doc (`:126-129`) already names + the constructive alternative. See §6.3. +- **IF COMMA DOESN'T SURVIVE MEASUREMENT, DELETE IT** — §3's comma evidence is + from OTHER carriers (quorum/awareness/DFT); zero placement-tier comma + measurements exist. The comma enters this design as HYPOTHESIS only. + +## §6 The permeability question, answered (INFERENCE on VERIFIED parts) + +**"Can temporal.rs amortize enough so that the cycles become permeable?"** + +Decompose "permeable" by who wants through the wall: + +### §6.1 Reader-plane permeability is the retired race — don't reopen it; make the wall THIN instead + +A query reader consuming open-cycle state as history is precisely the +`DurableWitness` retirement (§4.1). The need it serves — fresher-than-published +reads — is met without it by raising **publication frequency**: smaller, more +frequent cycles. Two costs currently forbid that, and each has an owner: + +- **Write-side cost** (seal + append per cycle): attacked by §6.3. +- **Read-side cost** (a reader now spans MANY small versions): this is exactly + what `temporal.rs` amortizes — `QueryReference::at(v, rung)` + deinterlace + is a zero-copy projection over a version RANGE; N small versions read as one + sorted stream. **HYPOTHESIS to measure (F-AMORT): range-read cost stays + ~flat as version count per unit of work rises 10×.** If F-AMORT holds, the + wall's reader-visible thickness is bounded by commit latency, not batch + size — permeability's *effect* without its race. + +### §6.2 Compute-plane permeability is legal TODAY, and the crash story already exists + +The operator's *"SoA allowed to pull through with 4-16 cycle processing"* is a +bounded pipeline: compute cycles N+1..N+k proceed while seals/appends for +N-k..N trail behind. Two regimes, sharply different: + +- **Regime A (trailing publication, published-horizon reads):** every cycle + still reads a PUBLISHED `Vn`; the horizon lags the compute frontier by ≤k. + No epistemic rule is touched — the Vn rule constrains what may be READ AS + HISTORY, not how far compute may run ahead of durability. Crash semantics + come free: the prescribed recovery is already *deterministic regeneration + from the unchanged Vn* (§1.5) — lose the in-flight window, regenerate ≤k + cycles. The bounded window bounds the regeneration blast radius. +- **Regime B (frontier chaining):** cycle N+1 reads cycle N's sealed-but- + unpublished in-memory image. This is dataflow forwarding, not epistemic + history — but the committed frame's `base_version` must then be the version + N *will* publish, a speculative basis that collapses the whole window on + N's failure. Regime B is HYPOTHESIS; it needs the F-NO-SPECULATION / + F-NO-BARRIER falsifiers and council review. Regime A needs neither. + +### §6.3 The L1 claim, made mechanical — and the operator's mechanism bet lands on the defect's root + +64K × 512 B = 32 MB: the whole batch is not L1-resident (nor L2). What the +4-16-cycle pull-through actually buys is **loop interchange**: the per-cycle +barrier forces cycle-major iteration (stream all 32 MB through DRAM every +cycle); a k-deep pipeline permits row-tile-major iteration — process one +petal-sized tile (16 rows × 512 B = 8 KB, comfortably L1) for k consecutive +cycles while it is hot, then move on. DRAM traffic for the compute phase drops +~k×; the seal's three O(bytes) passes (§1.4) overlap compute instead of +serializing behind it. **That is the honest form of "self-amortize for L1 +cache cycles" — the barrier is not just a latency wall, it forces a +cache-hostile iteration order.** (INFERENCE; the benchmark plan owes +cycle-major vs tile-major at k ∈ {1,4,16}.) + +And the mechanism bet — *"lotus/Pythagorean comma and Morton self-alignment +sorting would try to make batches cheap enough"* — attacks §1.4's three passes +at the root they share with the F-ORD defect: + +1. **Sort → gone.** Identity-derived Morton slots mean the canonical form + exists AT CAST TIME; sealing scatters into predetermined positions — the + constructive canonicalization `order_cycle_stably`'s own doc already + concedes is cheaper (`:126-129`). +2. **Clone → gone.** Content-addressed petal objects read the SoA backing + store zero-copy at flush — the descriptor doctrine `batch_writer.rs` + already states (*"deltas stay in the SoA backing store; the sink reads + them at flush time"*). +3. **Hash → incremental and arrival-independent by construction.** A petal + tree `H(parent)=H(child slots 0..15)` re-hashes only dirty petals, + parallelizes 16-way, and — because slot positions are identity-derived — + **is the F-ORD fix**: publication identity stops eating arrival + coordinates because arrival no longer mints coordinates. + +**One design, two problems:** the seal-cost problem and the +publication-identity defect are the same disease (canonicalization as repair +over arrival-minted coordinates) and the lotus placement is one cure for both. +The comma's role is confined and testable: Morton clustering gives intra-tile +locality (the L1 tiling above) but can hot-spot petals under strided identity +sequences; a deterministic bijective comma permutation at the petal-INDEX tier +spreads stride resonance while leaving intra-petal Morton locality intact. +That is a measurable trade (locality vs resonance), pre-registered as +F-COMMA-*; per §5's maxim, if it doesn't survive measurement it is deleted. + +### §6.4 The rung ladder makes the frontier rung-QUALIFIED — sudoku filling without poisoning hindsight + +The operator's closing refinement: *"temporal.rs rung ladder dependency would +allow for sudoku-like filling of known unknowns without poisoning hindsight."* + +The load-bearing surface already exists, VERIFIED: `temporal.rs` grades **what +a reader at a given rung is allowed to know** (`EpistemicMode`, `:74-99`) — +`for_rung(0..=4) = Strict` (*"reason strictly in the present"*), +`for_rung(5..=8) = Aware` (*"admit hindsight"*), `for_rung(9..) = Retro` +(*"may spoiler-read"*); `QueryReference::at(ref_version, rung)` (`:167-173`) +carries it per read. The epistemic dimension the frontier needs is not new +machinery — it is this ladder pointed at the OTHER direction (ahead of the +publication horizon instead of behind it). + +That reconciles §6.2's Regime B with §4.1's retirement, as HYPOTHESIS: + +- The lotus tree's UNRESOLVED slots are **known unknowns** — the slot exists + (identity-derived, §6.3), only its value is pending. Filling them is + constraint propagation in arbitrary arrival order (the sudoku shape: + arrival fills a slot, never creates it), petals sealing as their 16 + children resolve. +- **Visibility is rung-qualified, not binary.** A frontier fill is consumable + only by readers whose rung explicitly opted into frontier awareness; the + hindsight rungs (`Retro`, and everything reading published history as `Vn`) + see ONLY sealed-and-published state. The 2026-08-02 retirement reasoning + (*"an open cycle's output is never visible as a `Vn` input"*) is thereby + PRESERVED at the hindsight rungs and relaxed only where a reader typed its + tolerance — the race is not reopened, it is stratified. +- The falsifier this pre-registers (fold into F-VISIBILITY/F-STALE): + **hindsight-invariance** — a hindsight-rung read must be bit-identical + whether or not any frontier activity existed at read time. Frontier-rung + reads may vary with the frontier; hindsight reads never. That is "without + poisoning hindsight," made mechanical and two-sided. + +### §6.5 The workload split — texts are LINEAR; GridLake tiles must not claim them + +Operator (2026-08-18): *"Texts are organized linearly. Gridlake tiles would +make it contradictory."* This is a scoping law for the whole design, and it +exposes a SECOND verified doc-vs-code tension: + +`SweepSlot::stream_position`'s own contract (`persist_sink.rs:168-183`) says +it is *"the caller's EXISTING canonical (textual/stream) order key — the +write-side deinterlace input. NOT a new coordinate… the witness-fabric order +key, already monotonic."* But `collect_casts` MINTS it from arrival +(`position_base + cast.0`, §1.2). The persistence contract was written for a +caller-supplied SEMANTIC key; the supervisor supplies an arrival mint. Two +workload families therefore need two different repairs, and conflating them +is the contradiction the operator names: + +- **Linear/textual streams** (the temporal.rs sorted stream, episodic + Markov-on-stream per `E-MARKOV-TEMPORAL-STREAM-1`, witness chains): stream + order IS semantic. The fix is to CARRY the true textual/witness position + through the cast instead of minting one — and then hashing it is CORRECT, + because the coordinate is semantic identity, not arrival residue. Morton + tiling must never claim this order key; deinterlace depends on it. +- **Grid/tile workloads** (owner-keyed SoA rows, spatial/semantic tiles): + no meaningful linear order exists; the coordinate should be + identity-derived (the lotus slot, §6.3) or absent from identity entirely. + +The binding precedent already exists in canon: **domains bind the axes** (the +tier reading is class-resolved — "OSM: literal x/y; semantic: PQ subspace +pairs"). The order-key reading joins that list as a per-class/per-tenant +resolution — linear-stream classes keep a supplied semantic key; tile classes +get derived placement. One substrate, two sanctioned readings, resolved where +every other reading is resolved: at the class, never hardcoded in the sink. + +### §6.6 RAM: chunking when the thought happens vs forcing the batch to freeze + +Operator (2026-08-18): *"Forcing the whole batch to open for displaying a map +also affects RAM usage. The question is: is chunking when the thought happens +more efficient than forcing a batch to freeze?"* + +**The freeze model's RAM shape, VERIFIED — the batch is resident up to THREE +times at seal:** + +1. Staging holds OWNED payload bytes — `BatchWriter.pending_payloads: + Vec<(CastId, P)>` with the production-shaped instantiation being + `BatchWriter>` (`collect_casts`'s signature, + `cycle_driver.rs:357-358`). This is itself a doc-vs-code drift: the + module's own descriptor doctrine (`batch_writer.rs:30-39`, operator ruling + Addendum-6: *"P is a DESCRIPTOR — never owned delta bytes; deltas stay in + the SoA backing store; the sink reads them at flush time"*) is unrealized + on the only wired path. +2. `freeze` clones every payload into the image (§1.4.2). +3. The retry cache holds a third copy until commit success (§1.4.2). + +At 64K × 512 B that is ~32 MB × up to 3 transient, per cycle — plus the +read side: a map viewport that needs a handful of tiles today hydrates +whole-batch-shaped state, because nothing smaller than the cycle image exists +to hydrate. + +**Chunk-at-thought (the lotus shape) converts the spike into a stream:** a +thought's 512 B lands in its identity-derived petal buffer when produced; +a petal seals at 16 resolved children (8 KB — one L1-resident tile, §6.3); +RAM high-water becomes O(open petals), not O(batch) × 3; and a map reader +hydrates exactly the petals its viewport touches — partial hydration is what +the content-addressed tree is FOR (the git-object read path). + +**The honest counterweights (INFERENCE — this is a benchmark question, not a +settled one):** + +- **Coalescing is a real freeze benefit** — *"64 same-row breaths durably cost + ONE image row"* (`cycle_sink.rs:107`). Chunk-at-thought either writes every + breath (write amplification × breaths/row) or keeps a per-petal coalescing + window — the window is small (16 rows), but it must exist. +- **Small-write amplification** — 4,096 sealed petals × 8 KB as individual + durable objects is the object-store anti-pattern; git itself answers this + with packfiles. The petal tier likely stays in-memory (or in one streaming + WAL) until root publication regardless — in which case the WRITE-side I/O + comparison may be a wash and the real wins are RAM high-water, seal + incrementality (§6.3), and read-side partial hydration. +- **Publication atomicity is untouched either way** — one cycle, one version + (§4/§2.1); chunking changes what exists BELOW the commit, never how many + commits there are. + +Pre-registered as the F-MEM falsifier plus a benchmark row (Phase 5): +RAM high-water and seal latency, freeze-model vs petal-model, at 64K × 512 B +with breaths/row ∈ {1, 4, 64}, plus a viewport-read row (hydrate 1% of tiles: +bytes touched, freeze vs petal). If the petal model does not win RAM +high-water by a large factor, it is not carrying its complexity. + +### §6.7 Verdict shape for the phases ahead + +- The RED F-ORD-REAL falsifier lands NOW (this PR) — it is true regardless of + which fix wins, and it is the gate every fix must turn green. +- Regime A pipelining + F-AMORT measurement need no epistemic weakening and no + lance capability beyond what §2.1 verified — they are the low-risk half. +- Prepared petals (the git-object pattern) and Regime B are gated on §2.2's + BLOCKER and on council review of §4's retirement reasoning. +- No metaphor becomes a type name without council approval (charter). + +## §7 Index + +| Deliverable | Status | +|---|---| +| 1. LOTUS-FRONTIER-AUDIT.md | this document | +| 2. F-ORD-REAL-FALSIFIER.md | landed alongside, with the pre-registered tests | +| 3-9 (design/experiment/benchmark/verdict docs) | NOT STARTED; 6 gated on §2.2, comma work gated on F-ORD fix design | + +Falsifiers referenced: F-ORD-REAL (landed), F-AMORT, F-NO-BARRIER, +F-NO-SPECULATION, F-VISIBILITY, F-COMMA-*, F-LOCAL-SEAL, F-RESTART, F-DUP, +F-STALE, F-PUBLISH, F-64K, F-MEM — pre-registration doc is deliverable 8. From 72b06243e90698dcb7d6669aab28c4ffbfbbcca3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 17:39:12 +0000 Subject: [PATCH 3/8] board: #961 arc entry + LATEST_STATE + D-LOTUS-1/2 Shipped flips Co-Authored-By: Claude --- .claude/board/LATEST_STATE.md | 18 ++++++++++++++++++ .claude/board/PR_ARC_INVENTORY.md | 27 +++++++++++++++++++++++++++ .claude/board/STATUS_BOARD.md | 4 ++-- 3 files changed, 47 insertions(+), 2 deletions(-) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 8e27987a..5f46b7c8 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,21 @@ +## 2026-08-18 — lance-graph #961 (MERGED 0cc171f) — LOTUS research arc opens: frontier audit + F-ORD-REAL falsifier + +### Current Contract Inventory — no new types (docs + tests only) + +- `docs/lotus/LOTUS-FRONTIER-AUDIT.md` — Phase 0 archaeology; the verified + F-ORD defect chain (`content_hash` folds arrival-minted `stream_position`, + `persist_sink.rs:414`, contradicting the struct's own doc); seal = 3× + O(bytes) passes with the batch resident up to 3× (descriptor doctrine + unrealized: `BatchWriter>`); lance crate source ABSENT from sandbox + (Phase 6 BLOCKED); §6: cycles become THIN not permeable (Regime A + pipelining; rung-qualified frontier; linear-text vs tile split per class). +- `docs/lotus/F-ORD-REAL-FALSIFIER.md` + `cycle_driver.rs` tests: GREEN + defect pin `f_ord_real_defect_pin_arrival_order_leaks_into_batch_hash` + (fails loudly when a fix lands → deliberate re-pin) + `#[ignore]`d RED + `f_ord_real_publication_identity_is_arrival_order_independent`. +- Gates: supervisor suite 28 passed / 1 ignored (`--features cycle-driver`); + RED verified red under `--ignored`; fmt clean; zero new clippy warnings. + ## 2026-08-17 — branch `claude/hydrate-staging-merge` — `lance-graph-hydrate`: the deferred staging/publish merge lands (closes `ISS-HYDRATE-DIR-AND-FILE-DUPLICATE-THEIR-STAGING-BODIES`) ### Current Contract Inventory — new private module (`crates/lance-graph-hydrate/src/publish.rs`) diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index ae6035f2..f43b5021 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -1,3 +1,30 @@ +## 2026-08-18 — lance-graph #961 (MERGED, squash 0cc171f) — LOTUS Phase 0/1: frontier audit + F-ORD-REAL pre-registered falsifier + +- **Added:** `docs/lotus/LOTUS-FRONTIER-AUDIT.md` (Phase 0 archaeology, 4 + lenses, graded VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER; §6 answers the + operator's permeability/amortization question: the cycle becomes THIN, not + permeable — trailing-publication pipelining Regime A; rung-qualified + frontier via the shipped `EpistemicMode::for_rung` ladder; the linear-text + vs tile workload split resolved per class); `docs/lotus/ + F-ORD-REAL-FALSIFIER.md`; two pre-registered tests in `cycle_driver.rs` + behind the existing `cycle-driver` feature (GREEN two-sided defect pin + + `#[ignore]`d RED falsifier, verified red under `--ignored`: hashes + 2604999916736672513 vs 4858955943411201665 for the identical semantic set). +- **Locked:** the falsifier perturbs the process that CREATES the key (cast() + call order), never post-mint slots (the operator's F-ORD sharpening); no + fix mechanism prescribed — four candidates listed, per-workload, Phase 2+ + council-gated. +- **Deferred / gated:** D-LOTUS-3..9 (design/comma/temporal-seam/benchmarks/ + verdict); D-LOTUS-6 BLOCKED on the absent lance crate source (sandbox has + no registry copy; needs `cargo fetch` or sanctioned consult); + `temporal.rs:396`/`:410` retired-type doc-drift correction owed to the next + phase PR. +- **Docs:** EPIPHANIES `E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1`; + STATUS_BOARD `lotus-seal-fractal-commit-frontier` D-LOTUS-1..9. +- **Confidence:** high on the defect chain (every link read at file:line this + session; the doc-contradiction is quoted); the §6 amortization mechanisms + are INFERENCE until the Phase 5 benchmarks run. + ## 2026-08-12 — lance-graph #946 (MERGED) — D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix - **The finding that forced the run.** `substrate-comfort-zones-v1` §1's diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 6368f397..8a20c3e7 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -7,8 +7,8 @@ metaphor becomes a type name without council approval. | D-id | Deliverable | Status | |---|---|---| -| D-LOTUS-1 | LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology, 4 lenses, VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER graded; §6 answers the permeability/amortization question | **In PR** | -| D-LOTUS-2 | F-ORD-REAL-FALSIFIER.md + the pre-registered tests (GREEN defect pin + `#[ignore]`d RED falsifier, `cycle_driver.rs`) | **In PR** — RED lands BEFORE any fix, per charter | +| D-LOTUS-1 | LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology, 4 lenses, VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER graded; §6 answers the permeability/amortization question | **Shipped** — PR #961, 0cc171f | +| D-LOTUS-2 | F-ORD-REAL-FALSIFIER.md + the pre-registered tests (GREEN defect pin + `#[ignore]`d RED falsifier, `cycle_driver.rs`) | **Shipped** — PR #961, 0cc171f; RED landed BEFORE any fix, per charter | | D-LOTUS-3 | LOTUS-SEAL-DESIGN.md (16⁴ tree, petal seal, H(parent)=H(children) — NEW design per audit §2.3) | Queued (post-verdict) | | D-LOTUS-4 | COMMA-MODULATION-EXPERIMENT.md (petal-index-tier comma; locality-vs-resonance trade; "if it doesn't survive measurement, delete it") | Queued | | D-LOTUS-5 | TEMPORAL-FRONTIER-SEAM.md (rung-qualified frontier visibility; hindsight-invariance falsifier; audit §6.4) | Queued | From c872c2a1cc47bcb075e7a95bd3db361ff230aace Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 21:57:33 +0000 Subject: [PATCH 4/8] =?UTF-8?q?operator=20pin=20ruling:=20DF=2054.1=20only?= =?UTF-8?q?=20=E2=80=94=20remove=20broken=20delta=20feature;=20RP-SEAL=20c?= =?UTF-8?q?harter=20+=20boards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ruling (E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1): lance 9 / lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, always, across AdaWorldAPI forks, usually via [patch -> upstream repository git]. Measured before acting: the only DF-53 source was deltalake 0.32.4 (^53.1.0) behind the already-non-default, already-broken delta feature; registry check shows no DF-54 deltalake exists. Removed: delta feature, deltalake + url optional deps, DeltaTableReader (module docs + Cargo.toml carry dated removal notes; DataSourceFormat::Delta stays as a catalog tag). Post-removal Cargo.lock: exactly ONE datafusion = 54.1.0, zero deltalake entries. cargo check -p lance-graph green; fmt clean; full suite running as the PR gate. Docker pins surveyed: root + avx512 = Rust 1.97.1 + protobuf-compiler, no delta references — removal is docker-safe (stale flag noted: crates/symbiont/Dockerfile rust:1.95). Also: RP-SEAL research charter committed as .claude/plans/erasure-seals-compaction-research-v1.md (15-researcher program, independent pass dispatched as background workflow wf_ca974718-1b4; adversaries strongest-tier, builders/scouts grindwork tier); INTEGRATION_PLANS + STATUS_BOARD entries; D-LOTUS-6 BLOCKER lifted (operator-sanctioned upstream-git source consult; exact v9.0.0 tag on disk matching the lock checksum) with the audit carrying the dated lift note; CLAUDE.md 'BOTH MAJORS ARE REQUIRED' note superseded in place. Co-Authored-By: Claude --- .claude/board/EPIPHANIES.md | 43 + .claude/board/INTEGRATION_PLANS.md | 8 + .claude/board/STATUS_BOARD.md | 15 +- .../erasure-seals-compaction-research-v1.md | 272 +++ CLAUDE.md | 22 +- Cargo.lock | 1555 +++-------------- crates/lance-graph/Cargo.toml | 16 +- crates/lance-graph/src/lib.rs | 2 - crates/lance-graph/src/table_readers.rs | 91 +- docs/lotus/LOTUS-FRONTIER-AUDIT.md | 12 + 10 files changed, 615 insertions(+), 1421 deletions(-) create mode 100644 .claude/plans/erasure-seals-compaction-research-v1.md diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 8f0cfcee..6a5054b5 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,46 @@ +## 2026-08-18 — E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1 + +**Status:** RULING `[operator]` + same-day discharge, measured. + +**The ruling.** lance 9 / lancedb 0.33 / datafusion 54.1 (**no DF 53**) / +arrow 58 are ALWAYS pinned across AdaWorldAPI forks, usually consumed via +`[patch]` of the upstream repository git. DF 53 existed only for older syntax +in an out-of-scope research crate; all of lance-graph is DF-54-certified, +and single-major DF 54.1 is the discipline ("without DF 53/54 parallelism +it's much better, easier, faster to handle"). + +**Measured before acting:** the lock carried datafusion 53.1.0 AND 54.1.0; +the ONLY DF-53 source was `deltalake 0.32.4` (`^53.1.0`) behind the `delta` +feature — which was already NON-default and documented BROKEN (0.32 removed +`DeltaTableProvider::try_new`; the reader was never refactored). Checked the +registry: deltalake-core tops out at 0.32.4 / DF ^53.1.0 — no DF-54 deltalake +exists, so a bump could not discharge the ruling; removal could. + +**Discharged:** `delta` feature + `deltalake`/`url` optional deps + +`DeltaTableReader` removed (Cargo.toml carries the dated removal note; a +future Delta reader returns only with a DF-54-compatible deltalake, as its +own deliberate PR). `DataSourceFormat::Delta` (a catalog metadata tag) stays. +**Post-removal lock: exactly ONE datafusion = 54.1.0; zero deltalake +entries.** `cargo check -p lance-graph` green (needed `protobuf-compiler` +installed in-sandbox — lance's prost build; the Dockerfiles already install +it). Docker pins surveyed on request: root + avx512 Dockerfiles = Rust +1.97.1 + protobuf-compiler/cmake, no feature references to delta, versions +purely from Cargo.lock — removal is docker-safe. One stale flag (not +touched): `crates/symbiont/Dockerfile` still `rust:1.95-bookworm` +(pre-1.97.1-sweep residue; symbiont is excluded/standalone). + +**Second discharge — the lotus BLOCKER:** the [patch → upstream-git] +mechanism sanctioned consulting the upstream lance repository as source; the +exact v9.0.0 tag is cloned (`/tmp/sources/lance-9`, matching the Cargo.lock +checksum) plus current upstream (`/tmp/sources/lance-main`) for the RP-SEAL +two-column discipline. D-LOTUS-6's "lance source absent" BLOCKER is LIFTED; +capability findings deliberately enter through the RP-SEAL Domain-A +independent passes, not this entry (independence rule). + +**Supersedes in place:** CLAUDE.md's "BOTH MAJORS ARE REQUIRED — do NOT fix +Cargo.lock to one" datafusion note (written when `delta` was default and +load-bearing) — corrected in the same commit. + ## 2026-08-18 — E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1 **Status:** FINDING (VERIFIED at file:line) + pre-registered falsifier landed diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 60705ceb..e0582a6b 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,11 @@ +## 2026-08-18 — erasure-seals-compaction-research-v1 (RP-SEAL) — ACTIVE + +Operator-delivered 15-researcher program (5 domains x builder/adversary/ +scout) on bidirectional erasure-coded seals + locality-aware compaction in +versioned SoA grid stores. `.claude/plans/erasure-seals-compaction-research-v1.md`. +Independent pass dispatched as workflow `wf_ca974718-1b4`; consolidation + +tiered attack follow. Subsumes the lotus charter's phases 2-8. + ## 2026-08-13 — weather-soa-bake-v1 (PLAN; the missing Zarr→NodeRow bake) Plan: `.claude/plans/weather-soa-bake-v1.md`. **Does not supersede diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 8a20c3e7..edcc1ab2 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,16 @@ +## erasure-seals-compaction-research-v1 — RP-SEAL (operator charter, DISPATCHED 2026-08-18) + +Plan: `.claude/plans/erasure-seals-compaction-research-v1.md` (the operator's +15-researcher program; supersedes lotus deliverables 3-9's sequencing). +Independent pass = background workflow `wf_ca974718-1b4`: 5 domains +(A Lance storage / B spatial layout / C erasure coding / D temporal / +E SoA-cache) x 3 roles (builder/adversary/scout); adversaries on the +strongest tier, builders+scouts on the grindwork tier; first pass blind to +docs/lotus/** and all boards (independence rule). Consolidation (evidence +matrix + cross-domain graph) and the tiered attack pass run main-thread +AFTER all 15 reports land. Reports: session scratchpad `rp-seal-v1/`, +committed at consolidation as Appendix H. + ## lotus-seal-fractal-commit-frontier — research arc (PRE-REGISTERED 2026-08-18) Charter: operator, 2026-08-18 (research + architecture; "first audit the @@ -12,7 +25,7 @@ metaphor becomes a type name without council approval. | D-LOTUS-3 | LOTUS-SEAL-DESIGN.md (16⁴ tree, petal seal, H(parent)=H(children) — NEW design per audit §2.3) | Queued (post-verdict) | | D-LOTUS-4 | COMMA-MODULATION-EXPERIMENT.md (petal-index-tier comma; locality-vs-resonance trade; "if it doesn't survive measurement, delete it") | Queued | | D-LOTUS-5 | TEMPORAL-FRONTIER-SEAM.md (rung-qualified frontier visibility; hindsight-invariance falsifier; audit §6.4) | Queued | -| D-LOTUS-6 | PREPARED-ARTIFACT-PUBLICATION.md | **BLOCKED — lance crate source absent from sandbox** (audit §2.2); needs `cargo fetch` or sanctioned source consult | +| D-LOTUS-6 | PREPARED-ARTIFACT-PUBLICATION.md | **UNBLOCKED 2026-08-18** (operator pin ruling sanctioned the upstream-git source consult; exact v9.0.0 tag + current upstream on disk) — now routed through RP-SEAL Domain A rather than written solo, per the research charter | | D-LOTUS-7 | BENCHMARK-PLAN.md (seal/append split; cycle-major vs tile-major k∈{1,4,16}; F-AMORT version-range scaling; F-MEM RAM high-water freeze-vs-petal; viewport partial hydration) | Queued | | D-LOTUS-8 | PRE-REGISTERED-FALSIFIERS.md (the F-* catalogue) | Queued | | D-LOTUS-9 | Final verdict A-E | Queued — gates all implementation | diff --git a/.claude/plans/erasure-seals-compaction-research-v1.md b/.claude/plans/erasure-seals-compaction-research-v1.md new file mode 100644 index 00000000..24353dfb --- /dev/null +++ b/.claude/plans/erasure-seals-compaction-research-v1.md @@ -0,0 +1,272 @@ +# erasure-seals-compaction-research-v1 — RP-SEAL (operator charter, 2026-08-18) + +> **Status: DISPATCHED 2026-08-18** — independent pass running as a 15-agent +> background workflow (run `wf_ca974718-1b4`; reports land under the session +> scratchpad `rp-seal-v1/`, committed at consolidation as Appendix H). +> Tier allocation per the operator's "Opus for filigrane, Sonnet for +> grindwork": the five ADVERSARY cells (A2/B2/C2/D2/E2) = strongest tier; +> the five BUILDER and five SCOUT cells = grindwork tier. Consolidation +> (§12) and the final attack pass (§13) run on the main thread AFTER all 15 +> reports exist. This plan file is the operator's program, recorded as the +> arc's source of truth. It SUPERSEDES the sequencing of the earlier lotus +> charter's phases 2–8 (docs/lotus/ deliverables 3–9 now route through this +> program); the landed Phase 0/1 deliverables (audit + F-ORD-REAL falsifier, +> PR #961) stand as prior session output that the researchers deliberately +> do NOT read during the first pass (§11 independence). + +## Title + +An Experimental Study of Bidirectional Erasure-Coded Seals and +Locality-Aware Compaction in Versioned Structure-of-Arrays Grid Stores + +Subtitle: Morton-Cascade Amortization, Temporal Deinterlacing, Reed–Solomon +Repair Locality, and Index-Remapped Rewrite Policies in Lance + +## 0. Posture + +This is a research program, NOT an implementation sprint. Do not begin by +implementing the proposed architecture. Fifteen independent researchers: +5 research domains × 3 independent agents per domain. They FIRST investigate +independently, THEN consolidate evidence. Only AFTER consolidation may they +attack implementation / upstream candidate work. + +The objective is not to vindicate the proposed idea. The objective is to +determine which parts survive contact with: exact Lance source; measurements; +coding theory; storage-system literature; temporal/database literature; +cache/SoA behavior; adversarial falsification. Unexpected negative results +are high-value results. Actively search for previously unnoticed structural +equivalences, cross-domain correspondences, and counterexamples. Do NOT claim +novelty merely because no one on this team remembers prior art. Every +suspected novel pattern requires a prior-art search. + +## 1. Source of truth + +Repositories: AdaWorldAPI/lance-graph, AdaWorldAPI/OGAR, AdaWorldAPI/ndarray, +AdaWorldAPI/rustynum, lance upstream. Temporal reference: OGAR commit +`386a6fd848334b1d880c8408b3810f045d135cfe`, `docs/TEMPORAL-TIME-TRAVEL.md` — +read it literally. Temporal concepts: current tick / current Lance version; +last awareness tick / reference horizon; write tick; hindsight knowledge; +STRICT / AWARE / RETRO access; causal vs wall-clock order; possible HLC +coordinate. Do not reduce temporal.rs to an ordering utility: the research +hypothesis treats it as temporal deinterlacing `T_now × A_last × W_write` +with an interlacing window resembling `A_last < W_write <= T_now` — +investigate the actual implementation before relying on this shorthand. + +## 2. Strict Lance source discipline + +TWO evidence columns: (A) EXACT PINNED LANCE 9.0.0; (B) CURRENT LANCE +UPSTREAM. Never infer 9.0.0 capability from current upstream. Record per +claim: exact crate/version, Cargo.lock checksum, file, line, feature gate, +API stability, public API vs internal mechanism. Upstream appears to contain +machinery around: file compaction; reencode/binary-copy modes; incremental +compaction limits; excluded fragment boundaries; deferred index remapping; +fragment reuse indices; old-row-address → new-row-address remapping — +VERIFY all independently. + +Session anchors (dispatch note): column A at `/tmp/sources/lance-9` +(upstream tag v9.0.0; workspace Cargo.lock checksum +`23d04bed056e254bc6e31264b031c8492507ca57939586f016924081dcf221a9`); +column B at `/tmp/sources/lance-main` (cloned 2026-08-18). + +## 3. Baseline data geometry (hypotheses, not frozen architecture) + +One V3-style logical row = 512 B. Small work field = 4×4 rows = 16 rows = +8 KiB. Cycle = 4096 fields × 8 KiB = 32 MiB = 65,536 rows. 2-D hierarchy: +4=2×2, 16=4×4, 64=8×8, 256=16×16, 1024=32×32, 4096=64×64. Inverse +reduction: 4096→1024→256→64→16→4→1. + +## 4. Core distinctions + +Never conflate: semantic identity; logical grid locus; worker completion +order; semantic/write time; awareness time; Lance DatasetVersion; physical +fragment; physical row address; secondary-index row address; compaction +order; parity group. Arrival may determine WHEN work executes; it must not +accidentally determine semantic identity. Physical relocation must not change +logical identity. Reed–Solomon protection, if used, is initially defined over +canonical LOGICAL bytes/symbols, not encoded Lance page bytes. + +## 5. The 5×3 research matrix + +- **A — Lance storage/compaction**: A1 builder (map exact 9.0.0: + transactions, fragments, writer path, compaction planner, fragment IDs, + row IDs, stable IDs, index remapping, version publication, GC, prepared + writes); A2 adversary (destroy the thesis that random first-come chunk + placement is cheap: fragmentation, index-remap, page-locality, point-read, + range-read, long-retention failures); A3 upstream scout (map evolution + beyond 9.0.0; identify generic extension seams proposable upstream without + GridLake semantics). +- **B — Spatial layout / Morton cascade**: B1 builder (locality-key + generators: Morton, Hilbert, raster, random, project-native Morton/HHTL); + B2 adversary (adversarial query distributions: stripes, checkerboards, + diagonals, sparse hotspots, moving windows, skewed clusters, temporal + bands); B3 literature scout (SFCs, clustering metrics, cache-oblivious + layout, learned SFCs — do not assume Morton wins). +- **C — Erasure coding / seals**: C1 builder (checksum/hash; XOR P; RAID6/RS + P+Q; 2-D product codes; hierarchical cascade parity); C2 adversary (inject: + single/double erasure, silent corruption, wrong-slot substitution, stale + chunk, duplicate chunk, correlated/boundary/phase-aligned failures; measure + detect/localize/repair/falsely-accept); C3 coding-theory scout (RS repair, + MDS, LRC, regenerating codes, product/array codes, repair bandwidth, + subpacketization, repair locality — no home-grown ECC accepted merely + because tests pass). +- **D — Temporal deinterlacing**: D1 builder (map temporal.rs + OGAR + time-travel to versioned reads; model T_now × A_last × W_write; test + STRICT/AWARE/RETRO where implemented); D2 adversary (late facts, reordered + workers, same semantic write under different scheduler timing, + cross-version reads, stale awareness, clock skew, restart, cross-server + causal ambiguity; prove scheduler chronology does not become semantic + chronology); D3 literature scout (bitemporal DBs, transaction vs valid + time, snapshot isolation, causal consistency, HLC, temporal provenance, + retroactive queries). +- **E — SoA / cache / amortization**: E1 builder (benchmark harnesses: + 512 B rows, 8 KiB fields, cascade thresholds, first-come vs whole-64K, + canonical sort vs scatter vs index-remapped random placement); E2 + adversary (allocations, resident bytes, cache misses, bytes + copied/hashed/rewritten, fragment count, versions, index-remap cost, + read/write amplification, recovery cost; one experiment with compaction + disabled); E3 systems scout (LSM compaction design space, write + amplification, columnar reorganization, data skipping, clustering, + cache-oblivious structures, tiered/leveled, adaptive physical design). + +## 6. Required prior-art starting packet + +arXiv:2202.04522 (LSM compaction design space → trigger/layout/granularity/ +data-movement policy); arXiv:1206.3804 (LRC → repair locality, distance, +storage-rate tradeoff); arXiv:1509.04764 (Repairing RS → exact repair, +repair bandwidth, MDS constraints); arXiv:1806.04437 (EC for distributed +storage overview → LRC, regenerating, RS repair, access cost); +arXiv:1801.07399 (Onion Curve → SFC clustering metric, query-shape +dependence, counterexamples to Hilbert/Morton optimality); arXiv:2008.01684 +(SFCs for high-performance data mining → locality preservation, +cache-oblivious applications, transformation cost); arXiv:2009.06309 +(Data-Driven SFCs → adaptive ordering, multiscale/quadtree); +arXiv:1812.07123 (CausalSpartanX + HLC literature → causal timestamping, +visibility latency, clock anomalies, snapshot semantics). Find more; do not +stop at this packet. + +## 7. The phase/comma hypothesis + +"Pythagorean comma is ECC" is FORBIDDEN as a claim. The admissible +hypothesis: a deterministic incommensurate / phase-progressive schedule may +be useful for choosing an independent RS/product-code coefficient orientation +across spatial cascade levels. Candidate sources: comma-like phase +progression, QuintenZirkel, BGZ17/highheelBGZ arithmetic, φX, 11/17 +residues, CurveRuler. FIRST locate and characterize actual implementations; +then determine whether any candidate schedule (1) preserves matrix rank, +(2) preserves MDS/correction properties, (3) reduces cross-level syndrome +collisions, (4) improves localization, (5) costs ≤ SIMD work, (6) can be +reconstructed from locus/level instead of stored. Controls: standard RS +coefficients; sequential finite-field powers; random-but-seeded valid +coefficients; no modulation. If the project schedule loses to boring RS: +DELETE IT. + +## 8. GridLake / bidirectional code hypothesis + +Evaluate (not assume): A flat RS(k+2,k); B row P/Q + column P/Q; C product +code; D local P/Q + cascade parity; E hash-only syndromes; F payload parity; +G hybrids (strong per-chunk hashes + erasure parity). Report actual +redundancy; never call 12.5% "free"; distinguish CPU from storage cost. + +## 9. Compaction research questions (upstream must stay general) + +Locality-key ordered compaction; layout-aware rewrite grouping; optional +group boundaries; caller-provided sortable layout key; page/block permutation +when reencode is unnecessary; post-compaction permutation receipts; locality +statistics in CompactionMetrics; locality-debt trigger; stable logical row +IDs across physical moves; deferred physical clustering; incremental +locality repair. Do NOT upstream Morton/GridLake/Lotus/comma/temporal.rs/ +graph semantics where a generic u64 / expression / layout policy suffices. + +## 10. Cross-domain discovery mandate + +Every researcher looks for equivalences (questions, not conclusions): +compaction as locality "repair"; shared grouping for erasure repair and +query locality; temporal closure as compaction trigger; cascade completion +as spatial carry chain; fragment-reuse maps as permutation witnesses; +parity-group boundaries as compaction boundaries; locality debt from +index-remap entropy; one SFC order minimizing both query scatter and RS +repair scatter; prepared fragments as immutable code symbols before one +manifest publication; multi-scale syndromes identifying stale-version +contamination; temporal delta altering only a secondary syndrome +orientation; identity-derived layout keys making canonicalization +constructive. Classify every surprise: KNOWN / TRANSFER / NOVEL-CANDIDATE +(documented search required) / DISPROVED. + +## 11. Independence rule + +The three researchers inside one domain must NOT share intermediate +conclusions during the first pass. Each returns independently: SOURCE +ARCHAEOLOGY / MECHANISM / EXPECTED BENEFIT / EXPECTED FAILURE / EXPERIMENT / +PRIOR ART / SURPRISES / VERDICT. Only after all 15 reports exist do +consolidation agents see them. (Dispatch note: the first pass is also blind +to `docs/lotus/**` and all board files — prior sessions' conclusions would +synchronize the researchers.) + +## 12. Consolidation pass + +One evidence matrix: claim / source evidence / measurement evidence / +literature evidence / supporting agents / dissenting agents / confidence / +falsifier / next experiment. Then a cross-domain relationship graph. +Independently rediscovered equivalences are high-value; unanimity with +shared ancestry is NOT independent confirmation. + +## 13. Final attack pass (tiering) + +TIER 0 measurement/falsifiers only; TIER 1 lance-graph internal experiment; +TIER 2 generic Lance prototype; TIER 3 credible Lance upstream RFC/PR; +TIER 4 paper-worthy result. Every upstream candidate answers: general user +benefit; no lance-graph semantics in the API; backward compatibility; +effects on fragment format / indexes / compaction / binary-copy / object +storage / version retention; failure behavior; the benchmark that convinces +an upstream maintainer. + +## 14. Primary metrics + +logical bytes produced; physical bytes written; physical bytes rewritten; +write amplification; read amplification; point-read latency; +grid-neighborhood latency; random take latency; fragment count; files +touched/query; pages touched/query; index remap bytes; index remap CPU; +compaction CPU; compaction wall time; repair read bytes; repair write bytes; +repair helpers/chunks touched; seal CPU; seal metadata bytes; cache misses; +peak RSS; restart recovery work; version count; stale-version retention +bytes. Do not optimize a metric that was not measured. + +## 15. Falsifiers + +F-ARRIVAL (same semantic input, adversarial completion order → same +canonical result); F-LAYOUT (random physical placement → correct reads and +seals); F-COMPACT (before/after compaction → same logical payload, identity, +logical RS validity); F-RS (all promised erasure patterns reconstruct +exactly); F-SILENT (silent corruption detected before reconstruction); +F-PHASE (project phase progression performs no worse than standard baseline, +retains rank); F-NOPHASE (no measurable advantage → remove it); F-SFC +(Morton must defeat or justify itself against raster/Hilbert/Onion); +F-TEMPORAL (scheduler timing never changes semantic time); F-HINDSIGHT +(STRICT/AWARE/RETRO return the expected epistemic projections); F-NOCOMPACT +(healthy with compaction disabled for the soak window); F-AMPLIFY (locality +improvement justifies bytes rewritten); F-UPSTREAM (generic proposal useful +with GridLake completely absent). + +## 16. Final paper-shaped deliverable + +Title / Abstract / 1 Problem / 2 Background / 3 Lance archaeology / +4 Temporal model / 5 Grid-SoA model / 6 Erasure-code model / 7 +Locality-layout model / 8 Methodology / 9 Results / 10 Negative results / +11 Cross-domain discoveries / 12 Prior art / 13 Novelty audit / 14 Candidate +Lance extensions / 15 Threats to validity / 16 Conclusion. Appendices: +A source provenance; B falsifiers; C benchmarks; D coding matrices; +E locality curves; F compaction traces; G temporal traces; H agent +disagreement record. Dry systems-paper prose. Avoid "Lotus magic" / +"holographic" / "quantum" / "time machine" / "perfect random access" / +"self-healing" unless quantitatively demonstrated. Preferred vocabulary: +multi-scale, locality-preserving, version-aware, temporal-reference horizon, +incremental closure, logical-to-physical remapping, erasure-coded seal, +product-code, Reed–Solomon, space-filling curve, rewrite amplification, +repair locality, layout debt, compaction policy, structure-of-arrays, +prepared publication, causal timestamp, bidirectional syndrome, hierarchical +locality. + +## 17. Research maxim + +DO NOT MAKE THE BEAUTIFUL IDEA TRUE. MAKE IT SURVIVE FIFTEEN PEOPLE TRYING +TO MAKE IT FALSE. diff --git a/CLAUDE.md b/CLAUDE.md index f4ca8263..1cb6871e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1163,13 +1163,18 @@ datafusion = "54" # OUR direct pin, in every crate that DEPENDS on it # `datafusion = []` under [features] — a FEATURE NAME, not a # version pin. It is not a counter-example.) # - # ⚠ BOTH MAJORS ARE REQUIRED — do NOT "fix" Cargo.lock to one. - # `datafusion 53.1.0` (+ -datasource, -physical-expr-adapter) - # also resolves because `deltalake-core 0.32.4` pins DF 53 - # upstream, and it backs the optional `delta` feature. Two - # semver majors legitimately coexist; collapsing them breaks - # `delta`. 53 is NOT a stale leftover — it is a live upstream - # constraint that lifts only when deltalake moves to DF 54. + # ⊘ SUPERSEDED 2026-08-18 (operator pin ruling, + # E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1): the + # "BOTH MAJORS ARE REQUIRED" note that stood here is + # RETIRED. DF 53 resolved only through deltalake 0.32 + # behind the (already non-default, already broken) + # `delta` feature; deltalake tops out at 0.32.4/DF^53, + # so the feature was REMOVED and the lock now carries + # exactly ONE datafusion = 54.1.0. The ruling: lance 9 / + # lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, + # always, across AdaWorldAPI forks. A Delta reader + # returns only with a DF-54-compatible deltalake, as its + # own deliberate PR. # Probe: .claude/plans/lance9-datafusion54-upgrade-probe-v1.md lance = "=9.0.0" # exact-pinned: lancedb 0.33.0 requires lance =9.0.0 lance-linalg = "=9.0.0" @@ -1181,7 +1186,8 @@ rust = "1.97.1" # rust-toolchain.toml is authoritative; see its bump l ndarray = { path = "../../../ndarray" } # AdaWorldAPI fork, default, optional fallback nom = "7.1" snafu = "0.8" -deltalake = "0.32" # optional +# deltalake — REMOVED 2026-08-18 with the `delta` feature (see the datafusion +# supersession note above; it was the sole DF-53 source). ``` --- diff --git a/Cargo.lock b/Cargo.lock index 68b711c4..e5803888 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -363,7 +363,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed" dependencies = [ "bitflags 2.13.0", - "serde", "serde_core", "serde_json", ] @@ -608,31 +607,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "aws-sdk-dynamodb" -version = "1.116.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c937c55ae3030bec4431c0d9146e33d2b3e5f54bb47ed32597068200c56affc" -dependencies = [ - "arc-swap", - "aws-credential-types", - "aws-runtime", - "aws-smithy-async", - "aws-smithy-http", - "aws-smithy-json", - "aws-smithy-observability", - "aws-smithy-runtime", - "aws-smithy-runtime-api", - "aws-smithy-types", - "aws-types", - "bytes", - "fastrand", - "http 0.2.12", - "http 1.4.2", - "regex-lite", - "tracing", -] - [[package]] name = "aws-sdk-sso" version = "1.102.0" @@ -892,7 +866,6 @@ dependencies = [ "base64-simd", "bytes", "bytes-utils", - "futures-core", "http 0.2.12", "http 1.4.2", "http-body 0.4.6", @@ -905,8 +878,6 @@ dependencies = [ "ryu", "serde", "time", - "tokio", - "tokio-util", ] [[package]] @@ -1252,50 +1223,6 @@ version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" -[[package]] -name = "buoyant_kernel" -version = "0.22.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2235eb320cd7178862a32dd111bd0c0f71a368e393add4914c50129add478eab" -dependencies = [ - "arrow", - "buoyant_kernel_derive", - "bytes", - "chrono", - "crc", - "futures", - "indexmap 2.14.0", - "itertools 0.14.0", - "object_store", - "parquet", - "percent-encoding", - "rand 0.9.4", - "reqwest 0.13.4", - "roaring", - "rustc_version", - "serde", - "serde_json", - "strum 0.27.2", - "thiserror 2.0.18", - "tokio", - "tracing", - "tracing-subscriber", - "url", - "uuid", - "z85", -] - -[[package]] -name = "buoyant_kernel_derive" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3448e05bba811d98c73a466843abd8c16d0416dc083f92b66847693fcb0714f4" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "bytecheck" version = "0.8.2" @@ -1703,15 +1630,6 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" -[[package]] -name = "convert_case" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db05ffb6856bf0ecdf6367558a76a0e8a77b1713044eb92845c692100ed50190" -dependencies = [ - "unicode-segmentation", -] - [[package]] name = "core-foundation" version = "0.9.4" @@ -1765,21 +1683,6 @@ dependencies = [ "libc", ] -[[package]] -name = "crc" -version = "3.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" -dependencies = [ - "crc-catalog", -] - -[[package]] -name = "crc-catalog" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" - [[package]] name = "crc32c" version = "0.6.8" @@ -1941,19 +1844,8 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "424e0138278faeb2b401f174ad17e715c829512d74f3d1e81eb43365c2e0590e" dependencies = [ - "ctor-proc-macro 0.0.7", - "dtor 0.1.1", -] - -[[package]] -name = "ctor" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83cf0d42651b16c6dfe68685716d18480d18a9c39c62d76e8cf3eb6ed5d8bcbf" -dependencies = [ - "ctor-proc-macro 0.0.13", - "dtor 0.8.1", - "link-section 0.2.1", + "ctor-proc-macro", + "dtor", ] [[package]] @@ -1962,7 +1854,7 @@ version = "1.0.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d83cb7e7a873830708d6b02a78cd36a592c6fa14bf267b68725103b85c0d77f" dependencies = [ - "link-section 0.19.2", + "link-section", "linktime-proc-macro", ] @@ -1972,12 +1864,6 @@ version = "0.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52560adf09603e58c9a7ee1fe1dcb95a16927b17c127f0ac02d6e768a0e25bc1" -[[package]] -name = "ctor-proc-macro" -version = "0.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a949c44fcacbbbb7ada007dc7acb34603dd97cd47de5d054f2b6493ecebb483" - [[package]] name = "ctutils" version = "0.4.2" @@ -2082,61 +1968,6 @@ version = "2.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" -[[package]] -name = "datafusion" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93db0e623840612f7f2cd757f7e8a8922064192363732c88692e0870016e141b" -dependencies = [ - "arrow", - "arrow-schema", - "async-trait", - "bytes", - "bzip2", - "chrono", - "datafusion-catalog 53.1.0", - "datafusion-catalog-listing 53.1.0", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-datasource-arrow 53.1.0", - "datafusion-datasource-csv 53.1.0", - "datafusion-datasource-json 53.1.0", - "datafusion-datasource-parquet 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-functions-aggregate 53.1.0", - "datafusion-functions-nested 53.1.0", - "datafusion-functions-table 53.1.0", - "datafusion-functions-window 53.1.0", - "datafusion-optimizer 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-optimizer 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "datafusion-sql 53.1.0", - "flate2", - "futures", - "itertools 0.14.0", - "liblzma", - "log", - "object_store", - "parking_lot", - "parquet", - "rand 0.9.4", - "regex", - "sqlparser 0.61.0", - "tempfile", - "tokio", - "url", - "uuid", - "zstd", -] - [[package]] name = "datafusion" version = "54.1.0" @@ -2148,31 +1979,31 @@ dependencies = [ "async-trait", "bzip2", "chrono", - "datafusion-catalog 54.1.0", - "datafusion-catalog-listing 54.1.0", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-datasource-arrow 54.1.0", - "datafusion-datasource-csv 54.1.0", - "datafusion-datasource-json 54.1.0", - "datafusion-datasource-parquet 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate 54.1.0", - "datafusion-functions-nested 54.1.0", - "datafusion-functions-table 54.1.0", - "datafusion-functions-window 54.1.0", - "datafusion-optimizer 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-optimizer 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", - "datafusion-sql 54.1.0", + "datafusion-catalog", + "datafusion-catalog-listing", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-datasource-parquet", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-nested", + "datafusion-functions-table", + "datafusion-functions-window", + "datafusion-optimizer", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-optimizer", + "datafusion-physical-plan", + "datafusion-session", + "datafusion-sql", "flate2", "futures", "indexmap 2.14.0", @@ -2182,7 +2013,7 @@ dependencies = [ "object_store", "parking_lot", "parquet", - "sqlparser 0.62.0", + "sqlparser", "tempfile", "tokio", "url", @@ -2190,31 +2021,6 @@ dependencies = [ "zstd", ] -[[package]] -name = "datafusion-catalog" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37cefde60b26a7f4ff61e9d2ff2833322f91df2b568d7238afe67bde5bdffb66" -dependencies = [ - "arrow", - "async-trait", - "dashmap", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "itertools 0.14.0", - "log", - "object_store", - "parking_lot", - "tokio", -] - [[package]] name = "datafusion-catalog" version = "54.1.0" @@ -2224,14 +2030,14 @@ dependencies = [ "arrow", "async-trait", "dashmap", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-session", "futures", "itertools 0.14.0", "log", @@ -2240,29 +2046,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-catalog-listing" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17e112307715d6a7a331111a4c2330ff54bc237183511c319e3708a4cff431fb" -dependencies = [ - "arrow", - "async-trait", - "datafusion-catalog 53.1.0", - "datafusion-common 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "futures", - "itertools 0.14.0", - "log", - "object_store", -] - [[package]] name = "datafusion-catalog-listing" version = "54.1.0" @@ -2271,46 +2054,21 @@ checksum = "f0668fb32c12065ec242be0e5b4bc62bd7a06a0be3ecd83791ef877e4be67e02" dependencies = [ "arrow", "async-trait", - "datafusion-catalog 54.1.0", - "datafusion-common 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-catalog", + "datafusion-common", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", "futures", "itertools 0.14.0", "log", "object_store", ] -[[package]] -name = "datafusion-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d72a11ca44a95e1081870d3abb80c717496e8a7acb467a1d3e932bb636af5cc2" -dependencies = [ - "ahash", - "arrow", - "arrow-ipc", - "chrono", - "half", - "hashbrown 0.16.1", - "indexmap 2.14.0", - "itertools 0.14.0", - "libc", - "log", - "object_store", - "parquet", - "paste", - "recursive", - "sqlparser 0.61.0", - "tokio", - "web-time", -] - [[package]] name = "datafusion-common" version = "54.1.0" @@ -2331,23 +2089,12 @@ dependencies = [ "object_store", "parquet", "recursive", - "sqlparser 0.62.0", + "sqlparser", "tokio", "uuid", "web-time", ] -[[package]] -name = "datafusion-common-runtime" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89f4afaed29670ec4fd6053643adc749fe3f4bc9d1ce1b8c5679b22c67d12def" -dependencies = [ - "futures", - "log", - "tokio", -] - [[package]] name = "datafusion-common-runtime" version = "54.1.0" @@ -2359,41 +2106,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-datasource" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9fb386e1691355355a96419978a0022b7947b44d4a24a6ea99f00b6b485cbb6" -dependencies = [ - "arrow", - "async-compression", - "async-trait", - "bytes", - "bzip2", - "chrono", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "flate2", - "futures", - "glob", - "itertools 0.14.0", - "liblzma", - "log", - "object_store", - "rand 0.9.4", - "tokio", - "tokio-util", - "url", - "zstd", -] - [[package]] name = "datafusion-datasource" version = "54.1.0" @@ -2406,15 +2118,15 @@ dependencies = [ "bytes", "bzip2", "chrono", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "flate2", "futures", "glob", @@ -2430,30 +2142,6 @@ dependencies = [ "zstd", ] -[[package]] -name = "datafusion-datasource-arrow" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffa6c52cfed0734c5f93754d1c0175f558175248bf686c944fb05c373e5fc096" -dependencies = [ - "arrow", - "arrow-ipc", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "itertools 0.14.0", - "object_store", - "tokio", -] - [[package]] name = "datafusion-datasource-arrow" version = "54.1.0" @@ -2464,43 +2152,20 @@ dependencies = [ "arrow-ipc", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "futures", "itertools 0.14.0", "object_store", "tokio", ] -[[package]] -name = "datafusion-datasource-csv" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "503f29e0582c1fc189578d665ff57d9300da1f80c282777d7eb67bb79fb8cdca" -dependencies = [ - "arrow", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "object_store", - "regex", - "tokio", -] - [[package]] name = "datafusion-datasource-csv" version = "54.1.0" @@ -2510,44 +2175,20 @@ dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "futures", "object_store", "regex", "tokio", ] -[[package]] -name = "datafusion-datasource-json" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e33804749abc8d0c8cb7473228483cb8070e524c6f6086ee1b85a64debe2b3d2" -dependencies = [ - "arrow", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "object_store", - "serde_json", - "tokio", - "tokio-stream", -] - [[package]] name = "datafusion-datasource-json" version = "54.1.0" @@ -2557,50 +2198,20 @@ dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "futures", "object_store", "tokio", "tokio-stream", ] -[[package]] -name = "datafusion-datasource-parquet" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a8e0365e0e08e8ff94d912f0ababcf9065a1a304018ba90b1fc83c855b4997" -dependencies = [ - "arrow", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-pruning 53.1.0", - "datafusion-session 53.1.0", - "futures", - "itertools 0.14.0", - "log", - "object_store", - "parking_lot", - "parquet", - "tokio", -] - [[package]] name = "datafusion-datasource-parquet" version = "54.1.0" @@ -2610,19 +2221,19 @@ dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-pruning 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "datafusion-session", "futures", "itertools 0.14.0", "log", @@ -2632,41 +2243,12 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-doc" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8de6ac0df1662b9148ad3c987978b32cbec7c772f199b1d53520c8fa764a87ee" - [[package]] name = "datafusion-doc" version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d69bb69d8769e34f76839c960dbde24c1ac0c885a79b6c3c2287bdc56ec67891" -[[package]] -name = "datafusion-execution" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c03c7fbdaefcca4ef6ffe425a5fc2325763bfb426599bb0bf4536466efabe709" -dependencies = [ - "arrow", - "arrow-buffer", - "async-trait", - "chrono", - "dashmap", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "futures", - "log", - "object_store", - "parking_lot", - "rand 0.9.4", - "tempfile", - "url", -] - [[package]] name = "datafusion-execution" version = "54.1.0" @@ -2677,9 +2259,9 @@ dependencies = [ "arrow-buffer", "async-trait", "dashmap", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr-common", "futures", "log", "object_store", @@ -2689,29 +2271,6 @@ dependencies = [ "url", ] -[[package]] -name = "datafusion-expr" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "574b9b6977fedbd2a611cbff12e5caf90f31640ad9dc5870f152836d94bad0dd" -dependencies = [ - "arrow", - "async-trait", - "chrono", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-functions-window-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "indexmap 2.14.0", - "itertools 0.14.0", - "paste", - "recursive", - "serde_json", - "sqlparser 0.61.0", -] - [[package]] name = "datafusion-expr" version = "54.1.0" @@ -2722,30 +2281,17 @@ dependencies = [ "arrow-schema", "async-trait", "chrono", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-functions-window-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr-common", "indexmap 2.14.0", "itertools 0.14.0", "recursive", "serde_json", - "sqlparser 0.62.0", -] - -[[package]] -name = "datafusion-expr-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d7c3adf3db8bf61e92eb90cb659c8e8b734593a8f7c8e12a843c7ddba24b87e" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "indexmap 2.14.0", - "itertools 0.14.0", - "paste", + "sqlparser", ] [[package]] @@ -2755,43 +2301,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7b19a8c95522bee8cbb313d74263b85e355d2b52f42e67ef5694bf5de9e9356" dependencies = [ "arrow", - "datafusion-common 54.1.0", + "datafusion-common", "indexmap 2.14.0", "itertools 0.14.0", ] -[[package]] -name = "datafusion-functions" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f28aa4e10384e782774b10e72aca4d93ef7b31aa653095d9d4536b0a3dbc51b6" -dependencies = [ - "arrow", - "arrow-buffer", - "base64", - "blake2", - "blake3", - "chrono", - "chrono-tz", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-macros 53.1.0", - "hex", - "itertools 0.14.0", - "log", - "md-5 0.10.6", - "memchr", - "num-traits", - "rand 0.9.4", - "regex", - "sha2 0.10.9", - "unicode-segmentation", - "uuid", -] - [[package]] name = "datafusion-functions" version = "54.1.0" @@ -2805,13 +2319,13 @@ dependencies = [ "blake3", "chrono", "chrono-tz", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-macros", + "datafusion-physical-expr-common", "hex", "itertools 0.14.0", "log", @@ -2824,28 +2338,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "datafusion-functions-aggregate" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00aa6217e56098ba84e0a338176fe52f0a84cca398021512c6c8c5eff806d0ad" -dependencies = [ - "ahash", - "arrow", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-macros 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "half", - "log", - "num-traits", - "paste", -] - [[package]] name = "datafusion-functions-aggregate" version = "54.1.0" @@ -2853,33 +2345,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89bc17041e424a47ed062f43df24d84aab8b57c4c3221e5c1a5eef46d6c5718b" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "foldhash 0.2.0", "half", "log", "num-traits", ] -[[package]] -name = "datafusion-functions-aggregate-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b511250349407db7c43832ab2de63f5557b19a20dfd236b39ca2c04468b50d47" -dependencies = [ - "ahash", - "arrow", - "datafusion-common 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", -] - [[package]] name = "datafusion-functions-aggregate-common" version = "54.1.0" @@ -2887,34 +2366,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97dd2a9e865c6108059f5b37b77934f84b50bfb108f837bd0e5c9536e03f0545" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", -] - -[[package]] -name = "datafusion-functions-nested" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef13a858e20d50f0a9bb5e96e7ac82b4e7597f247515bccca4fdd2992df0212a" -dependencies = [ - "arrow", - "arrow-ord", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-functions-aggregate 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-macros 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "hashbrown 0.16.1", - "itertools 0.14.0", - "itoa", - "log", - "paste", + "datafusion-common", + "datafusion-expr-common", + "datafusion-physical-expr-common", ] [[package]] @@ -2925,16 +2379,16 @@ checksum = "75f0bdfeef16d96417b9632ef855645376b242e9006a126dfd0bedfc54a93f5f" dependencies = [ "arrow", "arrow-ord", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr-common", "hashbrown 0.17.1", "itertools 0.14.0", "itoa", @@ -2942,22 +2396,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "datafusion-functions-table" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b40d3f5bbb3905f9ccb1ce9485a9595c77b69758a7c24d3ba79e334ff51e7e" -dependencies = [ - "arrow", - "async-trait", - "datafusion-catalog 53.1.0", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-plan 53.1.0", - "parking_lot", - "paste", -] - [[package]] name = "datafusion-functions-table" version = "54.1.0" @@ -2966,32 +2404,14 @@ checksum = "f4e4941673c917819616877e9993da4503e4f4739812be0bc32c5356184c6383" dependencies = [ "arrow", "async-trait", - "datafusion-catalog 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-catalog", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", "parking_lot", ] -[[package]] -name = "datafusion-functions-window" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4e88ec9d57c9b685d02f58bfee7be62d72610430ddcedb82a08e5d9925dbfb6" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-window-common 53.1.0", - "datafusion-macros 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "log", - "paste", -] - [[package]] name = "datafusion-functions-window" version = "54.1.0" @@ -2999,45 +2419,24 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "12dd2e16c12b84b6f6b41b19f55b366dd1c46876bb35b86896c6349067379e8d" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-window-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-expr", + "datafusion-functions-window-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "log", ] -[[package]] -name = "datafusion-functions-window-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8307bb93519b1a91913723a1130cfafeee3f72200d870d88e91a6fc5470ede5c" -dependencies = [ - "datafusion-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", -] - [[package]] name = "datafusion-functions-window-common" version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cdc5e4b6f8b6ef823cc1c761f85088ad4c884fe8df64df3cbcc6b2b84698441" dependencies = [ - "datafusion-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", -] - -[[package]] -name = "datafusion-macros" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e367e6a71051d0ebdd29b2f85d12059b38b1d1f172c6906e80016da662226bd" -dependencies = [ - "datafusion-doc 53.1.0", - "quote", - "syn", + "datafusion-common", + "datafusion-physical-expr-common", ] [[package]] @@ -3046,31 +2445,11 @@ version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1a3614234dd93578c92428cb4f408e020874f0d2b7e6c90c928d9d28b5df2ceb" dependencies = [ - "datafusion-doc 54.1.0", + "datafusion-doc", "quote", "syn", ] -[[package]] -name = "datafusion-optimizer" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e929015451a67f77d9d8b727b2bf3a40c4445fdef6cdc53281d7d97c76888ace" -dependencies = [ - "arrow", - "chrono", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "indexmap 2.14.0", - "itertools 0.14.0", - "log", - "recursive", - "regex", - "regex-syntax", -] - [[package]] name = "datafusion-optimizer" version = "54.1.0" @@ -3079,10 +2458,10 @@ checksum = "a0635620b050b81bb92764e99250868f654e2cd5ad1bece413283b3f73c83179" dependencies = [ "arrow", "chrono", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", "indexmap 2.14.0", "itertools 0.14.0", "log", @@ -3091,30 +2470,6 @@ dependencies = [ "regex-syntax", ] -[[package]] -name = "datafusion-physical-expr" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b1e68aba7a4b350401cfdf25a3d6f989ad898a7410164afe9ca52080244cb59" -dependencies = [ - "ahash", - "arrow", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "half", - "hashbrown 0.16.1", - "indexmap 2.14.0", - "itertools 0.14.0", - "parking_lot", - "paste", - "petgraph 0.8.3", - "recursive", - "tokio", -] - [[package]] name = "datafusion-physical-expr" version = "54.1.0" @@ -3122,11 +2477,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8cabf7a86eb70b816729e33c81bf7767c936ee1226f607a114f5dac2decac8d0" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr-common", "half", "hashbrown 0.17.1", "indexmap 2.14.0", @@ -3137,21 +2492,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-physical-expr-adapter" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea22315f33cf2e0adc104e8ec42e285f6ed93998d565c65e82fec6a9ee9f9db4" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "itertools 0.14.0", -] - [[package]] name = "datafusion-physical-expr-adapter" version = "54.1.0" @@ -3159,31 +2499,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "de222e04f7e6744501555a54ab0abe26bfdfebee380af79a9bdc175704246859" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "itertools 0.14.0", ] -[[package]] -name = "datafusion-physical-expr-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b04b45ea8ad3ac2d78f2ea2a76053e06591c9629c7a603eda16c10649ecf4362" -dependencies = [ - "ahash", - "arrow", - "chrono", - "datafusion-common 53.1.0", - "datafusion-expr-common 53.1.0", - "hashbrown 0.16.1", - "indexmap 2.14.0", - "itertools 0.14.0", - "parking_lot", -] - [[package]] name = "datafusion-physical-expr-common" version = "54.1.0" @@ -3192,8 +2515,8 @@ checksum = "72d0d0057fc5a502d45c870cb6d47c66eb7bdd5edb1bd71ad6f3f724975ac2a8" dependencies = [ "arrow", "chrono", - "datafusion-common 54.1.0", - "datafusion-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr-common", "hashbrown 0.17.1", "indexmap 2.14.0", "itertools 0.14.0", @@ -3202,73 +2525,22 @@ dependencies = [ ] [[package]] -name = "datafusion-physical-optimizer" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cb13397809a425918f608dfe8653f332015a3e330004ab191b4404187238b95" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-pruning 53.1.0", - "itertools 0.14.0", - "recursive", -] - -[[package]] -name = "datafusion-physical-optimizer" -version = "54.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86046eed10950c5f9aaed9acfd148e9bd2e1dfdfe4f9aef607d1447b271e4183" -dependencies = [ - "arrow", - "datafusion-common 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-pruning 54.1.0", - "itertools 0.14.0", - "recursive", -] - -[[package]] -name = "datafusion-physical-plan" -version = "53.1.0" +name = "datafusion-physical-optimizer" +version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5edc023675791af9d5fb4cc4c24abf5f7bd3bd4dcf9e5bd90ea1eff6976dcc79" +checksum = "86046eed10950c5f9aaed9acfd148e9bd2e1dfdfe4f9aef607d1447b271e4183" dependencies = [ - "ahash", "arrow", - "arrow-ord", - "arrow-schema", - "async-trait", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-functions-window-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "futures", - "half", - "hashbrown 0.16.1", - "indexmap 2.14.0", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", "itertools 0.14.0", - "log", - "num-traits", - "parking_lot", - "pin-project-lite", - "tokio", + "recursive", ] [[package]] @@ -3283,15 +2555,15 @@ dependencies = [ "arrow-ord", "arrow-schema", "async-trait", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-functions-window-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "futures", "half", "hashbrown 0.17.1", @@ -3304,62 +2576,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-proto" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a387aaef949dc16bb6abc81bd1af850ec7449183aef011214f9724957495738" -dependencies = [ - "arrow", - "chrono", - "datafusion-catalog 53.1.0", - "datafusion-catalog-listing 53.1.0", - "datafusion-common 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-datasource-arrow 53.1.0", - "datafusion-datasource-csv 53.1.0", - "datafusion-datasource-json 53.1.0", - "datafusion-datasource-parquet 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-table 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-proto-common", - "object_store", - "prost 0.14.4", - "rand 0.9.4", -] - -[[package]] -name = "datafusion-proto-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16e614c7c53a9c304c6a850b821010bb492e57300311835f1180613f9d2c63d9" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "prost 0.14.4", -] - -[[package]] -name = "datafusion-pruning" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac8c76860e355616555081cab5968cec1af7a80701ff374510860bcd567e365a" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "itertools 0.14.0", - "log", -] - [[package]] name = "datafusion-pruning" version = "54.1.0" @@ -3367,29 +2583,15 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eb63eeac6de19be40f487b65dd84e546195f783c5a9928618e0c4f2a3569b0d7" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", "log", ] -[[package]] -name = "datafusion-session" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5412111aa48e2424ba926112e192f7a6b7e4ccb450145d25ce5ede9f19dc491e" -dependencies = [ - "async-trait", - "datafusion-common 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-plan 53.1.0", - "parking_lot", -] - [[package]] name = "datafusion-session" version = "54.1.0" @@ -3397,32 +2599,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f961d209177f91bd014db5cbb2c33b7d28a2597b9003e77f17aeb712964315a" dependencies = [ "async-trait", - "datafusion-common 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-plan", "parking_lot", ] -[[package]] -name = "datafusion-sql" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa0d133ddf8b9b3b872acac900157f783e7b879fe9a6bccf389abebbfac45ec1" -dependencies = [ - "arrow", - "bigdecimal", - "chrono", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-nested 53.1.0", - "indexmap 2.14.0", - "log", - "recursive", - "regex", - "sqlparser 0.61.0", -] - [[package]] name = "datafusion-sql" version = "54.1.0" @@ -3432,14 +2615,14 @@ dependencies = [ "arrow", "bigdecimal", "chrono", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-nested 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-nested", "indexmap 2.14.0", "log", "recursive", "regex", - "sqlparser 0.62.0", + "sqlparser", ] [[package]] @@ -3500,146 +2683,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "deltalake" -version = "0.32.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2695b91fd02afd5b4726a190fd58019674492d824e30598391390b0307644da8" -dependencies = [ - "buoyant_kernel", - "ctor 0.10.1", - "deltalake-aws", - "deltalake-azure", - "deltalake-core", - "deltalake-gcp", -] - -[[package]] -name = "deltalake-aws" -version = "0.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b3ef837d5d58bb0cd62dc9ee3bc201805751090f4bab2ca0ff09669661814fd4" -dependencies = [ - "async-trait", - "aws-config", - "aws-credential-types", - "aws-sdk-dynamodb", - "aws-sdk-sts", - "aws-smithy-runtime-api", - "backon", - "bytes", - "chrono", - "deltalake-core", - "futures", - "object_store", - "regex", - "thiserror 2.0.18", - "tokio", - "tracing", - "typed-builder", - "url", - "uuid", -] - -[[package]] -name = "deltalake-azure" -version = "0.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1b7099351f8d452374d75eda599cd7633a50f4a36ec318532ec6c5d1f9d908d" -dependencies = [ - "bytes", - "deltalake-core", - "object_store", - "thiserror 2.0.18", - "tokio", - "url", -] - -[[package]] -name = "deltalake-core" -version = "0.32.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4588e95ff3b2ccdba56d9ec262bd3467c0593000f729402528706f62be8be1ca" -dependencies = [ - "arrow", - "arrow-arith", - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-ipc", - "arrow-json", - "arrow-ord", - "arrow-row", - "arrow-schema", - "arrow-select", - "async-trait", - "buoyant_kernel", - "bytes", - "cfg-if 1.0.4", - "chrono", - "dashmap", - "datafusion 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-proto", - "deltalake-derive", - "dirs", - "either", - "futures", - "humantime", - "indexmap 2.14.0", - "itertools 0.14.0", - "num_cpus", - "object_store", - "parking_lot", - "parquet", - "percent-encoding", - "percent-encoding-rfc3986", - "pin-project-lite", - "rand 0.10.1", - "regex", - "serde", - "serde_json", - "sqlparser 0.61.0", - "strum 0.27.2", - "thiserror 2.0.18", - "tokio", - "tracing", - "url", - "uuid", - "validator", -] - -[[package]] -name = "deltalake-derive" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d5a646e2b05608db1e5f91e75b6af841d92294ea0864789bb01f3f818e5f7b25" -dependencies = [ - "convert_case", - "itertools 0.14.0", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "deltalake-gcp" -version = "0.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "909d4c00c77ed1359e2d3b6a25641740f7a068a8c20e951e479e1212e67004ea" -dependencies = [ - "async-trait", - "bytes", - "deltalake-core", - "futures", - "object_store", - "thiserror 2.0.18", - "tokio", - "tracing", - "url", -] - [[package]] name = "der" version = "0.7.10" @@ -3771,16 +2814,7 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "404d02eeb088a82cfd873006cb713fe411306c7d182c344905e101fb1167d301" dependencies = [ - "dtor-proc-macro 0.0.6", -] - -[[package]] -name = "dtor" -version = "0.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edf234dd1594d6dd434a8fb8cada51ddbbc593e40e4a01556a0b31c62da2775b" -dependencies = [ - "dtor-proc-macro 0.0.13", + "dtor-proc-macro", ] [[package]] @@ -3789,12 +2823,6 @@ version = "0.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f678cf4a922c215c63e0de95eb1ff08a958a81d47e485cf9da1e27bf6305cfa5" -[[package]] -name = "dtor-proc-macro" -version = "0.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2647271c92754afcb174e758003cfd1cbf1e43e5a7853d7b1813e63e19e39a73" - [[package]] name = "dunce" version = "1.0.5" @@ -4230,7 +3258,7 @@ dependencies = [ "arrow-array", "arrow-buffer", "arrow-schema", - "datafusion 54.1.0", + "datafusion", "geo", "geo-traits", "geoarrow-array", @@ -5275,11 +4303,11 @@ dependencies = [ "crossbeam-queue", "crossbeam-skiplist", "dashmap", - "datafusion 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-plan", "either", "fst", "futures", @@ -5398,8 +4426,8 @@ dependencies = [ "async-trait", "byteorder", "bytes", - "datafusion-common 54.1.0", - "datafusion-sql 54.1.0", + "datafusion-common", + "datafusion-sql", "futures", "itertools 0.14.0", "lance-arrow", @@ -5440,10 +4468,10 @@ dependencies = [ "arrow-select", "async-trait", "chrono", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-functions", + "datafusion-physical-expr", "futures", "jsonb", "lance-arrow", @@ -5541,7 +4569,7 @@ dependencies = [ "async-trait", "byteorder", "bytes", - "datafusion-common 54.1.0", + "datafusion-common", "futures", "lance-arrow", "lance-core", @@ -5563,7 +4591,7 @@ version = "9.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97739c10f2c818ac4aa8a798a115e11c790c174d016c0836792aa3b8d0f981ab" dependencies = [ - "datafusion 54.1.0", + "datafusion", "geo-traits", "geo-types", "geoarrow-array", @@ -5586,13 +4614,12 @@ dependencies = [ "bytes", "causal-edge", "chrono", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-aggregate 54.1.0", - "datafusion-sql 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-aggregate", + "datafusion-sql", "deepnsm", - "deltalake", "futures", "lance", "lance-arrow", @@ -5613,7 +4640,6 @@ dependencies = [ "snafu 0.8.9", "tempfile", "tokio", - "url", ] [[package]] @@ -5650,7 +4676,7 @@ dependencies = [ "axum 0.8.9", "chrono", "clap", - "datafusion 54.1.0", + "datafusion", "flate2", "futures", "lance", @@ -5674,7 +4700,7 @@ version = "0.5.4" dependencies = [ "arrow-schema", "async-trait", - "datafusion 54.1.0", + "datafusion", "lance-namespace", "reqwest 0.12.28", "serde", @@ -5836,10 +4862,10 @@ dependencies = [ "bytes", "chrono", "crossbeam-queue", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", "dirs", "fst", "futures", @@ -5899,9 +4925,9 @@ dependencies = [ "arrow-select", "async-trait", "bytes", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-expr", "futures", "lance-core", "lance-io", @@ -5999,8 +5025,8 @@ dependencies = [ "arrow-schema", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-common", + "datafusion-physical-plan", "futures", "lance", "lance-core", @@ -6124,15 +5150,15 @@ dependencies = [ "async-trait", "bytes", "chrono", - "datafusion 54.1.0", - "datafusion-catalog 54.1.0", - "datafusion-common 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-sql 54.1.0", + "datafusion", + "datafusion-catalog", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-sql", "futures", "half", "lance", @@ -6352,12 +5378,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "link-section" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b685d66585d646efe09fec763d796c291049c8b6bf84e04954bffc8748341f0d" - [[package]] name = "link-section" version = "0.19.2" @@ -7503,12 +6523,6 @@ version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" -[[package]] -name = "percent-encoding-rfc3986" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3637c05577168127568a64e9dc5a6887da720efef07b3d9472d45f63ab191166" - [[package]] name = "permutation" version = "0.4.1" @@ -8501,10 +7515,8 @@ checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" dependencies = [ "base64", "bytes", - "encoding_rs", "futures-core", "futures-util", - "h2", "http 1.4.2", "http-body 1.0.1", "http-body-util", @@ -8513,7 +7525,6 @@ dependencies = [ "hyper-util", "js-sys", "log", - "mime", "percent-encoding", "pin-project-lite", "quinn", @@ -9302,17 +8313,6 @@ dependencies = [ "der", ] -[[package]] -name = "sqlparser" -version = "0.61.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dbf5ea8d4d7c808e1af1cbabebca9a2abe603bcefc22294c5b95018d53200cb7" -dependencies = [ - "log", - "recursive", - "sqlparser_derive", -] - [[package]] name = "sqlparser" version = "0.62.0" @@ -9406,15 +8406,6 @@ dependencies = [ "strum_macros 0.26.4", ] -[[package]] -name = "strum" -version = "0.27.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" -dependencies = [ - "strum_macros 0.27.2", -] - [[package]] name = "strum" version = "0.28.0" @@ -9437,18 +8428,6 @@ dependencies = [ "syn", ] -[[package]] -name = "strum_macros" -version = "0.27.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "strum_macros" version = "0.28.0" @@ -10242,26 +9221,6 @@ dependencies = [ "rand 0.9.4", ] -[[package]] -name = "typed-builder" -version = "0.23.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31aa81521b70f94402501d848ccc0ecaa8f93c8eb6999eb9747e72287757ffda" -dependencies = [ - "typed-builder-macro", -] - -[[package]] -name = "typed-builder-macro" -version = "0.23.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "076a02dc54dd46795c2e9c8282ed40bcfb1e22747e955de9389a1de28190fb26" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "typenum" version = "1.20.1" @@ -10335,7 +9294,6 @@ dependencies = [ "idna", "percent-encoding", "serde", - "serde_derive", ] [[package]] @@ -10376,42 +9334,11 @@ checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" dependencies = [ "getrandom 0.4.3", "js-sys", - "rand 0.10.1", "serde_core", "sha1_smol", "wasm-bindgen", ] -[[package]] -name = "validator" -version = "0.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0b4a29d8709210980a09379f27ee31549b73292c87ab9899beee1c0d3be6303" -dependencies = [ - "idna", - "once_cell", - "regex", - "serde", - "serde_derive", - "serde_json", - "url", - "validator_derive", -] - -[[package]] -name = "validator_derive" -version = "0.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bac855a2ce6f843beb229757e6e570a42e837bcb15e5f449dd48d5747d41bf77" -dependencies = [ - "darling 0.20.11", - "once_cell", - "proc-macro-error2", - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "valuable" version = "0.1.1" @@ -11197,12 +10124,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "z85" -version = "3.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6e61e59a957b7ccee15d2049f86e8bfd6f66968fcd88f018950662d9b86e675" - [[package]] name = "zerocopy" version = "0.8.52" diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index 8267d026..b9274732 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -50,8 +50,6 @@ serde_json = "1" serde_yaml = "0.9" bytes = "1" snafu = "0.8" -deltalake = { version = "0.32", features = ["datafusion", "s3", "azure", "gcs"], optional = true } -url = { version = "2", optional = true } # AdaWorldAPI/ndarray fork with 55 HPC modules (Fingerprint, CAM-PQ, CLAM, BLAS, ZeckF64) # Source: https://github.com/AdaWorldAPI/ndarray @@ -75,13 +73,13 @@ bgz-tensor = { path = "../bgz-tensor", optional = true } [features] default = ["unity-catalog", "ndarray-hpc", "planner", "bgz17-codec", "tensor-codec"] unity-catalog = ["lance-graph-catalog/unity-catalog"] -# TODO(lance-bump-delta): deltalake 0.32 removed `DeltaTableProvider::try_new`; -# new API is `TableProviderBuilder::new().with_snapshot(...).with_log_store(...).build().await`. -# Refactor `table_readers.rs::DeltaTableReader` to the new builder API in a -# follow-up PR. Dropped from `default` features for now so CI stays green on -# the lance/lancedb bump; still opt-in via `--features delta` if a caller -# accepts the broken state. -delta = ["dep:deltalake", "dep:url"] +# The `delta` feature (deltalake + DeltaTableReader) was REMOVED 2026-08-18 +# (operator pin ruling: lance 9 / lancedb 0.33 / datafusion 54.1 — NO DF 53 — +# / arrow 58). deltalake tops out at 0.32.4 pinning datafusion ^53.1.0, so the +# feature was the sole source of the DF-53 duplicate major in Cargo.lock; it +# was also already non-default AND documented broken (0.32 removed +# `DeltaTableProvider::try_new`). A future Delta reader returns only when a +# DF-54-compatible deltalake exists, as its own deliberate PR. # lancedb: high-level table/connection API (Python/JS SDK equivalent in Rust). # Enables LanceDB-compatible connection strings, table management, and the # vector search API surface that third-party tooling (DuckDB ext, Polars) expects. diff --git a/crates/lance-graph/src/lib.rs b/crates/lance-graph/src/lib.rs index 1c8633b9..7dfd809b 100644 --- a/crates/lance-graph/src/lib.rs +++ b/crates/lance-graph/src/lib.rs @@ -80,6 +80,4 @@ pub use lance_graph_catalog::{UnityCatalogConfig, UnityCatalogProvider}; pub use lance_vector_search::VectorSearch; pub use query::{CypherQuery, ExecutionStrategy, SqlDialect}; pub use sql_query::SqlQuery; -#[cfg(feature = "delta")] -pub use table_readers::DeltaTableReader; pub use table_readers::{default_table_readers, ParquetTableReader}; diff --git a/crates/lance-graph/src/table_readers.rs b/crates/lance-graph/src/table_readers.rs index 4ebf17b3..617c599c 100644 --- a/crates/lance-graph/src/table_readers.rs +++ b/crates/lance-graph/src/table_readers.rs @@ -4,7 +4,9 @@ //! Built-in [`TableReader`] implementations for common data formats. //! //! - [`ParquetTableReader`] — reads Parquet tables using DataFusion's built-in support. -//! - [`DeltaTableReader`] — reads Delta Lake tables (behind `delta` feature flag). +//! +//! (A Delta Lake reader existed behind the removed `delta` feature — see +//! Cargo.toml's removal note, 2026-08-18.) use std::collections::HashMap; use std::sync::Arc; @@ -57,90 +59,11 @@ impl TableReader for ParquetTableReader { } } -/// Reads Delta Lake tables using the `deltalake` crate. -/// -/// Opens the Delta table at the storage location and registers it as a -/// DataFusion `TableProvider`, enabling full SQL query support including -/// time travel, schema evolution, and partition pruning. -/// -/// Supports cloud storage (S3, Azure, GCS) via `storage_options`. -#[cfg(feature = "delta")] -pub struct DeltaTableReader; - -#[cfg(feature = "delta")] -#[async_trait] -impl TableReader for DeltaTableReader { - fn name(&self) -> &str { - "delta" - } - - fn supported_formats(&self) -> &[DataSourceFormat] { - &[DataSourceFormat::Delta] - } - - async fn register_table( - &self, - ctx: &SessionContext, - table_name: &str, - table_info: &TableInfo, - _schema: arrow_schema::SchemaRef, - storage_options: &HashMap, - ) -> CatalogResult<()> { - let location = table_info.storage_location.as_deref().ok_or_else(|| { - CatalogError::Other(format!("Table '{}' has no storage_location", table_name)) - })?; - - let table_url = url::Url::parse(location).map_err(|e| { - CatalogError::Other(format!( - "Invalid storage location URL '{}': {}", - location, e - )) - })?; - - let delta_table = if storage_options.is_empty() { - deltalake::open_table(table_url).await - } else { - deltalake::open_table_with_storage_options(table_url, storage_options.clone()).await - } - .map_err(|e| { - CatalogError::Other(format!( - "Failed to open Delta table '{}' at '{}': {}", - table_name, location, e - )) - })?; - - let snapshot = delta_table - .snapshot() - .map_err(|e| CatalogError::Other(format!("Failed to get Delta snapshot: {e}")))? - .snapshot() - .clone(); - let log_store = delta_table.log_store(); - let scan_config = Default::default(); - let provider = deltalake::delta_datafusion::DeltaTableProvider::try_new( - snapshot, - log_store, - scan_config, - ) - .map_err(|e| CatalogError::Other(format!("Failed to create DeltaTableProvider: {e}")))?; - ctx.register_table(table_name, Arc::new(provider)) - .map_err(|e| { - CatalogError::Other(format!( - "Failed to register Delta table '{}': {}", - table_name, e - )) - })?; - - Ok(()) - } -} - /// Returns the default set of table readers. /// -/// Includes Parquet support, and Delta Lake support when the `delta` feature is enabled. +/// Includes Parquet support. (Delta Lake support was removed with the `delta` +/// feature, 2026-08-18 — see Cargo.toml's note; a `DataSourceFormat::Delta` +/// table simply has no registered reader, exactly as when the feature was off.) pub fn default_table_readers() -> Vec> { - #[allow(unused_mut)] - let mut readers: Vec> = vec![Arc::new(ParquetTableReader)]; - #[cfg(feature = "delta")] - readers.push(Arc::new(DeltaTableReader)); - readers + vec![Arc::new(ParquetTableReader)] } diff --git a/docs/lotus/LOTUS-FRONTIER-AUDIT.md b/docs/lotus/LOTUS-FRONTIER-AUDIT.md index ae714e85..d23fccb0 100644 --- a/docs/lotus/LOTUS-FRONTIER-AUDIT.md +++ b/docs/lotus/LOTUS-FRONTIER-AUDIT.md @@ -152,6 +152,18 @@ OFFERS. Phase 6 (`PREPARED-ARTIFACT-PUBLICATION.md`) is **gated** on a `cargo fetch` (network + disk budget permitting) or an operator-sanctioned alternative source consult. Do not write Phase 6 from memory of lance APIs. +> **⊘ BLOCKER LIFTED 2026-08-18 (same day, operator pin ruling).** The +> operator's dependency ruling ("pinned via [patch] of the upstream +> repository git") sanctioned the upstream source consult: the EXACT v9.0.0 +> tag is on disk (`/tmp/sources/lance-9`, matching the Cargo.lock checksum) +> plus current upstream (`/tmp/sources/lance-main`). Capability findings are +> deliberately NOT recorded here — they enter through the RP-SEAL research +> program's Domain-A independent passes (`.claude/plans/ +> erasure-seals-compaction-research-v1.md` §11 independence rule) and are +> cross-checked at consolidation. The paragraph above stands as the honest +> record of what THIS session's Phase 0 could and could not verify at +> composition time. + ### §2.3 The existing "merkle" is not a tree (VERIFIED) `MerkleRoot::from_fingerprint` (`crates/lance-graph/src/graph/spo/merkle.rs:18-38`) From 3fa7f7969f31a132fdc6af28294d96a23fe6ea7c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 22:04:01 +0000 Subject: [PATCH 5/8] =?UTF-8?q?scope=20pivot=20(operator):=20rustynum=20st?= =?UTF-8?q?ruck=20=E2=80=94=20everything=20ndarray;=20symbiont=20deprecate?= =?UTF-8?q?d=20(#879/#911/#912/#913)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recorded in the RP-SEAL plan header + the pin-ruling EPIPHANIES entry; the workflow script was corrected in place (source map + the two cell briefs naming rustynum); the in-flight independent pass could not be force-stopped in this harness build, so the ruling binds consolidation as a hard filter. Deltalake removal ratified same exchange. Co-Authored-By: Claude --- .claude/board/EPIPHANIES.md | 16 +++++++++++++--- .../erasure-seals-compaction-research-v1.md | 18 +++++++++++++++++- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 6a5054b5..5dd51ebe 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -25,9 +25,19 @@ entries.** `cargo check -p lance-graph` green (needed `protobuf-compiler` installed in-sandbox — lance's prost build; the Dockerfiles already install it). Docker pins surveyed on request: root + avx512 Dockerfiles = Rust 1.97.1 + protobuf-compiler/cmake, no feature references to delta, versions -purely from Cargo.lock — removal is docker-safe. One stale flag (not -touched): `crates/symbiont/Dockerfile` still `rust:1.95-bookworm` -(pre-1.97.1-sweep residue; symbiont is excluded/standalone). +purely from Cargo.lock — removal is docker-safe. The symbiont +Dockerfile's stale `rust:1.95` pin is MOOT: **symbiont is DEPRECATED** +(operator, same day — superseded by the supervisor/persistence arc, +PRs #879/#911/#912/#913) and is not live surface. + +**Same-day scope pivot (operator, BINDING for RP-SEAL):** rustynum is NOT a +source of truth — "no rustynum, everything ndarray"; it is the historical +donor already ported into AdaWorldAPI/ndarray. The RP-SEAL workflow script +was corrected in place (source map + the two cell briefs naming rustynum); +the in-flight independent pass could not be force-stopped in this harness +build, so the ruling additionally binds consolidation as a hard filter +(re-anchor or surgically re-run any finding citing rustynum/symbiont). +Deltalake removal ratified in the same exchange ("we don't need deltalake"). **Second discharge — the lotus BLOCKER:** the [patch → upstream-git] mechanism sanctioned consulting the upstream lance repository as source; the diff --git a/.claude/plans/erasure-seals-compaction-research-v1.md b/.claude/plans/erasure-seals-compaction-research-v1.md index 24353dfb..b7142c2d 100644 --- a/.claude/plans/erasure-seals-compaction-research-v1.md +++ b/.claude/plans/erasure-seals-compaction-research-v1.md @@ -8,6 +8,21 @@ > the five BUILDER and five SCOUT cells = grindwork tier. Consolidation > (§12) and the final attack pass (§13) run on the main thread AFTER all 15 > reports exist. This plan file is the operator's program, recorded as the +> **⊘ SCOPE PIVOT (operator, 2026-08-18, mid-dispatch — BINDING):** +> **rustynum is struck from the source-of-truth list — everything ndarray.** +> rustynum is the historical donor already ported into AdaWorldAPI/ndarray; +> every primitive/capability/SIMD question routes to `/home/user/ndarray`. +> **crates/symbiont is DEPRECATED** (superseded by the supervisor/persistence +> arc, lance-graph PRs #879/#911/#912/#913) and is not live surface. +> Application: the workflow script was corrected in place (preamble source +> map + the two cell briefs that named rustynum); the in-flight first pass +> could not be force-stopped in this harness build, so the ruling ALSO binds +> consolidation as a hard filter — any first-pass finding citing rustynum or +> symbiont is re-anchored to ndarray / marked deprecated-source, or its cell +> is surgically re-run with the corrected brief. No second-pass or attack +> work may touch either. **Deltalake removal ratified** ("we don't need +> deltalake") — see E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1. +> > arc's source of truth. It SUPERSEDES the sequencing of the earlier lotus > charter's phases 2–8 (docs/lotus/ deliverables 3–9 now route through this > program); the landed Phase 0/1 deliverables (audit + F-ORD-REAL falsifier, @@ -42,7 +57,8 @@ suspected novel pattern requires a prior-art search. ## 1. Source of truth Repositories: AdaWorldAPI/lance-graph, AdaWorldAPI/OGAR, AdaWorldAPI/ndarray, -AdaWorldAPI/rustynum, lance upstream. Temporal reference: OGAR commit +lance upstream. (AdaWorldAPI/rustynum appeared in the original program text +and was STRUCK by the same-day scope pivot — see the header note.) Temporal reference: OGAR commit `386a6fd848334b1d880c8408b3810f045d135cfe`, `docs/TEMPORAL-TIME-TRAVEL.md` — read it literally. Temporal concepts: current tick / current Lance version; last awareness tick / reference horizon; write tick; hindsight knowledge; From 1479bc98dccbbf4a77d05de68b0f2af12527c91c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 22:06:08 +0000 Subject: [PATCH 6/8] correct the delta-removal notes: delta-rs MAIN is already DF 54 + arrow 58 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator-pointed, verified from delta-io/delta-rs main Cargo.toml (datafusion = 54.0.0, arrow = 58): only the crates.io releases top out at DF 53. Restoration is available now via a git-pin on upstream + a reader refactor to the current builder API — as its own deliberate PR if a consumer needs Delta. The removal itself stands on the need ruling ('we don't need deltalake'), not on availability; notes in Cargo.toml, CLAUDE.md, and the EPIPHANIES entry corrected accordingly. Co-Authored-By: Claude --- .claude/board/EPIPHANIES.md | 10 ++++++++++ CLAUDE.md | 6 ++++-- crates/lance-graph/Cargo.toml | 10 ++++++++-- 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 5dd51ebe..1856a2ab 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -39,6 +39,16 @@ build, so the ruling additionally binds consolidation as a hard filter (re-anchor or surgically re-run any finding citing rustynum/symbiont). Deltalake removal ratified in the same exchange ("we don't need deltalake"). +**Same-day factual correction (operator-pointed, verified at source):** the +"no DF-54 deltalake exists" premise holds only for crates.io RELEASES +(deltalake-core <= 0.32.4 / DF ^53). delta-io/delta-rs **main** already pins +`datafusion = 54.0.0` + `arrow = 58` (workspace Cargo.toml, fetched +2026-08-18) — so restoration is available NOW via the house mechanism (git +dep on the upstream repository, pinned rev) + a `DeltaTableReader` refactor +to the current builder API. The removal stands on the NEED ruling, not on +availability; the in-repo notes were corrected so no future session treats +restoration as blocked on a registry release. + **Second discharge — the lotus BLOCKER:** the [patch → upstream-git] mechanism sanctioned consulting the upstream lance repository as source; the exact v9.0.0 tag is cloned (`/tmp/sources/lance-9`, matching the Cargo.lock diff --git a/CLAUDE.md b/CLAUDE.md index 1cb6871e..862b9bee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1173,8 +1173,10 @@ datafusion = "54" # OUR direct pin, in every crate that DEPENDS on it # exactly ONE datafusion = 54.1.0. The ruling: lance 9 / # lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, # always, across AdaWorldAPI forks. A Delta reader - # returns only with a DF-54-compatible deltalake, as its - # own deliberate PR. + # can return via a git-pin on delta-rs upstream — its + # MAIN already pins DF 54 + arrow 58 (verified + # 2026-08-18; crates.io releases still DF 53) — as its + # own deliberate PR if a consumer needs Delta. # Probe: .claude/plans/lance9-datafusion54-upgrade-probe-v1.md lance = "=9.0.0" # exact-pinned: lancedb 0.33.0 requires lance =9.0.0 lance-linalg = "=9.0.0" diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index b9274732..fdfe4e8c 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -78,8 +78,14 @@ unity-catalog = ["lance-graph-catalog/unity-catalog"] # / arrow 58). deltalake tops out at 0.32.4 pinning datafusion ^53.1.0, so the # feature was the sole source of the DF-53 duplicate major in Cargo.lock; it # was also already non-default AND documented broken (0.32 removed -# `DeltaTableProvider::try_new`). A future Delta reader returns only when a -# DF-54-compatible deltalake exists, as its own deliberate PR. +# `DeltaTableProvider::try_new`). CORRECTION (same day, operator-pointed): +# delta-rs MAIN already pins datafusion 54 + arrow 58 (verified 2026-08-18 +# from delta-io/delta-rs main Cargo.toml) — only the crates.io releases top +# out at DF 53. Restoration is therefore available NOW via the house +# mechanism (git dep on the upstream repository, pinned to a rev) + a reader +# refactor to the current builder API — as its own deliberate PR if a +# consumer needs Delta. The removal itself stands ("we don't need +# deltalake"). # lancedb: high-level table/connection API (Python/JS SDK equivalent in Rust). # Enables LanceDB-compatible connection strings, table management, and the # vector search API surface that third-party tooling (DuckDB ext, Polars) expects. From 40e3255c5898cf4fe0c8f97f4b20e9861bf92ac4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 22:14:28 +0000 Subject: [PATCH 7/8] =?UTF-8?q?profile.dev=20debug=3D0=20=E2=80=94=20small?= =?UTF-8?q?er/faster=20builds=20(operator=20ruling)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full-debuginfo test binaries grew target/ to 17 GB and SIGBUS'd the linker on a full disk this session; debug=0 shrinks the lance/datafusion-stack test binaries ~an order of magnitude. Matches the gate already running with CARGO_PROFILE_DEV_DEBUG=0 (same resolved profile, cache reuse). line-tables-only noted as the fallback if line-numbered backtraces are ever needed. Co-Authored-By: Claude --- Cargo.toml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Cargo.toml b/Cargo.toml index eeb4a1ea..dbb7f1c3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -240,3 +240,12 @@ resolver = "2" # unfetchable, so the git form deadlocks offline sessions. Path form = zero # network, identical source. ndarray = { path = "../ndarray" } + +# Dev/test debuginfo OFF (operator, 2026-08-18: "Cargo Debug*=0 or so makes it +# much smaller and faster"). Measured incident the same day: full debuginfo +# test binaries grew target/ to 17 GB and killed the linker with SIGBUS on a +# full disk; debug = 0 shrinks the lance/datafusion-stack test binaries by +# roughly an order of magnitude and links faster. Set "line-tables-only" here +# instead if line-numbered backtraces are ever worth the size again. +[profile.dev] +debug = 0 From 94556a162429d3898618fe1389b42d3db1af72cc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 22:24:43 +0000 Subject: [PATCH 8/8] drift cleanup: rustynum/symbiont no-go applied to live surfaces - lance-graph-cognitive: rustynum_accel was a name-only shim (zero rustynum dependency); renamed simd_accel (module + file + 11 call sites), rerouted through the sanctioned ndarray::simd re-export instead of ndarray::hpc::bitwise. Default compile green; the wip feature's pre-existing not-yet-compiling state has zero errors naming the rename. - docs/lotus audit: symbiont::domino::morton4 candidate row struck (census history only). - CLAUDE.md: symbiont removed from the binding-consumer lists via dated annotations. - EPIPHANIES: cleanup recorded in the pin-ruling entry. Audit also confirmed NO redo needed: A1's research report is clean (0 citations), the Java/Panama arc never touched either (ndarray::simd enforced), board history stays append-only. Co-Authored-By: Claude --- .claude/board/EPIPHANIES.md | 16 +++++++ CLAUDE.md | 4 +- crates/lance-graph-cognitive/Cargo.lock | 42 ------------------- .../src/core_full/fingerprint.rs | 2 +- .../src/core_full/mod.rs | 4 +- .../src/core_full/scent.rs | 4 +- .../{rustynum_accel.rs => simd_accel.rs} | 2 +- crates/lance-graph-cognitive/src/lib.rs | 12 +++--- .../src/search/hdr_cascade.rs | 8 ++-- .../lance-graph-cognitive/src/spo/gestalt.rs | 6 +-- docs/lotus/LOTUS-FRONTIER-AUDIT.md | 2 +- 11 files changed, 38 insertions(+), 64 deletions(-) rename crates/lance-graph-cognitive/src/core_full/{rustynum_accel.rs => simd_accel.rs} (99%) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 1856a2ab..5d5e6ecb 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -39,6 +39,22 @@ build, so the ruling additionally binds consolidation as a hard filter (re-anchor or surgically re-run any finding citing rustynum/symbiont). Deltalake removal ratified in the same exchange ("we don't need deltalake"). +**Drift cleanup executed (same day, on the operator's "absolute no-go"):** +audit result — A1's research report: 0 citations (clean); the Java/Panama +arc: zero involvement (ndarray::simd enforced throughout); merged docs +carried exactly one live symbiont citation (the lotus audit §3 candidate +row, now ⊘-struck) + CLAUDE.md's two symbiont-as-binding-consumer lines +(now ⊘-annotated). Live-code drift found and FIXED: +`lance-graph-cognitive`'s `rustynum_accel` shim was name-only (zero +rustynum dependency — it delegated to `ndarray::hpc::bitwise`); renamed +`simd_accel` (module + file + 11 call sites) and rerouted through the +sanctioned `ndarray::simd::hamming_distance_raw` re-export. Default +compile green; the `wip` feature's pre-existing not-yet-compiling state +carries zero errors naming the rename. Historical provenance comments +("replacing rustynum as of 2026-03-22", ndarray's port-history docs) kept +— the no-go bans USE, not history. Board history entries citing symbiont +stay append-only, superseded by this entry. + **Same-day factual correction (operator-pointed, verified at source):** the "no DF-54 deltalake exists" premise holds only for crates.io RELEASES (deltalake-core <= 0.32.4 / DF ^53). delta-io/delta-rs **main** already pins diff --git a/CLAUDE.md b/CLAUDE.md index 862b9bee..a4c4176a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -968,7 +968,7 @@ Architecture: Dependency chain: ladybug-rs ──► lance-graph-contract (traits) - in-tree ──► lance-graph-contract (planner / callcenter / smb-bridge / symbiont) + in-tree ──► lance-graph-contract (planner / callcenter / smb-bridge; symbiont ⊘ DEPRECATED 2026-08-18, operator no-go) lance-graph──► ndarray (default dep, with fallback) ``` @@ -1230,7 +1230,7 @@ YAML card → 23D vector → ThinkingStyle(36) → FieldModulation(7D) → ScanP ``` WHO DEPENDS ON lance-graph-contract: - in-tree — planner, callcenter, smb-bridge, symbiont (the BINDING consumers) + in-tree — planner, callcenter, smb-bridge (the BINDING consumers; symbiont ⊘ DEPRECATED 2026-08-18, operator no-go — dormant excluded crate, never live surface) ladybug-rs — Planner + CamPq + OrchestrationBridge (crewai-rust / n8n-rs — EVICTED 2026-06-21, operator-directed; superseded by ladybug-rs + the in-tree thinking-engine. NO LONGER binding consumers — the diff --git a/crates/lance-graph-cognitive/Cargo.lock b/crates/lance-graph-cognitive/Cargo.lock index c3e8d22a..cedb0b75 100644 --- a/crates/lance-graph-cognitive/Cargo.lock +++ b/crates/lance-graph-cognitive/Cargo.lock @@ -25,18 +25,6 @@ dependencies = [ "libc", ] -[[package]] -name = "arrayref" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" - -[[package]] -name = "arrayvec" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" - [[package]] name = "arrow-array" version = "58.4.0" @@ -101,20 +89,6 @@ dependencies = [ "serde", ] -[[package]] -name = "blake3" -version = "1.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4d2d5991425dfd0785aed03aedcf0b321d61975c9b5b3689c774a2610ae0b51e" -dependencies = [ - "arrayref", - "arrayvec", - "cc", - "cfg-if", - "constant_time_eq", - "cpufeatures", -] - [[package]] name = "bumpalo" version = "3.20.2" @@ -174,27 +148,12 @@ dependencies = [ "tiny-keccak", ] -[[package]] -name = "constant_time_eq" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" - [[package]] name = "core-foundation-sys" version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" -[[package]] -name = "cpufeatures" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" -dependencies = [ - "libc", -] - [[package]] name = "crunchy" version = "0.2.4" @@ -469,7 +428,6 @@ checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" name = "ndarray" version = "0.17.2" dependencies = [ - "blake3", "matrixmultiply", "num-complex", "num-integer", diff --git a/crates/lance-graph-cognitive/src/core_full/fingerprint.rs b/crates/lance-graph-cognitive/src/core_full/fingerprint.rs index 0ff4c989..bfc2e902 100644 --- a/crates/lance-graph-cognitive/src/core_full/fingerprint.rs +++ b/crates/lance-graph-cognitive/src/core_full/fingerprint.rs @@ -136,7 +136,7 @@ impl Fingerprint { /// Hamming distance to another fingerprint #[inline] pub fn hamming(&self, other: &Fingerprint) -> u32 { - super::rustynum_accel::hamming_distance(self, other) + super::simd_accel::hamming_distance(self, other) } /// Similarity (0.0 - 1.0) diff --git a/crates/lance-graph-cognitive/src/core_full/mod.rs b/crates/lance-graph-cognitive/src/core_full/mod.rs index 25abf7e5..04ec7689 100644 --- a/crates/lance-graph-cognitive/src/core_full/mod.rs +++ b/crates/lance-graph-cognitive/src/core_full/mod.rs @@ -5,12 +5,12 @@ mod fingerprint; mod scent; pub mod vsa; -pub mod rustynum_accel; +pub mod simd_accel; pub use buffer::BufferPool; pub use fingerprint::Fingerprint; pub use scent::*; -pub use rustynum_accel::{HammingEngine, batch_hamming, hamming_distance, simd_level}; +pub use simd_accel::{HammingEngine, batch_hamming, hamming_distance, simd_level}; pub use vsa::VsaOps; /// Dense embedding vector diff --git a/crates/lance-graph-cognitive/src/core_full/scent.rs b/crates/lance-graph-cognitive/src/core_full/scent.rs index c25c356b..088775dc 100644 --- a/crates/lance-graph-cognitive/src/core_full/scent.rs +++ b/crates/lance-graph-cognitive/src/core_full/scent.rs @@ -521,8 +521,8 @@ fn timestamp() -> u64 { .unwrap_or(0) } -// NOTE: SIMD-optimized scent scan is handled via rustynum runtime dispatch. -// No compile-time SIMD gates needed — rustynum detects AVX-512/AVX2 at runtime. +// NOTE: SIMD-optimized scent scan is handled via ndarray::simd runtime dispatch. +// No compile-time SIMD gates needed — the ndarray polyfill detects AVX-512/AVX2 at runtime. #[cfg(test)] mod tests { diff --git a/crates/lance-graph-cognitive/src/core_full/rustynum_accel.rs b/crates/lance-graph-cognitive/src/core_full/simd_accel.rs similarity index 99% rename from crates/lance-graph-cognitive/src/core_full/rustynum_accel.rs rename to crates/lance-graph-cognitive/src/core_full/simd_accel.rs index a9d07e3d..b3d29cb3 100644 --- a/crates/lance-graph-cognitive/src/core_full/rustynum_accel.rs +++ b/crates/lance-graph-cognitive/src/core_full/simd_accel.rs @@ -13,7 +13,7 @@ //! ## Usage //! //! ```rust,ignore -//! use ladybug::core::rustynum_accel::*; +//! use ladybug::core::simd_accel::*; //! //! let fp = Fingerprint::from_content("hello"); //! let pc = fingerprint_popcount(&fp); // VPOPCNTDQ when available diff --git a/crates/lance-graph-cognitive/src/lib.rs b/crates/lance-graph-cognitive/src/lib.rs index a267f034..71c25e26 100644 --- a/crates/lance-graph-cognitive/src/lib.rs +++ b/crates/lance-graph-cognitive/src/lib.rs @@ -86,29 +86,29 @@ pub mod storage { } /// Compatibility module mirroring ladybug-rs `crate::core::*`. -/// Modules reference `crate::core::Fingerprint`, `crate::core::rustynum_accel::*`, etc. +/// Modules reference `crate::core::Fingerprint`, `crate::core::simd_accel::*`, etc. pub mod core { pub use super::Fingerprint; pub use super::Embedding; pub const DIM: usize = super::FINGERPRINT_BITS; pub const DIM_U64: usize = super::FINGERPRINT_U64; - /// Compatibility shim for rustynum SIMD acceleration. + /// SIMD acceleration shim (renamed from the retired rustynum_accel name, 2026-08-18 operator no-go). /// Maps to ndarray::hpc::bitwise. - pub mod rustynum_accel { + pub mod simd_accel { pub fn hamming_distance(a: &[u8], b: &[u8]) -> u64 { - ndarray::hpc::bitwise::hamming_distance_raw(a, b) + ndarray::simd::hamming_distance_raw(a, b) } pub fn slice_hamming(a: &[u64], b: &[u64]) -> u64 { let a_bytes: Vec = a.iter().flat_map(|w| w.to_le_bytes()).collect(); let b_bytes: Vec = b.iter().flat_map(|w| w.to_le_bytes()).collect(); - ndarray::hpc::bitwise::hamming_distance_raw(&a_bytes, &b_bytes) + ndarray::simd::hamming_distance_raw(&a_bytes, &b_bytes) } pub fn batch_hamming(query: &[u8], database: &[u8], vec_len: usize) -> Vec { let n = database.len() / vec_len; (0..n).map(|i| { let start = i * vec_len; - ndarray::hpc::bitwise::hamming_distance_raw(query, &database[start..start + vec_len]) + ndarray::simd::hamming_distance_raw(query, &database[start..start + vec_len]) }).collect() } pub fn simd_level() -> &'static str { "ndarray" } diff --git a/crates/lance-graph-cognitive/src/search/hdr_cascade.rs b/crates/lance-graph-cognitive/src/search/hdr_cascade.rs index 1c2c7c9f..c10fe38c 100644 --- a/crates/lance-graph-cognitive/src/search/hdr_cascade.rs +++ b/crates/lance-graph-cognitive/src/search/hdr_cascade.rs @@ -78,10 +78,10 @@ const DEFAULT_INHIBIT: u32 = 5000; /// Compute exact Hamming distance. /// -/// Uses runtime-dispatched AVX-512 VPOPCNTDQ via rustynum. +/// Uses runtime-dispatched AVX-512 VPOPCNTDQ via ndarray::simd. #[inline] pub fn hamming_distance(a: &[u64; WORDS], b: &[u64; WORDS]) -> u32 { - crate::core::rustynum_accel::slice_hamming(a, b) as u32 + crate::core::simd_accel::slice_hamming(a, b) as u32 } /// Compute 1-bit sketch: which chunks differ at all? @@ -147,8 +147,8 @@ pub fn sketch_8bit_sum(sketch: &[u8; WORDS]) -> u32 { sketch.iter().map(|&b| b as u32).sum() } -// NOTE: All SIMD dispatch is handled by rustynum-core at runtime. -// The hamming_distance() function above delegates to rustynum_accel::slice_hamming() +// NOTE: All SIMD dispatch is handled by ndarray::simd at runtime. +// The hamming_distance() function above delegates to simd_accel::slice_hamming() // which uses AVX-512 VPOPCNTDQ -> AVX2 Harley-Seal -> scalar POPCNT. // No compile-time SIMD gates needed. diff --git a/crates/lance-graph-cognitive/src/spo/gestalt.rs b/crates/lance-graph-cognitive/src/spo/gestalt.rs index ef396bb6..1177c79b 100644 --- a/crates/lance-graph-cognitive/src/spo/gestalt.rs +++ b/crates/lance-graph-cognitive/src/spo/gestalt.rs @@ -319,9 +319,9 @@ pub fn detect_bundling( gate: &ndarray::hpc::kernels::SigmaGate, ) -> Option<(BundlingType, u32, u32, u32)> { // Per-plane Hamming distance - let s_dist = crate::core::rustynum_accel::slice_hamming(center_a_s, center_b_s) as u32; - let p_dist = crate::core::rustynum_accel::slice_hamming(center_a_p, center_b_p) as u32; - let o_dist = crate::core::rustynum_accel::slice_hamming(center_a_o, center_b_o) as u32; + let s_dist = crate::core::simd_accel::slice_hamming(center_a_s, center_b_s) as u32; + let p_dist = crate::core::simd_accel::slice_hamming(center_a_p, center_b_p) as u32; + let o_dist = crate::core::simd_accel::slice_hamming(center_a_o, center_b_o) as u32; // "Close" = Evidence level or better (2σ below noise) let close = gate.evidence; diff --git a/docs/lotus/LOTUS-FRONTIER-AUDIT.md b/docs/lotus/LOTUS-FRONTIER-AUDIT.md index d23fccb0..20b64aad 100644 --- a/docs/lotus/LOTUS-FRONTIER-AUDIT.md +++ b/docs/lotus/LOTUS-FRONTIER-AUDIT.md @@ -183,7 +183,7 @@ with it. | `hhtl::NiblePath`, `FAN_OUT=16` | lance-graph core | Production 16-ary router; the doctrine already says *"HHTL is the deterministic PLACE (16-ary); helix is the RESIDUE"* | | `CurveRuler` stride-4-over-17 | helix | **The shipped comma** — triple-falsified positive (DFT anti-aliasing ev7b; `comma_quorum` N_eff 11.00/12 vs 1.00 without; `comma_awareness`). D-QUANTGATE names it the mandatory quantized-layer phase generator | | `basin_placement_learning` 75.8% | probe | **Overclaim flag (INFERENCE):** the measured 75.8% is a BINARY split, not 4-ary — `E-BASIN-IS-A-NODE` cites it wider than the measurement supports; re-grade before reuse | -| `symbiont::domino::morton4` | symbiont | Real 4×4 bijective Morton, but SIMD-lane-scoped; not a placement authority | +| `symbiont::domino::morton4` | symbiont | ⊘ STRUCK 2026-08-18 (operator no-go: symbiont DEPRECATED per the #879/#911/#912/#913 arc) — was: real 4×4 bijective Morton, SIMD-lane-scoped, not a placement authority. Not a candidate; kept as census history only | | thinking-engine `domino.rs` | thinking-engine | FALSE LEAD — name collision, unrelated mechanism | | Feistel / XorShift mixers | — | ABSENT from the workspace; a comma design needing one imports new code |