diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 8f0cfcee..1856a2ab 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,66 @@ +## 2026-08-18 — E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1 + +**Status:** RULING `[operator]` + same-day discharge, measured. + +**The ruling.** lance 9 / lancedb 0.33 / datafusion 54.1 (**no DF 53**) / +arrow 58 are ALWAYS pinned across AdaWorldAPI forks, usually consumed via +`[patch]` of the upstream repository git. DF 53 existed only for older syntax +in an out-of-scope research crate; all of lance-graph is DF-54-certified, +and single-major DF 54.1 is the discipline ("without DF 53/54 parallelism +it's much better, easier, faster to handle"). + +**Measured before acting:** the lock carried datafusion 53.1.0 AND 54.1.0; +the ONLY DF-53 source was `deltalake 0.32.4` (`^53.1.0`) behind the `delta` +feature — which was already NON-default and documented BROKEN (0.32 removed +`DeltaTableProvider::try_new`; the reader was never refactored). Checked the +registry: deltalake-core tops out at 0.32.4 / DF ^53.1.0 — no DF-54 deltalake +exists, so a bump could not discharge the ruling; removal could. + +**Discharged:** `delta` feature + `deltalake`/`url` optional deps + +`DeltaTableReader` removed (Cargo.toml carries the dated removal note; a +future Delta reader returns only with a DF-54-compatible deltalake, as its +own deliberate PR). `DataSourceFormat::Delta` (a catalog metadata tag) stays. +**Post-removal lock: exactly ONE datafusion = 54.1.0; zero deltalake +entries.** `cargo check -p lance-graph` green (needed `protobuf-compiler` +installed in-sandbox — lance's prost build; the Dockerfiles already install +it). Docker pins surveyed on request: root + avx512 Dockerfiles = Rust +1.97.1 + protobuf-compiler/cmake, no feature references to delta, versions +purely from Cargo.lock — removal is docker-safe. The symbiont +Dockerfile's stale `rust:1.95` pin is MOOT: **symbiont is DEPRECATED** +(operator, same day — superseded by the supervisor/persistence arc, +PRs #879/#911/#912/#913) and is not live surface. + +**Same-day scope pivot (operator, BINDING for RP-SEAL):** rustynum is NOT a +source of truth — "no rustynum, everything ndarray"; it is the historical +donor already ported into AdaWorldAPI/ndarray. The RP-SEAL workflow script +was corrected in place (source map + the two cell briefs naming rustynum); +the in-flight independent pass could not be force-stopped in this harness +build, so the ruling additionally binds consolidation as a hard filter +(re-anchor or surgically re-run any finding citing rustynum/symbiont). +Deltalake removal ratified in the same exchange ("we don't need deltalake"). + +**Same-day factual correction (operator-pointed, verified at source):** the +"no DF-54 deltalake exists" premise holds only for crates.io RELEASES +(deltalake-core <= 0.32.4 / DF ^53). delta-io/delta-rs **main** already pins +`datafusion = 54.0.0` + `arrow = 58` (workspace Cargo.toml, fetched +2026-08-18) — so restoration is available NOW via the house mechanism (git +dep on the upstream repository, pinned rev) + a `DeltaTableReader` refactor +to the current builder API. The removal stands on the NEED ruling, not on +availability; the in-repo notes were corrected so no future session treats +restoration as blocked on a registry release. + +**Second discharge — the lotus BLOCKER:** the [patch → upstream-git] +mechanism sanctioned consulting the upstream lance repository as source; the +exact v9.0.0 tag is cloned (`/tmp/sources/lance-9`, matching the Cargo.lock +checksum) plus current upstream (`/tmp/sources/lance-main`) for the RP-SEAL +two-column discipline. D-LOTUS-6's "lance source absent" BLOCKER is LIFTED; +capability findings deliberately enter through the RP-SEAL Domain-A +independent passes, not this entry (independence rule). + +**Supersedes in place:** CLAUDE.md's "BOTH MAJORS ARE REQUIRED — do NOT fix +Cargo.lock to one" datafusion note (written when `delta` was default and +load-bearing) — corrected in the same commit. + ## 2026-08-18 — E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1 **Status:** FINDING (VERIFIED at file:line) + pre-registered falsifier landed diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 60705ceb..e0582a6b 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,11 @@ +## 2026-08-18 — erasure-seals-compaction-research-v1 (RP-SEAL) — ACTIVE + +Operator-delivered 15-researcher program (5 domains x builder/adversary/ +scout) on bidirectional erasure-coded seals + locality-aware compaction in +versioned SoA grid stores. `.claude/plans/erasure-seals-compaction-research-v1.md`. +Independent pass dispatched as workflow `wf_ca974718-1b4`; consolidation + +tiered attack follow. Subsumes the lotus charter's phases 2-8. + ## 2026-08-13 — weather-soa-bake-v1 (PLAN; the missing Zarr→NodeRow bake) Plan: `.claude/plans/weather-soa-bake-v1.md`. **Does not supersede diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 8e27987a..5f46b7c8 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,21 @@ +## 2026-08-18 — lance-graph #961 (MERGED 0cc171f) — LOTUS research arc opens: frontier audit + F-ORD-REAL falsifier + +### Current Contract Inventory — no new types (docs + tests only) + +- `docs/lotus/LOTUS-FRONTIER-AUDIT.md` — Phase 0 archaeology; the verified + F-ORD defect chain (`content_hash` folds arrival-minted `stream_position`, + `persist_sink.rs:414`, contradicting the struct's own doc); seal = 3× + O(bytes) passes with the batch resident up to 3× (descriptor doctrine + unrealized: `BatchWriter>`); lance crate source ABSENT from sandbox + (Phase 6 BLOCKED); §6: cycles become THIN not permeable (Regime A + pipelining; rung-qualified frontier; linear-text vs tile split per class). +- `docs/lotus/F-ORD-REAL-FALSIFIER.md` + `cycle_driver.rs` tests: GREEN + defect pin `f_ord_real_defect_pin_arrival_order_leaks_into_batch_hash` + (fails loudly when a fix lands → deliberate re-pin) + `#[ignore]`d RED + `f_ord_real_publication_identity_is_arrival_order_independent`. +- Gates: supervisor suite 28 passed / 1 ignored (`--features cycle-driver`); + RED verified red under `--ignored`; fmt clean; zero new clippy warnings. + ## 2026-08-17 — branch `claude/hydrate-staging-merge` — `lance-graph-hydrate`: the deferred staging/publish merge lands (closes `ISS-HYDRATE-DIR-AND-FILE-DUPLICATE-THEIR-STAGING-BODIES`) ### Current Contract Inventory — new private module (`crates/lance-graph-hydrate/src/publish.rs`) diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index ae6035f2..f43b5021 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -1,3 +1,30 @@ +## 2026-08-18 — lance-graph #961 (MERGED, squash 0cc171f) — LOTUS Phase 0/1: frontier audit + F-ORD-REAL pre-registered falsifier + +- **Added:** `docs/lotus/LOTUS-FRONTIER-AUDIT.md` (Phase 0 archaeology, 4 + lenses, graded VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER; §6 answers the + operator's permeability/amortization question: the cycle becomes THIN, not + permeable — trailing-publication pipelining Regime A; rung-qualified + frontier via the shipped `EpistemicMode::for_rung` ladder; the linear-text + vs tile workload split resolved per class); `docs/lotus/ + F-ORD-REAL-FALSIFIER.md`; two pre-registered tests in `cycle_driver.rs` + behind the existing `cycle-driver` feature (GREEN two-sided defect pin + + `#[ignore]`d RED falsifier, verified red under `--ignored`: hashes + 2604999916736672513 vs 4858955943411201665 for the identical semantic set). +- **Locked:** the falsifier perturbs the process that CREATES the key (cast() + call order), never post-mint slots (the operator's F-ORD sharpening); no + fix mechanism prescribed — four candidates listed, per-workload, Phase 2+ + council-gated. +- **Deferred / gated:** D-LOTUS-3..9 (design/comma/temporal-seam/benchmarks/ + verdict); D-LOTUS-6 BLOCKED on the absent lance crate source (sandbox has + no registry copy; needs `cargo fetch` or sanctioned consult); + `temporal.rs:396`/`:410` retired-type doc-drift correction owed to the next + phase PR. +- **Docs:** EPIPHANIES `E-FORD-REAL-PUBLICATION-IDENTITY-IS-ARRIVAL-DEPENDENT-1`; + STATUS_BOARD `lotus-seal-fractal-commit-frontier` D-LOTUS-1..9. +- **Confidence:** high on the defect chain (every link read at file:line this + session; the doc-contradiction is quoted); the §6 amortization mechanisms + are INFERENCE until the Phase 5 benchmarks run. + ## 2026-08-12 — lance-graph #946 (MERGED) — D-CZ-1 PASSES; D-CZ-0 had NO artifact behind it; + the arc's rated formula matrix - **The finding that forced the run.** `substrate-comfort-zones-v1` §1's diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 6368f397..edcc1ab2 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,16 @@ +## erasure-seals-compaction-research-v1 — RP-SEAL (operator charter, DISPATCHED 2026-08-18) + +Plan: `.claude/plans/erasure-seals-compaction-research-v1.md` (the operator's +15-researcher program; supersedes lotus deliverables 3-9's sequencing). +Independent pass = background workflow `wf_ca974718-1b4`: 5 domains +(A Lance storage / B spatial layout / C erasure coding / D temporal / +E SoA-cache) x 3 roles (builder/adversary/scout); adversaries on the +strongest tier, builders+scouts on the grindwork tier; first pass blind to +docs/lotus/** and all boards (independence rule). Consolidation (evidence +matrix + cross-domain graph) and the tiered attack pass run main-thread +AFTER all 15 reports land. Reports: session scratchpad `rp-seal-v1/`, +committed at consolidation as Appendix H. + ## lotus-seal-fractal-commit-frontier — research arc (PRE-REGISTERED 2026-08-18) Charter: operator, 2026-08-18 (research + architecture; "first audit the @@ -7,12 +20,12 @@ metaphor becomes a type name without council approval. | D-id | Deliverable | Status | |---|---|---| -| D-LOTUS-1 | LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology, 4 lenses, VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER graded; §6 answers the permeability/amortization question | **In PR** | -| D-LOTUS-2 | F-ORD-REAL-FALSIFIER.md + the pre-registered tests (GREEN defect pin + `#[ignore]`d RED falsifier, `cycle_driver.rs`) | **In PR** — RED lands BEFORE any fix, per charter | +| D-LOTUS-1 | LOTUS-FRONTIER-AUDIT.md — Phase 0 archaeology, 4 lenses, VERIFIED/INFERENCE/HYPOTHESIS/BLOCKER graded; §6 answers the permeability/amortization question | **Shipped** — PR #961, 0cc171f | +| D-LOTUS-2 | F-ORD-REAL-FALSIFIER.md + the pre-registered tests (GREEN defect pin + `#[ignore]`d RED falsifier, `cycle_driver.rs`) | **Shipped** — PR #961, 0cc171f; RED landed BEFORE any fix, per charter | | D-LOTUS-3 | LOTUS-SEAL-DESIGN.md (16⁴ tree, petal seal, H(parent)=H(children) — NEW design per audit §2.3) | Queued (post-verdict) | | D-LOTUS-4 | COMMA-MODULATION-EXPERIMENT.md (petal-index-tier comma; locality-vs-resonance trade; "if it doesn't survive measurement, delete it") | Queued | | D-LOTUS-5 | TEMPORAL-FRONTIER-SEAM.md (rung-qualified frontier visibility; hindsight-invariance falsifier; audit §6.4) | Queued | -| D-LOTUS-6 | PREPARED-ARTIFACT-PUBLICATION.md | **BLOCKED — lance crate source absent from sandbox** (audit §2.2); needs `cargo fetch` or sanctioned source consult | +| D-LOTUS-6 | PREPARED-ARTIFACT-PUBLICATION.md | **UNBLOCKED 2026-08-18** (operator pin ruling sanctioned the upstream-git source consult; exact v9.0.0 tag + current upstream on disk) — now routed through RP-SEAL Domain A rather than written solo, per the research charter | | D-LOTUS-7 | BENCHMARK-PLAN.md (seal/append split; cycle-major vs tile-major k∈{1,4,16}; F-AMORT version-range scaling; F-MEM RAM high-water freeze-vs-petal; viewport partial hydration) | Queued | | D-LOTUS-8 | PRE-REGISTERED-FALSIFIERS.md (the F-* catalogue) | Queued | | D-LOTUS-9 | Final verdict A-E | Queued — gates all implementation | diff --git a/.claude/plans/erasure-seals-compaction-research-v1.md b/.claude/plans/erasure-seals-compaction-research-v1.md new file mode 100644 index 00000000..b7142c2d --- /dev/null +++ b/.claude/plans/erasure-seals-compaction-research-v1.md @@ -0,0 +1,288 @@ +# erasure-seals-compaction-research-v1 — RP-SEAL (operator charter, 2026-08-18) + +> **Status: DISPATCHED 2026-08-18** — independent pass running as a 15-agent +> background workflow (run `wf_ca974718-1b4`; reports land under the session +> scratchpad `rp-seal-v1/`, committed at consolidation as Appendix H). +> Tier allocation per the operator's "Opus for filigrane, Sonnet for +> grindwork": the five ADVERSARY cells (A2/B2/C2/D2/E2) = strongest tier; +> the five BUILDER and five SCOUT cells = grindwork tier. Consolidation +> (§12) and the final attack pass (§13) run on the main thread AFTER all 15 +> reports exist. This plan file is the operator's program, recorded as the +> **⊘ SCOPE PIVOT (operator, 2026-08-18, mid-dispatch — BINDING):** +> **rustynum is struck from the source-of-truth list — everything ndarray.** +> rustynum is the historical donor already ported into AdaWorldAPI/ndarray; +> every primitive/capability/SIMD question routes to `/home/user/ndarray`. +> **crates/symbiont is DEPRECATED** (superseded by the supervisor/persistence +> arc, lance-graph PRs #879/#911/#912/#913) and is not live surface. +> Application: the workflow script was corrected in place (preamble source +> map + the two cell briefs that named rustynum); the in-flight first pass +> could not be force-stopped in this harness build, so the ruling ALSO binds +> consolidation as a hard filter — any first-pass finding citing rustynum or +> symbiont is re-anchored to ndarray / marked deprecated-source, or its cell +> is surgically re-run with the corrected brief. No second-pass or attack +> work may touch either. **Deltalake removal ratified** ("we don't need +> deltalake") — see E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1. +> +> arc's source of truth. It SUPERSEDES the sequencing of the earlier lotus +> charter's phases 2–8 (docs/lotus/ deliverables 3–9 now route through this +> program); the landed Phase 0/1 deliverables (audit + F-ORD-REAL falsifier, +> PR #961) stand as prior session output that the researchers deliberately +> do NOT read during the first pass (§11 independence). + +## Title + +An Experimental Study of Bidirectional Erasure-Coded Seals and +Locality-Aware Compaction in Versioned Structure-of-Arrays Grid Stores + +Subtitle: Morton-Cascade Amortization, Temporal Deinterlacing, Reed–Solomon +Repair Locality, and Index-Remapped Rewrite Policies in Lance + +## 0. Posture + +This is a research program, NOT an implementation sprint. Do not begin by +implementing the proposed architecture. Fifteen independent researchers: +5 research domains × 3 independent agents per domain. They FIRST investigate +independently, THEN consolidate evidence. Only AFTER consolidation may they +attack implementation / upstream candidate work. + +The objective is not to vindicate the proposed idea. The objective is to +determine which parts survive contact with: exact Lance source; measurements; +coding theory; storage-system literature; temporal/database literature; +cache/SoA behavior; adversarial falsification. Unexpected negative results +are high-value results. Actively search for previously unnoticed structural +equivalences, cross-domain correspondences, and counterexamples. Do NOT claim +novelty merely because no one on this team remembers prior art. Every +suspected novel pattern requires a prior-art search. + +## 1. Source of truth + +Repositories: AdaWorldAPI/lance-graph, AdaWorldAPI/OGAR, AdaWorldAPI/ndarray, +lance upstream. (AdaWorldAPI/rustynum appeared in the original program text +and was STRUCK by the same-day scope pivot — see the header note.) Temporal reference: OGAR commit +`386a6fd848334b1d880c8408b3810f045d135cfe`, `docs/TEMPORAL-TIME-TRAVEL.md` — +read it literally. Temporal concepts: current tick / current Lance version; +last awareness tick / reference horizon; write tick; hindsight knowledge; +STRICT / AWARE / RETRO access; causal vs wall-clock order; possible HLC +coordinate. Do not reduce temporal.rs to an ordering utility: the research +hypothesis treats it as temporal deinterlacing `T_now × A_last × W_write` +with an interlacing window resembling `A_last < W_write <= T_now` — +investigate the actual implementation before relying on this shorthand. + +## 2. Strict Lance source discipline + +TWO evidence columns: (A) EXACT PINNED LANCE 9.0.0; (B) CURRENT LANCE +UPSTREAM. Never infer 9.0.0 capability from current upstream. Record per +claim: exact crate/version, Cargo.lock checksum, file, line, feature gate, +API stability, public API vs internal mechanism. Upstream appears to contain +machinery around: file compaction; reencode/binary-copy modes; incremental +compaction limits; excluded fragment boundaries; deferred index remapping; +fragment reuse indices; old-row-address → new-row-address remapping — +VERIFY all independently. + +Session anchors (dispatch note): column A at `/tmp/sources/lance-9` +(upstream tag v9.0.0; workspace Cargo.lock checksum +`23d04bed056e254bc6e31264b031c8492507ca57939586f016924081dcf221a9`); +column B at `/tmp/sources/lance-main` (cloned 2026-08-18). + +## 3. Baseline data geometry (hypotheses, not frozen architecture) + +One V3-style logical row = 512 B. Small work field = 4×4 rows = 16 rows = +8 KiB. Cycle = 4096 fields × 8 KiB = 32 MiB = 65,536 rows. 2-D hierarchy: +4=2×2, 16=4×4, 64=8×8, 256=16×16, 1024=32×32, 4096=64×64. Inverse +reduction: 4096→1024→256→64→16→4→1. + +## 4. Core distinctions + +Never conflate: semantic identity; logical grid locus; worker completion +order; semantic/write time; awareness time; Lance DatasetVersion; physical +fragment; physical row address; secondary-index row address; compaction +order; parity group. Arrival may determine WHEN work executes; it must not +accidentally determine semantic identity. Physical relocation must not change +logical identity. Reed–Solomon protection, if used, is initially defined over +canonical LOGICAL bytes/symbols, not encoded Lance page bytes. + +## 5. The 5×3 research matrix + +- **A — Lance storage/compaction**: A1 builder (map exact 9.0.0: + transactions, fragments, writer path, compaction planner, fragment IDs, + row IDs, stable IDs, index remapping, version publication, GC, prepared + writes); A2 adversary (destroy the thesis that random first-come chunk + placement is cheap: fragmentation, index-remap, page-locality, point-read, + range-read, long-retention failures); A3 upstream scout (map evolution + beyond 9.0.0; identify generic extension seams proposable upstream without + GridLake semantics). +- **B — Spatial layout / Morton cascade**: B1 builder (locality-key + generators: Morton, Hilbert, raster, random, project-native Morton/HHTL); + B2 adversary (adversarial query distributions: stripes, checkerboards, + diagonals, sparse hotspots, moving windows, skewed clusters, temporal + bands); B3 literature scout (SFCs, clustering metrics, cache-oblivious + layout, learned SFCs — do not assume Morton wins). +- **C — Erasure coding / seals**: C1 builder (checksum/hash; XOR P; RAID6/RS + P+Q; 2-D product codes; hierarchical cascade parity); C2 adversary (inject: + single/double erasure, silent corruption, wrong-slot substitution, stale + chunk, duplicate chunk, correlated/boundary/phase-aligned failures; measure + detect/localize/repair/falsely-accept); C3 coding-theory scout (RS repair, + MDS, LRC, regenerating codes, product/array codes, repair bandwidth, + subpacketization, repair locality — no home-grown ECC accepted merely + because tests pass). +- **D — Temporal deinterlacing**: D1 builder (map temporal.rs + OGAR + time-travel to versioned reads; model T_now × A_last × W_write; test + STRICT/AWARE/RETRO where implemented); D2 adversary (late facts, reordered + workers, same semantic write under different scheduler timing, + cross-version reads, stale awareness, clock skew, restart, cross-server + causal ambiguity; prove scheduler chronology does not become semantic + chronology); D3 literature scout (bitemporal DBs, transaction vs valid + time, snapshot isolation, causal consistency, HLC, temporal provenance, + retroactive queries). +- **E — SoA / cache / amortization**: E1 builder (benchmark harnesses: + 512 B rows, 8 KiB fields, cascade thresholds, first-come vs whole-64K, + canonical sort vs scatter vs index-remapped random placement); E2 + adversary (allocations, resident bytes, cache misses, bytes + copied/hashed/rewritten, fragment count, versions, index-remap cost, + read/write amplification, recovery cost; one experiment with compaction + disabled); E3 systems scout (LSM compaction design space, write + amplification, columnar reorganization, data skipping, clustering, + cache-oblivious structures, tiered/leveled, adaptive physical design). + +## 6. Required prior-art starting packet + +arXiv:2202.04522 (LSM compaction design space → trigger/layout/granularity/ +data-movement policy); arXiv:1206.3804 (LRC → repair locality, distance, +storage-rate tradeoff); arXiv:1509.04764 (Repairing RS → exact repair, +repair bandwidth, MDS constraints); arXiv:1806.04437 (EC for distributed +storage overview → LRC, regenerating, RS repair, access cost); +arXiv:1801.07399 (Onion Curve → SFC clustering metric, query-shape +dependence, counterexamples to Hilbert/Morton optimality); arXiv:2008.01684 +(SFCs for high-performance data mining → locality preservation, +cache-oblivious applications, transformation cost); arXiv:2009.06309 +(Data-Driven SFCs → adaptive ordering, multiscale/quadtree); +arXiv:1812.07123 (CausalSpartanX + HLC literature → causal timestamping, +visibility latency, clock anomalies, snapshot semantics). Find more; do not +stop at this packet. + +## 7. The phase/comma hypothesis + +"Pythagorean comma is ECC" is FORBIDDEN as a claim. The admissible +hypothesis: a deterministic incommensurate / phase-progressive schedule may +be useful for choosing an independent RS/product-code coefficient orientation +across spatial cascade levels. Candidate sources: comma-like phase +progression, QuintenZirkel, BGZ17/highheelBGZ arithmetic, φX, 11/17 +residues, CurveRuler. FIRST locate and characterize actual implementations; +then determine whether any candidate schedule (1) preserves matrix rank, +(2) preserves MDS/correction properties, (3) reduces cross-level syndrome +collisions, (4) improves localization, (5) costs ≤ SIMD work, (6) can be +reconstructed from locus/level instead of stored. Controls: standard RS +coefficients; sequential finite-field powers; random-but-seeded valid +coefficients; no modulation. If the project schedule loses to boring RS: +DELETE IT. + +## 8. GridLake / bidirectional code hypothesis + +Evaluate (not assume): A flat RS(k+2,k); B row P/Q + column P/Q; C product +code; D local P/Q + cascade parity; E hash-only syndromes; F payload parity; +G hybrids (strong per-chunk hashes + erasure parity). Report actual +redundancy; never call 12.5% "free"; distinguish CPU from storage cost. + +## 9. Compaction research questions (upstream must stay general) + +Locality-key ordered compaction; layout-aware rewrite grouping; optional +group boundaries; caller-provided sortable layout key; page/block permutation +when reencode is unnecessary; post-compaction permutation receipts; locality +statistics in CompactionMetrics; locality-debt trigger; stable logical row +IDs across physical moves; deferred physical clustering; incremental +locality repair. Do NOT upstream Morton/GridLake/Lotus/comma/temporal.rs/ +graph semantics where a generic u64 / expression / layout policy suffices. + +## 10. Cross-domain discovery mandate + +Every researcher looks for equivalences (questions, not conclusions): +compaction as locality "repair"; shared grouping for erasure repair and +query locality; temporal closure as compaction trigger; cascade completion +as spatial carry chain; fragment-reuse maps as permutation witnesses; +parity-group boundaries as compaction boundaries; locality debt from +index-remap entropy; one SFC order minimizing both query scatter and RS +repair scatter; prepared fragments as immutable code symbols before one +manifest publication; multi-scale syndromes identifying stale-version +contamination; temporal delta altering only a secondary syndrome +orientation; identity-derived layout keys making canonicalization +constructive. Classify every surprise: KNOWN / TRANSFER / NOVEL-CANDIDATE +(documented search required) / DISPROVED. + +## 11. Independence rule + +The three researchers inside one domain must NOT share intermediate +conclusions during the first pass. Each returns independently: SOURCE +ARCHAEOLOGY / MECHANISM / EXPECTED BENEFIT / EXPECTED FAILURE / EXPERIMENT / +PRIOR ART / SURPRISES / VERDICT. Only after all 15 reports exist do +consolidation agents see them. (Dispatch note: the first pass is also blind +to `docs/lotus/**` and all board files — prior sessions' conclusions would +synchronize the researchers.) + +## 12. Consolidation pass + +One evidence matrix: claim / source evidence / measurement evidence / +literature evidence / supporting agents / dissenting agents / confidence / +falsifier / next experiment. Then a cross-domain relationship graph. +Independently rediscovered equivalences are high-value; unanimity with +shared ancestry is NOT independent confirmation. + +## 13. Final attack pass (tiering) + +TIER 0 measurement/falsifiers only; TIER 1 lance-graph internal experiment; +TIER 2 generic Lance prototype; TIER 3 credible Lance upstream RFC/PR; +TIER 4 paper-worthy result. Every upstream candidate answers: general user +benefit; no lance-graph semantics in the API; backward compatibility; +effects on fragment format / indexes / compaction / binary-copy / object +storage / version retention; failure behavior; the benchmark that convinces +an upstream maintainer. + +## 14. Primary metrics + +logical bytes produced; physical bytes written; physical bytes rewritten; +write amplification; read amplification; point-read latency; +grid-neighborhood latency; random take latency; fragment count; files +touched/query; pages touched/query; index remap bytes; index remap CPU; +compaction CPU; compaction wall time; repair read bytes; repair write bytes; +repair helpers/chunks touched; seal CPU; seal metadata bytes; cache misses; +peak RSS; restart recovery work; version count; stale-version retention +bytes. Do not optimize a metric that was not measured. + +## 15. Falsifiers + +F-ARRIVAL (same semantic input, adversarial completion order → same +canonical result); F-LAYOUT (random physical placement → correct reads and +seals); F-COMPACT (before/after compaction → same logical payload, identity, +logical RS validity); F-RS (all promised erasure patterns reconstruct +exactly); F-SILENT (silent corruption detected before reconstruction); +F-PHASE (project phase progression performs no worse than standard baseline, +retains rank); F-NOPHASE (no measurable advantage → remove it); F-SFC +(Morton must defeat or justify itself against raster/Hilbert/Onion); +F-TEMPORAL (scheduler timing never changes semantic time); F-HINDSIGHT +(STRICT/AWARE/RETRO return the expected epistemic projections); F-NOCOMPACT +(healthy with compaction disabled for the soak window); F-AMPLIFY (locality +improvement justifies bytes rewritten); F-UPSTREAM (generic proposal useful +with GridLake completely absent). + +## 16. Final paper-shaped deliverable + +Title / Abstract / 1 Problem / 2 Background / 3 Lance archaeology / +4 Temporal model / 5 Grid-SoA model / 6 Erasure-code model / 7 +Locality-layout model / 8 Methodology / 9 Results / 10 Negative results / +11 Cross-domain discoveries / 12 Prior art / 13 Novelty audit / 14 Candidate +Lance extensions / 15 Threats to validity / 16 Conclusion. Appendices: +A source provenance; B falsifiers; C benchmarks; D coding matrices; +E locality curves; F compaction traces; G temporal traces; H agent +disagreement record. Dry systems-paper prose. Avoid "Lotus magic" / +"holographic" / "quantum" / "time machine" / "perfect random access" / +"self-healing" unless quantitatively demonstrated. Preferred vocabulary: +multi-scale, locality-preserving, version-aware, temporal-reference horizon, +incremental closure, logical-to-physical remapping, erasure-coded seal, +product-code, Reed–Solomon, space-filling curve, rewrite amplification, +repair locality, layout debt, compaction policy, structure-of-arrays, +prepared publication, causal timestamp, bidirectional syndrome, hierarchical +locality. + +## 17. Research maxim + +DO NOT MAKE THE BEAUTIFUL IDEA TRUE. MAKE IT SURVIVE FIFTEEN PEOPLE TRYING +TO MAKE IT FALSE. diff --git a/CLAUDE.md b/CLAUDE.md index f4ca8263..862b9bee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1163,13 +1163,20 @@ datafusion = "54" # OUR direct pin, in every crate that DEPENDS on it # `datafusion = []` under [features] — a FEATURE NAME, not a # version pin. It is not a counter-example.) # - # ⚠ BOTH MAJORS ARE REQUIRED — do NOT "fix" Cargo.lock to one. - # `datafusion 53.1.0` (+ -datasource, -physical-expr-adapter) - # also resolves because `deltalake-core 0.32.4` pins DF 53 - # upstream, and it backs the optional `delta` feature. Two - # semver majors legitimately coexist; collapsing them breaks - # `delta`. 53 is NOT a stale leftover — it is a live upstream - # constraint that lifts only when deltalake moves to DF 54. + # ⊘ SUPERSEDED 2026-08-18 (operator pin ruling, + # E-PIN-LANCE9-LANCEDB033-DF541-ARROW58-NO-DF53-1): the + # "BOTH MAJORS ARE REQUIRED" note that stood here is + # RETIRED. DF 53 resolved only through deltalake 0.32 + # behind the (already non-default, already broken) + # `delta` feature; deltalake tops out at 0.32.4/DF^53, + # so the feature was REMOVED and the lock now carries + # exactly ONE datafusion = 54.1.0. The ruling: lance 9 / + # lancedb 0.33 / datafusion 54.1 (no DF 53) / arrow 58, + # always, across AdaWorldAPI forks. A Delta reader + # can return via a git-pin on delta-rs upstream — its + # MAIN already pins DF 54 + arrow 58 (verified + # 2026-08-18; crates.io releases still DF 53) — as its + # own deliberate PR if a consumer needs Delta. # Probe: .claude/plans/lance9-datafusion54-upgrade-probe-v1.md lance = "=9.0.0" # exact-pinned: lancedb 0.33.0 requires lance =9.0.0 lance-linalg = "=9.0.0" @@ -1181,7 +1188,8 @@ rust = "1.97.1" # rust-toolchain.toml is authoritative; see its bump l ndarray = { path = "../../../ndarray" } # AdaWorldAPI fork, default, optional fallback nom = "7.1" snafu = "0.8" -deltalake = "0.32" # optional +# deltalake — REMOVED 2026-08-18 with the `delta` feature (see the datafusion +# supersession note above; it was the sole DF-53 source). ``` --- diff --git a/Cargo.lock b/Cargo.lock index 68b711c4..e5803888 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -363,7 +363,6 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed" dependencies = [ "bitflags 2.13.0", - "serde", "serde_core", "serde_json", ] @@ -608,31 +607,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "aws-sdk-dynamodb" -version = "1.116.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c937c55ae3030bec4431c0d9146e33d2b3e5f54bb47ed32597068200c56affc" -dependencies = [ - "arc-swap", - "aws-credential-types", - "aws-runtime", - "aws-smithy-async", - "aws-smithy-http", - "aws-smithy-json", - "aws-smithy-observability", - "aws-smithy-runtime", - "aws-smithy-runtime-api", - "aws-smithy-types", - "aws-types", - "bytes", - "fastrand", - "http 0.2.12", - "http 1.4.2", - "regex-lite", - "tracing", -] - [[package]] name = "aws-sdk-sso" version = "1.102.0" @@ -892,7 +866,6 @@ dependencies = [ "base64-simd", "bytes", "bytes-utils", - "futures-core", "http 0.2.12", "http 1.4.2", "http-body 0.4.6", @@ -905,8 +878,6 @@ dependencies = [ "ryu", "serde", "time", - "tokio", - "tokio-util", ] [[package]] @@ -1252,50 +1223,6 @@ version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" -[[package]] -name = "buoyant_kernel" -version = "0.22.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2235eb320cd7178862a32dd111bd0c0f71a368e393add4914c50129add478eab" -dependencies = [ - "arrow", - "buoyant_kernel_derive", - "bytes", - "chrono", - "crc", - "futures", - "indexmap 2.14.0", - "itertools 0.14.0", - "object_store", - "parquet", - "percent-encoding", - "rand 0.9.4", - "reqwest 0.13.4", - "roaring", - "rustc_version", - "serde", - "serde_json", - "strum 0.27.2", - "thiserror 2.0.18", - "tokio", - "tracing", - "tracing-subscriber", - "url", - "uuid", - "z85", -] - -[[package]] -name = "buoyant_kernel_derive" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3448e05bba811d98c73a466843abd8c16d0416dc083f92b66847693fcb0714f4" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "bytecheck" version = "0.8.2" @@ -1703,15 +1630,6 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" -[[package]] -name = "convert_case" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db05ffb6856bf0ecdf6367558a76a0e8a77b1713044eb92845c692100ed50190" -dependencies = [ - "unicode-segmentation", -] - [[package]] name = "core-foundation" version = "0.9.4" @@ -1765,21 +1683,6 @@ dependencies = [ "libc", ] -[[package]] -name = "crc" -version = "3.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" -dependencies = [ - "crc-catalog", -] - -[[package]] -name = "crc-catalog" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" - [[package]] name = "crc32c" version = "0.6.8" @@ -1941,19 +1844,8 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "424e0138278faeb2b401f174ad17e715c829512d74f3d1e81eb43365c2e0590e" dependencies = [ - "ctor-proc-macro 0.0.7", - "dtor 0.1.1", -] - -[[package]] -name = "ctor" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83cf0d42651b16c6dfe68685716d18480d18a9c39c62d76e8cf3eb6ed5d8bcbf" -dependencies = [ - "ctor-proc-macro 0.0.13", - "dtor 0.8.1", - "link-section 0.2.1", + "ctor-proc-macro", + "dtor", ] [[package]] @@ -1962,7 +1854,7 @@ version = "1.0.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d83cb7e7a873830708d6b02a78cd36a592c6fa14bf267b68725103b85c0d77f" dependencies = [ - "link-section 0.19.2", + "link-section", "linktime-proc-macro", ] @@ -1972,12 +1864,6 @@ version = "0.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52560adf09603e58c9a7ee1fe1dcb95a16927b17c127f0ac02d6e768a0e25bc1" -[[package]] -name = "ctor-proc-macro" -version = "0.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a949c44fcacbbbb7ada007dc7acb34603dd97cd47de5d054f2b6493ecebb483" - [[package]] name = "ctutils" version = "0.4.2" @@ -2082,61 +1968,6 @@ version = "2.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" -[[package]] -name = "datafusion" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93db0e623840612f7f2cd757f7e8a8922064192363732c88692e0870016e141b" -dependencies = [ - "arrow", - "arrow-schema", - "async-trait", - "bytes", - "bzip2", - "chrono", - "datafusion-catalog 53.1.0", - "datafusion-catalog-listing 53.1.0", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-datasource-arrow 53.1.0", - "datafusion-datasource-csv 53.1.0", - "datafusion-datasource-json 53.1.0", - "datafusion-datasource-parquet 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-functions-aggregate 53.1.0", - "datafusion-functions-nested 53.1.0", - "datafusion-functions-table 53.1.0", - "datafusion-functions-window 53.1.0", - "datafusion-optimizer 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-optimizer 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "datafusion-sql 53.1.0", - "flate2", - "futures", - "itertools 0.14.0", - "liblzma", - "log", - "object_store", - "parking_lot", - "parquet", - "rand 0.9.4", - "regex", - "sqlparser 0.61.0", - "tempfile", - "tokio", - "url", - "uuid", - "zstd", -] - [[package]] name = "datafusion" version = "54.1.0" @@ -2148,31 +1979,31 @@ dependencies = [ "async-trait", "bzip2", "chrono", - "datafusion-catalog 54.1.0", - "datafusion-catalog-listing 54.1.0", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-datasource-arrow 54.1.0", - "datafusion-datasource-csv 54.1.0", - "datafusion-datasource-json 54.1.0", - "datafusion-datasource-parquet 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate 54.1.0", - "datafusion-functions-nested 54.1.0", - "datafusion-functions-table 54.1.0", - "datafusion-functions-window 54.1.0", - "datafusion-optimizer 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-optimizer 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", - "datafusion-sql 54.1.0", + "datafusion-catalog", + "datafusion-catalog-listing", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-datasource-arrow", + "datafusion-datasource-csv", + "datafusion-datasource-json", + "datafusion-datasource-parquet", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-nested", + "datafusion-functions-table", + "datafusion-functions-window", + "datafusion-optimizer", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-optimizer", + "datafusion-physical-plan", + "datafusion-session", + "datafusion-sql", "flate2", "futures", "indexmap 2.14.0", @@ -2182,7 +2013,7 @@ dependencies = [ "object_store", "parking_lot", "parquet", - "sqlparser 0.62.0", + "sqlparser", "tempfile", "tokio", "url", @@ -2190,31 +2021,6 @@ dependencies = [ "zstd", ] -[[package]] -name = "datafusion-catalog" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37cefde60b26a7f4ff61e9d2ff2833322f91df2b568d7238afe67bde5bdffb66" -dependencies = [ - "arrow", - "async-trait", - "dashmap", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "itertools 0.14.0", - "log", - "object_store", - "parking_lot", - "tokio", -] - [[package]] name = "datafusion-catalog" version = "54.1.0" @@ -2224,14 +2030,14 @@ dependencies = [ "arrow", "async-trait", "dashmap", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-session", "futures", "itertools 0.14.0", "log", @@ -2240,29 +2046,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-catalog-listing" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17e112307715d6a7a331111a4c2330ff54bc237183511c319e3708a4cff431fb" -dependencies = [ - "arrow", - "async-trait", - "datafusion-catalog 53.1.0", - "datafusion-common 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "futures", - "itertools 0.14.0", - "log", - "object_store", -] - [[package]] name = "datafusion-catalog-listing" version = "54.1.0" @@ -2271,46 +2054,21 @@ checksum = "f0668fb32c12065ec242be0e5b4bc62bd7a06a0be3ecd83791ef877e4be67e02" dependencies = [ "arrow", "async-trait", - "datafusion-catalog 54.1.0", - "datafusion-common 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-catalog", + "datafusion-common", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", "futures", "itertools 0.14.0", "log", "object_store", ] -[[package]] -name = "datafusion-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d72a11ca44a95e1081870d3abb80c717496e8a7acb467a1d3e932bb636af5cc2" -dependencies = [ - "ahash", - "arrow", - "arrow-ipc", - "chrono", - "half", - "hashbrown 0.16.1", - "indexmap 2.14.0", - "itertools 0.14.0", - "libc", - "log", - "object_store", - "parquet", - "paste", - "recursive", - "sqlparser 0.61.0", - "tokio", - "web-time", -] - [[package]] name = "datafusion-common" version = "54.1.0" @@ -2331,23 +2089,12 @@ dependencies = [ "object_store", "parquet", "recursive", - "sqlparser 0.62.0", + "sqlparser", "tokio", "uuid", "web-time", ] -[[package]] -name = "datafusion-common-runtime" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89f4afaed29670ec4fd6053643adc749fe3f4bc9d1ce1b8c5679b22c67d12def" -dependencies = [ - "futures", - "log", - "tokio", -] - [[package]] name = "datafusion-common-runtime" version = "54.1.0" @@ -2359,41 +2106,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-datasource" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9fb386e1691355355a96419978a0022b7947b44d4a24a6ea99f00b6b485cbb6" -dependencies = [ - "arrow", - "async-compression", - "async-trait", - "bytes", - "bzip2", - "chrono", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "flate2", - "futures", - "glob", - "itertools 0.14.0", - "liblzma", - "log", - "object_store", - "rand 0.9.4", - "tokio", - "tokio-util", - "url", - "zstd", -] - [[package]] name = "datafusion-datasource" version = "54.1.0" @@ -2406,15 +2118,15 @@ dependencies = [ "bytes", "bzip2", "chrono", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "flate2", "futures", "glob", @@ -2430,30 +2142,6 @@ dependencies = [ "zstd", ] -[[package]] -name = "datafusion-datasource-arrow" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffa6c52cfed0734c5f93754d1c0175f558175248bf686c944fb05c373e5fc096" -dependencies = [ - "arrow", - "arrow-ipc", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "itertools 0.14.0", - "object_store", - "tokio", -] - [[package]] name = "datafusion-datasource-arrow" version = "54.1.0" @@ -2464,43 +2152,20 @@ dependencies = [ "arrow-ipc", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "futures", "itertools 0.14.0", "object_store", "tokio", ] -[[package]] -name = "datafusion-datasource-csv" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "503f29e0582c1fc189578d665ff57d9300da1f80c282777d7eb67bb79fb8cdca" -dependencies = [ - "arrow", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "object_store", - "regex", - "tokio", -] - [[package]] name = "datafusion-datasource-csv" version = "54.1.0" @@ -2510,44 +2175,20 @@ dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "futures", "object_store", "regex", "tokio", ] -[[package]] -name = "datafusion-datasource-json" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e33804749abc8d0c8cb7473228483cb8070e524c6f6086ee1b85a64debe2b3d2" -dependencies = [ - "arrow", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-session 53.1.0", - "futures", - "object_store", - "serde_json", - "tokio", - "tokio-stream", -] - [[package]] name = "datafusion-datasource-json" version = "54.1.0" @@ -2557,50 +2198,20 @@ dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-session", "futures", "object_store", "tokio", "tokio-stream", ] -[[package]] -name = "datafusion-datasource-parquet" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a8e0365e0e08e8ff94d912f0ababcf9065a1a304018ba90b1fc83c855b4997" -dependencies = [ - "arrow", - "async-trait", - "bytes", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-pruning 53.1.0", - "datafusion-session 53.1.0", - "futures", - "itertools 0.14.0", - "log", - "object_store", - "parking_lot", - "parquet", - "tokio", -] - [[package]] name = "datafusion-datasource-parquet" version = "54.1.0" @@ -2610,19 +2221,19 @@ dependencies = [ "arrow", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-adapter 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-pruning 54.1.0", - "datafusion-session 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-datasource", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr", + "datafusion-physical-expr-adapter", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", + "datafusion-session", "futures", "itertools 0.14.0", "log", @@ -2632,41 +2243,12 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-doc" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8de6ac0df1662b9148ad3c987978b32cbec7c772f199b1d53520c8fa764a87ee" - [[package]] name = "datafusion-doc" version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d69bb69d8769e34f76839c960dbde24c1ac0c885a79b6c3c2287bdc56ec67891" -[[package]] -name = "datafusion-execution" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c03c7fbdaefcca4ef6ffe425a5fc2325763bfb426599bb0bf4536466efabe709" -dependencies = [ - "arrow", - "arrow-buffer", - "async-trait", - "chrono", - "dashmap", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "futures", - "log", - "object_store", - "parking_lot", - "rand 0.9.4", - "tempfile", - "url", -] - [[package]] name = "datafusion-execution" version = "54.1.0" @@ -2677,9 +2259,9 @@ dependencies = [ "arrow-buffer", "async-trait", "dashmap", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr-common", "futures", "log", "object_store", @@ -2689,29 +2271,6 @@ dependencies = [ "url", ] -[[package]] -name = "datafusion-expr" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "574b9b6977fedbd2a611cbff12e5caf90f31640ad9dc5870f152836d94bad0dd" -dependencies = [ - "arrow", - "async-trait", - "chrono", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-functions-window-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "indexmap 2.14.0", - "itertools 0.14.0", - "paste", - "recursive", - "serde_json", - "sqlparser 0.61.0", -] - [[package]] name = "datafusion-expr" version = "54.1.0" @@ -2722,30 +2281,17 @@ dependencies = [ "arrow-schema", "async-trait", "chrono", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-functions-window-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr-common", "indexmap 2.14.0", "itertools 0.14.0", "recursive", "serde_json", - "sqlparser 0.62.0", -] - -[[package]] -name = "datafusion-expr-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d7c3adf3db8bf61e92eb90cb659c8e8b734593a8f7c8e12a843c7ddba24b87e" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "indexmap 2.14.0", - "itertools 0.14.0", - "paste", + "sqlparser", ] [[package]] @@ -2755,43 +2301,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7b19a8c95522bee8cbb313d74263b85e355d2b52f42e67ef5694bf5de9e9356" dependencies = [ "arrow", - "datafusion-common 54.1.0", + "datafusion-common", "indexmap 2.14.0", "itertools 0.14.0", ] -[[package]] -name = "datafusion-functions" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f28aa4e10384e782774b10e72aca4d93ef7b31aa653095d9d4536b0a3dbc51b6" -dependencies = [ - "arrow", - "arrow-buffer", - "base64", - "blake2", - "blake3", - "chrono", - "chrono-tz", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-macros 53.1.0", - "hex", - "itertools 0.14.0", - "log", - "md-5 0.10.6", - "memchr", - "num-traits", - "rand 0.9.4", - "regex", - "sha2 0.10.9", - "unicode-segmentation", - "uuid", -] - [[package]] name = "datafusion-functions" version = "54.1.0" @@ -2805,13 +2319,13 @@ dependencies = [ "blake3", "chrono", "chrono-tz", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-macros", + "datafusion-physical-expr-common", "hex", "itertools 0.14.0", "log", @@ -2824,28 +2338,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "datafusion-functions-aggregate" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00aa6217e56098ba84e0a338176fe52f0a84cca398021512c6c8c5eff806d0ad" -dependencies = [ - "ahash", - "arrow", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-macros 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "half", - "log", - "num-traits", - "paste", -] - [[package]] name = "datafusion-functions-aggregate" version = "54.1.0" @@ -2853,33 +2345,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89bc17041e424a47ed062f43df24d84aab8b57c4c3221e5c1a5eef46d6c5718b" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "foldhash 0.2.0", "half", "log", "num-traits", ] -[[package]] -name = "datafusion-functions-aggregate-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b511250349407db7c43832ab2de63f5557b19a20dfd236b39ca2c04468b50d47" -dependencies = [ - "ahash", - "arrow", - "datafusion-common 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", -] - [[package]] name = "datafusion-functions-aggregate-common" version = "54.1.0" @@ -2887,34 +2366,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97dd2a9e865c6108059f5b37b77934f84b50bfb108f837bd0e5c9536e03f0545" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", -] - -[[package]] -name = "datafusion-functions-nested" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef13a858e20d50f0a9bb5e96e7ac82b4e7597f247515bccca4fdd2992df0212a" -dependencies = [ - "arrow", - "arrow-ord", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-functions-aggregate 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-macros 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "hashbrown 0.16.1", - "itertools 0.14.0", - "itoa", - "log", - "paste", + "datafusion-common", + "datafusion-expr-common", + "datafusion-physical-expr-common", ] [[package]] @@ -2925,16 +2379,16 @@ checksum = "75f0bdfeef16d96417b9632ef855645376b242e9006a126dfd0bedfc54a93f5f" dependencies = [ "arrow", "arrow-ord", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions", + "datafusion-functions-aggregate", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr-common", "hashbrown 0.17.1", "itertools 0.14.0", "itoa", @@ -2942,22 +2396,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "datafusion-functions-table" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b40d3f5bbb3905f9ccb1ce9485a9595c77b69758a7c24d3ba79e334ff51e7e" -dependencies = [ - "arrow", - "async-trait", - "datafusion-catalog 53.1.0", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-plan 53.1.0", - "parking_lot", - "paste", -] - [[package]] name = "datafusion-functions-table" version = "54.1.0" @@ -2966,32 +2404,14 @@ checksum = "f4e4941673c917819616877e9993da4503e4f4739812be0bc32c5356184c6383" dependencies = [ "arrow", "async-trait", - "datafusion-catalog 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-catalog", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", + "datafusion-physical-plan", "parking_lot", ] -[[package]] -name = "datafusion-functions-window" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4e88ec9d57c9b685d02f58bfee7be62d72610430ddcedb82a08e5d9925dbfb6" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-doc 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-window-common 53.1.0", - "datafusion-macros 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "log", - "paste", -] - [[package]] name = "datafusion-functions-window" version = "54.1.0" @@ -2999,45 +2419,24 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "12dd2e16c12b84b6f6b41b19f55b366dd1c46876bb35b86896c6349067379e8d" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-doc 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-window-common 54.1.0", - "datafusion-macros 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-doc", + "datafusion-expr", + "datafusion-functions-window-common", + "datafusion-macros", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "log", ] -[[package]] -name = "datafusion-functions-window-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8307bb93519b1a91913723a1130cfafeee3f72200d870d88e91a6fc5470ede5c" -dependencies = [ - "datafusion-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", -] - [[package]] name = "datafusion-functions-window-common" version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cdc5e4b6f8b6ef823cc1c761f85088ad4c884fe8df64df3cbcc6b2b84698441" dependencies = [ - "datafusion-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", -] - -[[package]] -name = "datafusion-macros" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e367e6a71051d0ebdd29b2f85d12059b38b1d1f172c6906e80016da662226bd" -dependencies = [ - "datafusion-doc 53.1.0", - "quote", - "syn", + "datafusion-common", + "datafusion-physical-expr-common", ] [[package]] @@ -3046,31 +2445,11 @@ version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1a3614234dd93578c92428cb4f408e020874f0d2b7e6c90c928d9d28b5df2ceb" dependencies = [ - "datafusion-doc 54.1.0", + "datafusion-doc", "quote", "syn", ] -[[package]] -name = "datafusion-optimizer" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e929015451a67f77d9d8b727b2bf3a40c4445fdef6cdc53281d7d97c76888ace" -dependencies = [ - "arrow", - "chrono", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "indexmap 2.14.0", - "itertools 0.14.0", - "log", - "recursive", - "regex", - "regex-syntax", -] - [[package]] name = "datafusion-optimizer" version = "54.1.0" @@ -3079,10 +2458,10 @@ checksum = "a0635620b050b81bb92764e99250868f654e2cd5ad1bece413283b3f73c83179" dependencies = [ "arrow", "chrono", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", "indexmap 2.14.0", "itertools 0.14.0", "log", @@ -3091,30 +2470,6 @@ dependencies = [ "regex-syntax", ] -[[package]] -name = "datafusion-physical-expr" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b1e68aba7a4b350401cfdf25a3d6f989ad898a7410164afe9ca52080244cb59" -dependencies = [ - "ahash", - "arrow", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "half", - "hashbrown 0.16.1", - "indexmap 2.14.0", - "itertools 0.14.0", - "parking_lot", - "paste", - "petgraph 0.8.3", - "recursive", - "tokio", -] - [[package]] name = "datafusion-physical-expr" version = "54.1.0" @@ -3122,11 +2477,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8cabf7a86eb70b816729e33c81bf7767c936ee1226f607a114f5dac2decac8d0" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr-common", "half", "hashbrown 0.17.1", "indexmap 2.14.0", @@ -3137,21 +2492,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-physical-expr-adapter" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea22315f33cf2e0adc104e8ec42e285f6ed93998d565c65e82fec6a9ee9f9db4" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "itertools 0.14.0", -] - [[package]] name = "datafusion-physical-expr-adapter" version = "54.1.0" @@ -3159,31 +2499,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "de222e04f7e6744501555a54ab0abe26bfdfebee380af79a9bdc175704246859" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "itertools 0.14.0", ] -[[package]] -name = "datafusion-physical-expr-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b04b45ea8ad3ac2d78f2ea2a76053e06591c9629c7a603eda16c10649ecf4362" -dependencies = [ - "ahash", - "arrow", - "chrono", - "datafusion-common 53.1.0", - "datafusion-expr-common 53.1.0", - "hashbrown 0.16.1", - "indexmap 2.14.0", - "itertools 0.14.0", - "parking_lot", -] - [[package]] name = "datafusion-physical-expr-common" version = "54.1.0" @@ -3192,8 +2515,8 @@ checksum = "72d0d0057fc5a502d45c870cb6d47c66eb7bdd5edb1bd71ad6f3f724975ac2a8" dependencies = [ "arrow", "chrono", - "datafusion-common 54.1.0", - "datafusion-expr-common 54.1.0", + "datafusion-common", + "datafusion-expr-common", "hashbrown 0.17.1", "indexmap 2.14.0", "itertools 0.14.0", @@ -3202,73 +2525,22 @@ dependencies = [ ] [[package]] -name = "datafusion-physical-optimizer" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cb13397809a425918f608dfe8653f332015a3e330004ab191b4404187238b95" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-pruning 53.1.0", - "itertools 0.14.0", - "recursive", -] - -[[package]] -name = "datafusion-physical-optimizer" -version = "54.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "86046eed10950c5f9aaed9acfd148e9bd2e1dfdfe4f9aef607d1447b271e4183" -dependencies = [ - "arrow", - "datafusion-common 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-pruning 54.1.0", - "itertools 0.14.0", - "recursive", -] - -[[package]] -name = "datafusion-physical-plan" -version = "53.1.0" +name = "datafusion-physical-optimizer" +version = "54.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5edc023675791af9d5fb4cc4c24abf5f7bd3bd4dcf9e5bd90ea1eff6976dcc79" +checksum = "86046eed10950c5f9aaed9acfd148e9bd2e1dfdfe4f9aef607d1447b271e4183" dependencies = [ - "ahash", "arrow", - "arrow-ord", - "arrow-schema", - "async-trait", - "datafusion-common 53.1.0", - "datafusion-common-runtime 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions 53.1.0", - "datafusion-functions-aggregate-common 53.1.0", - "datafusion-functions-window-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "futures", - "half", - "hashbrown 0.16.1", - "indexmap 2.14.0", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "datafusion-pruning", "itertools 0.14.0", - "log", - "num-traits", - "parking_lot", - "pin-project-lite", - "tokio", + "recursive", ] [[package]] @@ -3283,15 +2555,15 @@ dependencies = [ "arrow-ord", "arrow-schema", "async-trait", - "datafusion-common 54.1.0", - "datafusion-common-runtime 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-functions-aggregate-common 54.1.0", - "datafusion-functions-window-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", + "datafusion-common", + "datafusion-common-runtime", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "futures", "half", "hashbrown 0.17.1", @@ -3304,62 +2576,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "datafusion-proto" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a387aaef949dc16bb6abc81bd1af850ec7449183aef011214f9724957495738" -dependencies = [ - "arrow", - "chrono", - "datafusion-catalog 53.1.0", - "datafusion-catalog-listing 53.1.0", - "datafusion-common 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-datasource-arrow 53.1.0", - "datafusion-datasource-csv 53.1.0", - "datafusion-datasource-json 53.1.0", - "datafusion-datasource-parquet 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-table 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "datafusion-proto-common", - "object_store", - "prost 0.14.4", - "rand 0.9.4", -] - -[[package]] -name = "datafusion-proto-common" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16e614c7c53a9c304c6a850b821010bb492e57300311835f1180613f9d2c63d9" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "prost 0.14.4", -] - -[[package]] -name = "datafusion-pruning" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac8c76860e355616555081cab5968cec1af7a80701ff374510860bcd567e365a" -dependencies = [ - "arrow", - "datafusion-common 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-expr-common 53.1.0", - "datafusion-physical-expr 53.1.0", - "datafusion-physical-expr-common 53.1.0", - "datafusion-physical-plan 53.1.0", - "itertools 0.14.0", - "log", -] - [[package]] name = "datafusion-pruning" version = "54.1.0" @@ -3367,29 +2583,15 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eb63eeac6de19be40f487b65dd84e546195f783c5a9928618e0c4f2a3569b0d7" dependencies = [ "arrow", - "datafusion-common 54.1.0", - "datafusion-datasource 54.1.0", - "datafusion-expr-common 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-expr-common 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-common", + "datafusion-datasource", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", "log", ] -[[package]] -name = "datafusion-session" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5412111aa48e2424ba926112e192f7a6b7e4ccb450145d25ce5ede9f19dc491e" -dependencies = [ - "async-trait", - "datafusion-common 53.1.0", - "datafusion-execution 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-physical-plan 53.1.0", - "parking_lot", -] - [[package]] name = "datafusion-session" version = "54.1.0" @@ -3397,32 +2599,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f961d209177f91bd014db5cbb2c33b7d28a2597b9003e77f17aeb712964315a" dependencies = [ "async-trait", - "datafusion-common 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-plan", "parking_lot", ] -[[package]] -name = "datafusion-sql" -version = "53.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa0d133ddf8b9b3b872acac900157f783e7b879fe9a6bccf389abebbfac45ec1" -dependencies = [ - "arrow", - "bigdecimal", - "chrono", - "datafusion-common 53.1.0", - "datafusion-expr 53.1.0", - "datafusion-functions-nested 53.1.0", - "indexmap 2.14.0", - "log", - "recursive", - "regex", - "sqlparser 0.61.0", -] - [[package]] name = "datafusion-sql" version = "54.1.0" @@ -3432,14 +2615,14 @@ dependencies = [ "arrow", "bigdecimal", "chrono", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-nested 54.1.0", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-nested", "indexmap 2.14.0", "log", "recursive", "regex", - "sqlparser 0.62.0", + "sqlparser", ] [[package]] @@ -3500,146 +2683,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "deltalake" -version = "0.32.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2695b91fd02afd5b4726a190fd58019674492d824e30598391390b0307644da8" -dependencies = [ - "buoyant_kernel", - "ctor 0.10.1", - "deltalake-aws", - "deltalake-azure", - "deltalake-core", - "deltalake-gcp", -] - -[[package]] -name = "deltalake-aws" -version = "0.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b3ef837d5d58bb0cd62dc9ee3bc201805751090f4bab2ca0ff09669661814fd4" -dependencies = [ - "async-trait", - "aws-config", - "aws-credential-types", - "aws-sdk-dynamodb", - "aws-sdk-sts", - "aws-smithy-runtime-api", - "backon", - "bytes", - "chrono", - "deltalake-core", - "futures", - "object_store", - "regex", - "thiserror 2.0.18", - "tokio", - "tracing", - "typed-builder", - "url", - "uuid", -] - -[[package]] -name = "deltalake-azure" -version = "0.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1b7099351f8d452374d75eda599cd7633a50f4a36ec318532ec6c5d1f9d908d" -dependencies = [ - "bytes", - "deltalake-core", - "object_store", - "thiserror 2.0.18", - "tokio", - "url", -] - -[[package]] -name = "deltalake-core" -version = "0.32.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4588e95ff3b2ccdba56d9ec262bd3467c0593000f729402528706f62be8be1ca" -dependencies = [ - "arrow", - "arrow-arith", - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-ipc", - "arrow-json", - "arrow-ord", - "arrow-row", - "arrow-schema", - "arrow-select", - "async-trait", - "buoyant_kernel", - "bytes", - "cfg-if 1.0.4", - "chrono", - "dashmap", - "datafusion 53.1.0", - "datafusion-datasource 53.1.0", - "datafusion-physical-expr-adapter 53.1.0", - "datafusion-proto", - "deltalake-derive", - "dirs", - "either", - "futures", - "humantime", - "indexmap 2.14.0", - "itertools 0.14.0", - "num_cpus", - "object_store", - "parking_lot", - "parquet", - "percent-encoding", - "percent-encoding-rfc3986", - "pin-project-lite", - "rand 0.10.1", - "regex", - "serde", - "serde_json", - "sqlparser 0.61.0", - "strum 0.27.2", - "thiserror 2.0.18", - "tokio", - "tracing", - "url", - "uuid", - "validator", -] - -[[package]] -name = "deltalake-derive" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d5a646e2b05608db1e5f91e75b6af841d92294ea0864789bb01f3f818e5f7b25" -dependencies = [ - "convert_case", - "itertools 0.14.0", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "deltalake-gcp" -version = "0.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "909d4c00c77ed1359e2d3b6a25641740f7a068a8c20e951e479e1212e67004ea" -dependencies = [ - "async-trait", - "bytes", - "deltalake-core", - "futures", - "object_store", - "thiserror 2.0.18", - "tokio", - "tracing", - "url", -] - [[package]] name = "der" version = "0.7.10" @@ -3771,16 +2814,7 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "404d02eeb088a82cfd873006cb713fe411306c7d182c344905e101fb1167d301" dependencies = [ - "dtor-proc-macro 0.0.6", -] - -[[package]] -name = "dtor" -version = "0.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edf234dd1594d6dd434a8fb8cada51ddbbc593e40e4a01556a0b31c62da2775b" -dependencies = [ - "dtor-proc-macro 0.0.13", + "dtor-proc-macro", ] [[package]] @@ -3789,12 +2823,6 @@ version = "0.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f678cf4a922c215c63e0de95eb1ff08a958a81d47e485cf9da1e27bf6305cfa5" -[[package]] -name = "dtor-proc-macro" -version = "0.0.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2647271c92754afcb174e758003cfd1cbf1e43e5a7853d7b1813e63e19e39a73" - [[package]] name = "dunce" version = "1.0.5" @@ -4230,7 +3258,7 @@ dependencies = [ "arrow-array", "arrow-buffer", "arrow-schema", - "datafusion 54.1.0", + "datafusion", "geo", "geo-traits", "geoarrow-array", @@ -5275,11 +4303,11 @@ dependencies = [ "crossbeam-queue", "crossbeam-skiplist", "dashmap", - "datafusion 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-plan", "either", "fst", "futures", @@ -5398,8 +4426,8 @@ dependencies = [ "async-trait", "byteorder", "bytes", - "datafusion-common 54.1.0", - "datafusion-sql 54.1.0", + "datafusion-common", + "datafusion-sql", "futures", "itertools 0.14.0", "lance-arrow", @@ -5440,10 +4468,10 @@ dependencies = [ "arrow-select", "async-trait", "chrono", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-functions", + "datafusion-physical-expr", "futures", "jsonb", "lance-arrow", @@ -5541,7 +4569,7 @@ dependencies = [ "async-trait", "byteorder", "bytes", - "datafusion-common 54.1.0", + "datafusion-common", "futures", "lance-arrow", "lance-core", @@ -5563,7 +4591,7 @@ version = "9.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97739c10f2c818ac4aa8a798a115e11c790c174d016c0836792aa3b8d0f981ab" dependencies = [ - "datafusion 54.1.0", + "datafusion", "geo-traits", "geo-types", "geoarrow-array", @@ -5586,13 +4614,12 @@ dependencies = [ "bytes", "causal-edge", "chrono", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions-aggregate 54.1.0", - "datafusion-sql 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-expr", + "datafusion-functions-aggregate", + "datafusion-sql", "deepnsm", - "deltalake", "futures", "lance", "lance-arrow", @@ -5613,7 +4640,6 @@ dependencies = [ "snafu 0.8.9", "tempfile", "tokio", - "url", ] [[package]] @@ -5650,7 +4676,7 @@ dependencies = [ "axum 0.8.9", "chrono", "clap", - "datafusion 54.1.0", + "datafusion", "flate2", "futures", "lance", @@ -5674,7 +4700,7 @@ version = "0.5.4" dependencies = [ "arrow-schema", "async-trait", - "datafusion 54.1.0", + "datafusion", "lance-namespace", "reqwest 0.12.28", "serde", @@ -5836,10 +4862,10 @@ dependencies = [ "bytes", "chrono", "crossbeam-queue", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-physical-expr 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-expr", + "datafusion-physical-expr", "dirs", "fst", "futures", @@ -5899,9 +4925,9 @@ dependencies = [ "arrow-select", "async-trait", "bytes", - "datafusion 54.1.0", - "datafusion-common 54.1.0", - "datafusion-expr 54.1.0", + "datafusion", + "datafusion-common", + "datafusion-expr", "futures", "lance-core", "lance-io", @@ -5999,8 +5025,8 @@ dependencies = [ "arrow-schema", "async-trait", "bytes", - "datafusion-common 54.1.0", - "datafusion-physical-plan 54.1.0", + "datafusion-common", + "datafusion-physical-plan", "futures", "lance", "lance-core", @@ -6124,15 +5150,15 @@ dependencies = [ "async-trait", "bytes", "chrono", - "datafusion 54.1.0", - "datafusion-catalog 54.1.0", - "datafusion-common 54.1.0", - "datafusion-execution 54.1.0", - "datafusion-expr 54.1.0", - "datafusion-functions 54.1.0", - "datafusion-physical-expr 54.1.0", - "datafusion-physical-plan 54.1.0", - "datafusion-sql 54.1.0", + "datafusion", + "datafusion-catalog", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions", + "datafusion-physical-expr", + "datafusion-physical-plan", + "datafusion-sql", "futures", "half", "lance", @@ -6352,12 +5378,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "link-section" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b685d66585d646efe09fec763d796c291049c8b6bf84e04954bffc8748341f0d" - [[package]] name = "link-section" version = "0.19.2" @@ -7503,12 +6523,6 @@ version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" -[[package]] -name = "percent-encoding-rfc3986" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3637c05577168127568a64e9dc5a6887da720efef07b3d9472d45f63ab191166" - [[package]] name = "permutation" version = "0.4.1" @@ -8501,10 +7515,8 @@ checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" dependencies = [ "base64", "bytes", - "encoding_rs", "futures-core", "futures-util", - "h2", "http 1.4.2", "http-body 1.0.1", "http-body-util", @@ -8513,7 +7525,6 @@ dependencies = [ "hyper-util", "js-sys", "log", - "mime", "percent-encoding", "pin-project-lite", "quinn", @@ -9302,17 +8313,6 @@ dependencies = [ "der", ] -[[package]] -name = "sqlparser" -version = "0.61.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dbf5ea8d4d7c808e1af1cbabebca9a2abe603bcefc22294c5b95018d53200cb7" -dependencies = [ - "log", - "recursive", - "sqlparser_derive", -] - [[package]] name = "sqlparser" version = "0.62.0" @@ -9406,15 +8406,6 @@ dependencies = [ "strum_macros 0.26.4", ] -[[package]] -name = "strum" -version = "0.27.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" -dependencies = [ - "strum_macros 0.27.2", -] - [[package]] name = "strum" version = "0.28.0" @@ -9437,18 +8428,6 @@ dependencies = [ "syn", ] -[[package]] -name = "strum_macros" -version = "0.27.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7" -dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "strum_macros" version = "0.28.0" @@ -10242,26 +9221,6 @@ dependencies = [ "rand 0.9.4", ] -[[package]] -name = "typed-builder" -version = "0.23.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31aa81521b70f94402501d848ccc0ecaa8f93c8eb6999eb9747e72287757ffda" -dependencies = [ - "typed-builder-macro", -] - -[[package]] -name = "typed-builder-macro" -version = "0.23.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "076a02dc54dd46795c2e9c8282ed40bcfb1e22747e955de9389a1de28190fb26" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "typenum" version = "1.20.1" @@ -10335,7 +9294,6 @@ dependencies = [ "idna", "percent-encoding", "serde", - "serde_derive", ] [[package]] @@ -10376,42 +9334,11 @@ checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" dependencies = [ "getrandom 0.4.3", "js-sys", - "rand 0.10.1", "serde_core", "sha1_smol", "wasm-bindgen", ] -[[package]] -name = "validator" -version = "0.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0b4a29d8709210980a09379f27ee31549b73292c87ab9899beee1c0d3be6303" -dependencies = [ - "idna", - "once_cell", - "regex", - "serde", - "serde_derive", - "serde_json", - "url", - "validator_derive", -] - -[[package]] -name = "validator_derive" -version = "0.19.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bac855a2ce6f843beb229757e6e570a42e837bcb15e5f449dd48d5747d41bf77" -dependencies = [ - "darling 0.20.11", - "once_cell", - "proc-macro-error2", - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "valuable" version = "0.1.1" @@ -11197,12 +10124,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "z85" -version = "3.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6e61e59a957b7ccee15d2049f86e8bfd6f66968fcd88f018950662d9b86e675" - [[package]] name = "zerocopy" version = "0.8.52" diff --git a/Cargo.toml b/Cargo.toml index eeb4a1ea..dbb7f1c3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -240,3 +240,12 @@ resolver = "2" # unfetchable, so the git form deadlocks offline sessions. Path form = zero # network, identical source. ndarray = { path = "../ndarray" } + +# Dev/test debuginfo OFF (operator, 2026-08-18: "Cargo Debug*=0 or so makes it +# much smaller and faster"). Measured incident the same day: full debuginfo +# test binaries grew target/ to 17 GB and killed the linker with SIGBUS on a +# full disk; debug = 0 shrinks the lance/datafusion-stack test binaries by +# roughly an order of magnitude and links faster. Set "line-tables-only" here +# instead if line-numbered backtraces are ever worth the size again. +[profile.dev] +debug = 0 diff --git a/crates/lance-graph/Cargo.toml b/crates/lance-graph/Cargo.toml index 8267d026..fdfe4e8c 100644 --- a/crates/lance-graph/Cargo.toml +++ b/crates/lance-graph/Cargo.toml @@ -50,8 +50,6 @@ serde_json = "1" serde_yaml = "0.9" bytes = "1" snafu = "0.8" -deltalake = { version = "0.32", features = ["datafusion", "s3", "azure", "gcs"], optional = true } -url = { version = "2", optional = true } # AdaWorldAPI/ndarray fork with 55 HPC modules (Fingerprint, CAM-PQ, CLAM, BLAS, ZeckF64) # Source: https://github.com/AdaWorldAPI/ndarray @@ -75,13 +73,19 @@ bgz-tensor = { path = "../bgz-tensor", optional = true } [features] default = ["unity-catalog", "ndarray-hpc", "planner", "bgz17-codec", "tensor-codec"] unity-catalog = ["lance-graph-catalog/unity-catalog"] -# TODO(lance-bump-delta): deltalake 0.32 removed `DeltaTableProvider::try_new`; -# new API is `TableProviderBuilder::new().with_snapshot(...).with_log_store(...).build().await`. -# Refactor `table_readers.rs::DeltaTableReader` to the new builder API in a -# follow-up PR. Dropped from `default` features for now so CI stays green on -# the lance/lancedb bump; still opt-in via `--features delta` if a caller -# accepts the broken state. -delta = ["dep:deltalake", "dep:url"] +# The `delta` feature (deltalake + DeltaTableReader) was REMOVED 2026-08-18 +# (operator pin ruling: lance 9 / lancedb 0.33 / datafusion 54.1 — NO DF 53 — +# / arrow 58). deltalake tops out at 0.32.4 pinning datafusion ^53.1.0, so the +# feature was the sole source of the DF-53 duplicate major in Cargo.lock; it +# was also already non-default AND documented broken (0.32 removed +# `DeltaTableProvider::try_new`). CORRECTION (same day, operator-pointed): +# delta-rs MAIN already pins datafusion 54 + arrow 58 (verified 2026-08-18 +# from delta-io/delta-rs main Cargo.toml) — only the crates.io releases top +# out at DF 53. Restoration is therefore available NOW via the house +# mechanism (git dep on the upstream repository, pinned to a rev) + a reader +# refactor to the current builder API — as its own deliberate PR if a +# consumer needs Delta. The removal itself stands ("we don't need +# deltalake"). # lancedb: high-level table/connection API (Python/JS SDK equivalent in Rust). # Enables LanceDB-compatible connection strings, table management, and the # vector search API surface that third-party tooling (DuckDB ext, Polars) expects. diff --git a/crates/lance-graph/src/lib.rs b/crates/lance-graph/src/lib.rs index 1c8633b9..7dfd809b 100644 --- a/crates/lance-graph/src/lib.rs +++ b/crates/lance-graph/src/lib.rs @@ -80,6 +80,4 @@ pub use lance_graph_catalog::{UnityCatalogConfig, UnityCatalogProvider}; pub use lance_vector_search::VectorSearch; pub use query::{CypherQuery, ExecutionStrategy, SqlDialect}; pub use sql_query::SqlQuery; -#[cfg(feature = "delta")] -pub use table_readers::DeltaTableReader; pub use table_readers::{default_table_readers, ParquetTableReader}; diff --git a/crates/lance-graph/src/table_readers.rs b/crates/lance-graph/src/table_readers.rs index 4ebf17b3..617c599c 100644 --- a/crates/lance-graph/src/table_readers.rs +++ b/crates/lance-graph/src/table_readers.rs @@ -4,7 +4,9 @@ //! Built-in [`TableReader`] implementations for common data formats. //! //! - [`ParquetTableReader`] — reads Parquet tables using DataFusion's built-in support. -//! - [`DeltaTableReader`] — reads Delta Lake tables (behind `delta` feature flag). +//! +//! (A Delta Lake reader existed behind the removed `delta` feature — see +//! Cargo.toml's removal note, 2026-08-18.) use std::collections::HashMap; use std::sync::Arc; @@ -57,90 +59,11 @@ impl TableReader for ParquetTableReader { } } -/// Reads Delta Lake tables using the `deltalake` crate. -/// -/// Opens the Delta table at the storage location and registers it as a -/// DataFusion `TableProvider`, enabling full SQL query support including -/// time travel, schema evolution, and partition pruning. -/// -/// Supports cloud storage (S3, Azure, GCS) via `storage_options`. -#[cfg(feature = "delta")] -pub struct DeltaTableReader; - -#[cfg(feature = "delta")] -#[async_trait] -impl TableReader for DeltaTableReader { - fn name(&self) -> &str { - "delta" - } - - fn supported_formats(&self) -> &[DataSourceFormat] { - &[DataSourceFormat::Delta] - } - - async fn register_table( - &self, - ctx: &SessionContext, - table_name: &str, - table_info: &TableInfo, - _schema: arrow_schema::SchemaRef, - storage_options: &HashMap, - ) -> CatalogResult<()> { - let location = table_info.storage_location.as_deref().ok_or_else(|| { - CatalogError::Other(format!("Table '{}' has no storage_location", table_name)) - })?; - - let table_url = url::Url::parse(location).map_err(|e| { - CatalogError::Other(format!( - "Invalid storage location URL '{}': {}", - location, e - )) - })?; - - let delta_table = if storage_options.is_empty() { - deltalake::open_table(table_url).await - } else { - deltalake::open_table_with_storage_options(table_url, storage_options.clone()).await - } - .map_err(|e| { - CatalogError::Other(format!( - "Failed to open Delta table '{}' at '{}': {}", - table_name, location, e - )) - })?; - - let snapshot = delta_table - .snapshot() - .map_err(|e| CatalogError::Other(format!("Failed to get Delta snapshot: {e}")))? - .snapshot() - .clone(); - let log_store = delta_table.log_store(); - let scan_config = Default::default(); - let provider = deltalake::delta_datafusion::DeltaTableProvider::try_new( - snapshot, - log_store, - scan_config, - ) - .map_err(|e| CatalogError::Other(format!("Failed to create DeltaTableProvider: {e}")))?; - ctx.register_table(table_name, Arc::new(provider)) - .map_err(|e| { - CatalogError::Other(format!( - "Failed to register Delta table '{}': {}", - table_name, e - )) - })?; - - Ok(()) - } -} - /// Returns the default set of table readers. /// -/// Includes Parquet support, and Delta Lake support when the `delta` feature is enabled. +/// Includes Parquet support. (Delta Lake support was removed with the `delta` +/// feature, 2026-08-18 — see Cargo.toml's note; a `DataSourceFormat::Delta` +/// table simply has no registered reader, exactly as when the feature was off.) pub fn default_table_readers() -> Vec> { - #[allow(unused_mut)] - let mut readers: Vec> = vec![Arc::new(ParquetTableReader)]; - #[cfg(feature = "delta")] - readers.push(Arc::new(DeltaTableReader)); - readers + vec![Arc::new(ParquetTableReader)] } diff --git a/docs/lotus/LOTUS-FRONTIER-AUDIT.md b/docs/lotus/LOTUS-FRONTIER-AUDIT.md index ae714e85..d23fccb0 100644 --- a/docs/lotus/LOTUS-FRONTIER-AUDIT.md +++ b/docs/lotus/LOTUS-FRONTIER-AUDIT.md @@ -152,6 +152,18 @@ OFFERS. Phase 6 (`PREPARED-ARTIFACT-PUBLICATION.md`) is **gated** on a `cargo fetch` (network + disk budget permitting) or an operator-sanctioned alternative source consult. Do not write Phase 6 from memory of lance APIs. +> **⊘ BLOCKER LIFTED 2026-08-18 (same day, operator pin ruling).** The +> operator's dependency ruling ("pinned via [patch] of the upstream +> repository git") sanctioned the upstream source consult: the EXACT v9.0.0 +> tag is on disk (`/tmp/sources/lance-9`, matching the Cargo.lock checksum) +> plus current upstream (`/tmp/sources/lance-main`). Capability findings are +> deliberately NOT recorded here — they enter through the RP-SEAL research +> program's Domain-A independent passes (`.claude/plans/ +> erasure-seals-compaction-research-v1.md` §11 independence rule) and are +> cross-checked at consolidation. The paragraph above stands as the honest +> record of what THIS session's Phase 0 could and could not verify at +> composition time. + ### §2.3 The existing "merkle" is not a tree (VERIFIED) `MerkleRoot::from_fingerprint` (`crates/lance-graph/src/graph/spo/merkle.rs:18-38`)