docs: DATAFUSION-PERIMETER.md — what a DataFusion-free build would actually cost - #870
Conversation
Measured, not estimated. Written because the decisive fact is invisible to grep and would otherwise be re-derived. The headline kills the simple plan: `lance = 7.0.0` reaches DataFusion through three independent paths of its own (lance-datafusion, lance-geo/geodatafusion, lance-index). Feature-gating lance-graph's five DIRECT deps would remove ZERO crates while lance is present -- a flag that looks like a win and delivers nothing. Corollary: Lance storage and a Pi-sized binary are mutually exclusive today. Closures: lance-graph 598, cognitive 70, planner 47, contract 1. The Pi-viable substrate (contract + planner + cognitive + ndarray, NEON already there) has no DataFusion edge at all; its cost is Lance persistence, and whether lance-core / lance-file work without lance-datafusion is the open question that decides versioned time-travel vs memory-only state. Also recorded: error.rs does NOT cascade (3 lines, both cfg-able) which was the feasibility gate; the module-gate probe measured 5 errors = 4 real sites + 1 cascade; the public surface is three names; ten of seventeen children have zero pub items; vector_ops and ScanStrategy are the two needing a decision rather than a cfg; the batch writer imports neither DataFusion nor Lance (ACID/WAL is Lance's); RaBitQ is in thinking-engine; cam_pq is V1 6-byte, NOT the V3 6x(u8:u8) carving despite sharing the number 6. Plus the fossil -- planner's `[features] datafusion = []` is an empty feature, not a dep -- and why it stayed empty: datafusion_planner was hired as the orchestrator for 4096 command shapes before V3 answered that with classid -> ClassView, while the planner grew the replacement in parallel. Section 9 lists the four proxies that lied while producing this, including the one that matters: for dependency questions use cargo tree, not grep. The lance -> lance-datafusion edge appears in no grep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a measurement-driven document describing DataFusion’s dependency closure, source coupling, module-gating feasibility, excluded areas, analysis methodology, and remaining design questions. ChangesDataFusion perimeter analysis
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_b2342599-d9a3-4a71-a78e-a10458a1f9e0) |
|
@codex review Docs-only, but the claims are load-bearing for a product decision (can this substrate build on a Raspberry Pi), so please treat it as a factual audit rather than a prose review. Four specific asks, roughly in order of what would change the conclusion:
Standing invitation on §9: it lists four proxies that produced wrong answers while I wrote this (a Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a50299c1a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| durable storage*. **Open question, not yet measured:** can `lance-core` / | ||
| `lance-file` be used without `lance-datafusion`? That single answer decides | ||
| whether a Pi build gets real time-travel or memory-only state. |
There was a problem hiding this comment.
This is not actually decided by lance-core/lance-file: in Lance 7.0.0, lance-core's DataFusion dependencies are optional and lance-file has no lance-datafusion dependency (only datafusion-common), while the dataset-versioning primitives—Manifest, Transaction, and CommitHandler—live in lance-table, which likewise has no lance-datafusion edge. The real product question is whether the Pi substrate can build on lance-table or must recreate the higher-level Dataset conveniences; presenting the current decision as versioned storage versus memory-only state can therefore lead to the wrong architecture choice.
Useful? React with 👍 / 👎.
|
|
||
| ## 2. Dependency closures (measured) | ||
|
|
||
| `cargo tree -e normal --prefix none | sort -u | wc -l`: |
There was a problem hiding this comment.
This pipeline does not count unique crates: default cargo tree de-duplicates shared dependencies by printing later occurrences with a (*) suffix (cargo tree --help describes --no-dedupe as the option that “repeats all shared dependencies”), so sort -u treats foo v1 and foo v1 (*) as two different entries. Because these graphs contain many shared dependencies, the reported 598/70/47 totals are inflated rendered-line counts rather than transitive-crate counts; strip the marker or derive the closure from Cargo metadata before using these measurements for the Pi comparison.
Useful? React with 👍 / 👎.
| Compile *time* is the smaller half of the argument. Compiling DataFusion is | ||
| memory-hungry; a 4 GB Pi is more likely to thrash or OOM than to merely take | ||
| 27 minutes. A 47-crate closure is a Pi Zero 2W conversation, not a Pi 5 one. |
There was a problem hiding this comment.
When the target scenario is compiling natively on a Pi, the 47-crate figure is not the compilation closure because -e normal explicitly excludes Cargo's separate build edge kind. For example, lance-graph-contract is reported as one crate here even though its build script depends on serde_yaml, serde, and glob plus their transitive and proc-macro dependencies; those still compile on a native Pi and also enter the planner/cognitive builds through the contract. Use -e normal,build (with proper package deduplication) before drawing the Pi Zero versus Pi 5 conclusion.
Useful? React with 👍 / 👎.
|
|
||
| | files naming it | type | | ||
| |---|---| | ||
| | 6 | `ScanStrategy` | |
There was a problem hiding this comment.
The six files counted here belong to lance-graph-planner and use crate::ir::logical_op::ScanStrategy, which is a different type from lance_graph::datafusion_planner::cost_estimation::ScanStrategy. The DataFusion planner's enum is only consumed internally by predicate_pushdown.rs, so the claimed three-name boundary and the subsequent proposal to move this ScanStrategy into contract are based on a same-identifier grep rather than a resolved call-site census.
Useful? React with 👍 / 👎.
Codex reviewed the first edition (a50299c, merged as #870) and found four defects. All four verified; two changed what the document concludes. Superseded text is retained in place and labelled, never silently overwritten. 1. P1 -- the corollary "Lance storage and a Pi-sized binary are mutually exclusive" was WRONG. It equated a capability with a crate. Measured in ISOLATED scratch crates (a cargo tree -p inside the workspace resolves under feature unification and is not comparable): lance = 7.0.0 -> 571 crates, 32 datafusion lance-table = 7.0.0 -> 280 crates, 1 datafusion The one is datafusion-common, non-optional, via lance-file; lance-core's datafusion deps are behind an optional feature and lance-table's manifest names DataFusion nowhere. And lance-table is where the versioning lives -- Manifest, Transaction, CommitHandler. So the trade is not "versioned storage vs memory-only" (the first edition's top open question); it is whether the Dataset conveniences this repo actually uses can be re-created on lance-table. New section 2a. 2. P1 -- ScanStrategy was never on the datafusion_planner public surface. The 6 files were crate::ir::logical_op::ScanStrategy in the planner, a different type in a different crate; the DataFusion one is used in 2 files, both INSIDE the module. Corrected membership: DataFusionPlanner, GraphPhysicalPlanner, vector_ops -- four resolved imports, not a name grep. The "three names" headline survives; its membership does not, and the proposal to move ScanStrategy into contract is withdrawn as a change whose reason did not exist. 3. P2 -- -e normal excludes build edges. Added a column: contract goes 1 -> 16 (serde_yaml, serde, glob + transitives). "Zero deps" stays true of the linked artifact and was never true of the build. 4. P2 -- the printed recipe did not reproduce its own numbers. cargo tree marks repeat subtrees with (*), so foo and "foo (*)" survive sort -u; the printed pipeline yields 828 for lance-graph, not 598. Codex inferred the counts were inflated -- they were not, cross-checked two ways. The counts were right and the recipe was wrong, which is the worse of the two: a wrong number gets challenged, an unreproducible recipe gets trusted. Also finishes section 10 Q1: the lance call-site census is done and short -- open + append + list-versions, one public signature leaking &lance::Dataset (lance_vector_search.rs:222, an API break not a refactor), and one genuinely query-side use. Gated on an unmeasured round-trip: whether hand-rolled open/append over lance-table reads lance-written datasets byte-identically. Section 9 gains a round two, because all four defects have one shape: a proxy that is cheap to compute gets published as the quantity it resembles. A grep resembles a census, a facade crate resembles a capability, an in-workspace tree resembles a closure. Section 9 already named the same-identifier-grep failure -- and section 5 was written with one four sections later, so the defence has to be mechanical, not vigilance. Board hygiene (EPIPHANIES E-A-CHEAP-PROXY-GETS-PUBLISHED-AS-THE-QUANTITY-IT- RESEMBLES-1) follows in a second commit: it is a large append-only file and re-transmitting it whole through this API risks corrupting it, so it waits for git rather than being rewritten wholesale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
…on finding E-A-CHEAP-PROXY-GETS-PUBLISHED-AS-THE-QUANTITY-IT-RESEMBLES-1, owed since #871 (merged via GitHub API push while the Bash permission classifier was intermittently unavailable in-session -- a 14,447-line append-only file was not safe to re-transmit whole through that path, so it waited for git). Status updated to SHIPPED: both #870 (first edition) and #871 (correction pass) are on main now. Added the process paragraph the entry was missing -- this is the THIRD occurrence this session of a merge outrunning its review (#866 review-in-flight, #868 merged while DO-NOT-MERGE, #870 merged mid- correction-pass), and a fourth near-miss (#871 itself merged before CodeRabbit's rate-limited review could run). Sharpens E-A-GUARANTEE-WITH-A-RE-EXPORTED-BYPASS-IS-NOT-A-GUARANTEE-1's rule: the reviewable window is "until the operator reads it," which can be zero, so correctness has to be front-loaded into the PR rather than caught after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
Docs-only.
docs/DATAFUSION-PERIMETER.md, rebased onmain(ceca232).Motivation (operator): a substrate small enough for a Raspberry Pi — home automation, toy robots, anything wanting a breath of AGI without a data-warehouse build. Compiling DataFusion is outside that budget. This answers what it would cost, with measurements rather than estimates.
The finding that governs everything
lance = 7.0.0reaches DataFusion through three independent paths of its own —lance-datafusion,lance-geo → geodatafusion,lance-index → lance-datafusion. So feature-gating lance-graph's five direct deps would remove zero crates whilelanceis present: a flag that looks like a win and delivers nothing.Corollary: Lance storage and a Pi-sized binary are mutually exclusive today.
This is invisible to every grep — it appears only in the resolved dependency graph. It also inverts a conclusion I'd drawn earlier in the session from a module-gating probe, which is exactly why it's written down.
Closures
lance-graphlance-graph-cognitivelance-graph-plannerlance-graph-contractThe Pi-viable substrate is
contract+planner+cognitive+ndarray— reasoning seam, 16 strategies, MUL, thinking, SoA/kanban, NEON already present. No DataFusion edge anywhere in it. Its cost is Lance persistence.Also measured
error.rsdoes not cascade — 3 lines, both#[cfg]-able, no exhaustive matches. This was the feasibility gate and it came out on the afternoon side rather than the wave side.lib.rs'spub use query::{…}. Probe reverted; tree unchanged.ScanStrategy(6 files),GraphPhysicalPlanner(2),DataFusionPlanner(1), for a ~6K-LOC subsystem.pubitems — the wholebuilder/tree, scan and join ops. The relational machinery is unreachable from outside.vector_ops(15 pub items) andScanStrategyare the two pieces needing a decision rather than a#[cfg].thinking-engine;cam_pqis V1 6-byte, not the V36×(u8:u8)carving despite both involving the number 6.lance-graph-planner's[features] datafusion = []is an empty feature, not a dependency — a reserved slot for something that never arrived, because the planner grew the replacement in parallel.datafusion_plannerwas hired as the orchestrator for the 40960xFFFFFFcommand shapes before V3 answered that withclassid → ClassView. It now serves neither that role nor upstream's traversal-join one.§9 — proxies that lied
Four of them, recorded because they cost real time today: a single-line
Cargo.tomlgrep that read a[features]entry as a dependency; agrep -cthat can't tell sections apart; agrep -lwon module names wheremodscored 831 andhelpers58; and a compiler error count read as independent sites when one was a cascade.Rule extracted: for dependency questions use
cargo tree, notgrep.Open questions, in decision order
lance-core/lance-filebe used withoutlance-datafusion? — decides versioned storage vs memory-only on Pi.--no-default-features --all-targets).ScanStrategy— contract-shaped or DataFusion-shaped?vector_ops— substrate side or query side? (Note: the CAM-PQ it serves is V1.)No code change.
🤖 Generated with Claude Code
https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
Generated by Claude Code
Summary by CodeRabbit