diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 846e1373..f56d4b57 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,84 @@ +## 2026-08-10 — E-CLAUDE-MD-KEY-DEPENDENCIES-WENT-STALE-AND-PROPAGATED-A-WRONG-PIN-INTO-A-PLAN-1 + +**Status:** FINDING `[G]` (verified against `Cargo.lock` + every crate manifest). + +`CLAUDE.md` § Key Dependencies — **the mandatory first read for every session** — +carried four stale pins, dated *"Verified against Cargo.lock 2026-06-14"*, which +pre-dates the lance-9 sweep (`b2b08b07`, 2026-08-05): + +| CLAUDE.md said | tree actually has | +|---|---| +| `lance = "=7.0.0"` | **`=9.0.0`** | +| `lance-linalg = "=7.0.0"` | **`=9.0.0`** | +| `lancedb = "=0.30.0"` | **`=0.33.0`** | +| `datafusion = "53"` | **`54`** | + +**The propagation is the finding, not the staleness.** The wrong `datafusion 53` +travelled: stale `CLAUDE.md` → restated in-session → written into +`weather-substrate-poc-v2.md` §6 (PR #915, merged) → *and* mischaracterized there, +because `datafusion 54.1.0` in `Cargo.lock` was filed as suspicious drift when 54 is +in fact our correct, deliberate, **MEASURED** direct pin +(`.claude/plans/lance9-datafusion54-upgrade-probe-v1.md`, 2026-08-05). **A POC +session trusting either document would have pinned lance 7 against a lance-9 tree +and failed to build.** + +**⊘ Self-correction, same hour (operator-caught).** The first version of THIS entry +then called `datafusion 53.1.0` a *"residual transitive."* Wrong, and more dangerous +than the original error: it frames a required dependency as cruft, inviting a future +session to collapse `Cargo.lock` to a single major and silently break the `delta` +feature. **Both majors are REQUIRED and the dual state is documented upstream:** +`deltalake-core 0.32.4` pins `datafusion 53.1.0` (+ `datafusion-datasource`, +`datafusion-physical-expr-adapter` at 53.1.0); the lance family pins 54. Cargo +permits the coexistence precisely because they are different semver majors. It +resolves only when deltalake moves to DF 54 — not by any action here. + +**The meta-lesson:** "this version looks unexpected" has three possible causes — +*stale doc*, *real drift*, and **legitimate multi-major coexistence**. I reached for +the first two and skipped the third, twice in opposite directions. Before labelling +a version anomalous, read *who requires it*; a lockfile entry with a live requirer +is a constraint, never a leftover. + +The lockstep discipline itself was never violated: every manifest carries exact +`=9.0.0` / `=0.33.0`. **Only the docs lagged** — which is the more dangerous +failure, because the code compiles and the doc is what a new session reads first. + +**Rule:** a version block asserting *"verified against Cargo.lock "* is a +claim with an expiry. When a dependency sweep lands, the sweep's PR must update +every doc that restates its pins — the same-commit board-hygiene rule, applied to +version facts. Grep for the old version string across `.claude/` and `CLAUDE.md` +before closing a bump. + +Corrected in this PR: `CLAUDE.md` (all four, + `lance-index`, + a `rust` line +pointing at `rust-toolchain.toml` as authoritative), the plan's §6 with a dated +`⊘ CORRECTION` block (append-only, not a silent edit). + +## 2026-08-10 — E-A-COMMENT-THAT-RESTATES-A-PINNED-VALUE-GOES-STALE-EVERY-BUMP-1 + +**Status:** FINDING `[G]` (two recurrences in the same file, both in git history). + +`rust-toolchain.toml`'s comment restated the pinned version in prose. It has +therefore been **wrong twice**: + +1. said `1.94.1` after a bump → fixed by `10f87fb6` + (*"docs(toolchain): fix stale 1.94.1 comment"*); +2. said `1.95.0` after the bump to `1.97.1` (`b2b08b07`) — **the identical failure, + ~3 months later**, because the earlier fix corrected the *value* and left the + *structure* that guarantees the value goes stale. + +A bump edits `channel = "…"` and nobody re-reads the paragraph below it. The +one-off correction resets the clock; it does not stop it. + +**Structural fix (this PR):** the comment **no longer restates the version at all** +— it points at the `channel` line as authoritative — and carries an **append-only +bump log** instead, one line per bump with its commit/PR and reason. Appending +cannot contradict; re-narrating always can. + +**Generalizes beyond this file:** any prose that duplicates a machine-readable +value has a half-life. Either derive it, or make the duplicate append-only. (Same +shape as `E-CLAUDE-MD-KEY-DEPENDENCIES-WENT-STALE-…-1` above, found the same hour — +one is a config comment, the other a doc table, both duplicated a pin and both went +wrong.) + ## 2026-08-10 — E-JC-AND-NDARRAY-BOTH-SHIP-A-RELIABILITY-BATTERY-WITH-DIFFERENT-DEGENERATE-CONTRACTS-1 **Status:** FINDING `[G]` (source-verified 2026-08-10, both files read). diff --git a/.claude/plans/weather-substrate-poc-v2.md b/.claude/plans/weather-substrate-poc-v2.md index 22a88ab9..9f33994a 100644 --- a/.claude/plans/weather-substrate-poc-v2.md +++ b/.claude/plans/weather-substrate-poc-v2.md @@ -271,19 +271,40 @@ session needs them added. ``` rust 1.97.1 (rust-toolchain.toml) -lance 9.0.0 lance-encoding 9.0.0 · lance-linalg 9.0.0 -lancedb 0.33.0 +lance =9.0.0 lance-encoding 9.0.0 · lance-linalg =9.0.0 · lance-index =9.0.0 +lancedb =0.33.0 arrow 58.3.0 -datafusion 53 +datafusion 54 ← NOT 53; see the correction below ``` -The lance family moves in **exact lockstep**; a bump is one deliberate PR, never a -drift. Two drift items recorded during verification, neither blocking: +The lance family moves in **exact lockstep** (`=X.Y.Z` in every manifest, verified); +a bump is one deliberate PR, never a drift. -1. `Cargo.lock` resolves **datafusion 53.1.0 *and* 54.1.0** — plausibly a transitive - pulling 54, but the lockstep discipline exists to prevent exactly this shape. -2. `rust-toolchain.toml`'s channel is `1.97.1` while its own comment still says - *"Pinned to 1.95.0"* — the comment did not follow the bump. +> **⊘ CORRECTION (2026-08-10, same day, before any POC work started; corrected +> twice).** This section's first version said **`datafusion 53`** and filed the +> presence of 54.1.0 in `Cargo.lock` as suspicious drift. **That was backwards.** +> `datafusion = "54"` is our direct pin in every crate manifest, and `lance` / +> `lancedb` / `lance-index` / `lance-datafusion` all require 54; the move is +> recorded and **MEASURED** (`lance9-datafusion54-upgrade-probe-v1.md`, 2026-08-05). +> +> **Second correction (operator):** the first fix then called 53 a *"residual +> transitive"* — also wrong, and more dangerous, because it invites someone to +> collapse the lock to one version. **Both majors are REQUIRED.** +> `deltalake-core 0.32.4` pins `datafusion 53.1.0` (+ `-datasource`, +> `-physical-expr-adapter`) upstream, backing the optional `delta` feature. Two +> semver majors coexisting is the *correct, documented* state — not a defect to +> tidy. It lifts only when deltalake moves to DF 54. +> +> **Root cause, worth more than the fact:** `CLAUDE.md`'s Key Dependencies block — +> the mandatory first read for every session — was itself stale (`lance = "=7.0.0"`, +> `lancedb = "=0.30.0"`, `datafusion = "53"`, dated 2026-06-14, pre-dating the +> lance-9 sweep). The wrong pin propagated *from* that block *into* this plan. A POC +> session that trusted either would have pinned lance 7 against a lance-9 tree and +> failed to build. Both are corrected in the same PR as this note. + +The `rust-toolchain.toml` comment drift noted in the first version is also fixed +there — **structurally**: the comment no longer restates the channel value (which is +what made it go stale twice), and carries an append-only bump log instead. --- diff --git a/CLAUDE.md b/CLAUDE.md index 77241b3a..f4ca8263 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1150,12 +1150,34 @@ cd crates/lance-graph-python && maturin develop ## Key Dependencies ```toml -# Verified against Cargo.lock 2026-06-14. The lance family moves in lockstep at =7.0.0 (PR #445). -arrow = "58" -datafusion = "53" -lance = "=7.0.0" # exact-pinned: lancedb 0.30.0 transitively requires lance =7.0.0 -lance-linalg = "=7.0.0" -lancedb = "=0.30.0" # 0.30 → lance 7 → object_store 0.13.2 (0.29 → lance 6 → object_store 0.12) +# Verified against Cargo.lock + every crate manifest 2026-08-10. The lance family +# moves in EXACT lockstep — currently =9.0.0 (lance 7 was the 2026-06-14 state, +# superseded by the lance-9 sweep, b2b08b07 / PR #896 arc). +arrow = "58" # 58.3.0 resolved; unmoved by the lance-9 sweep +datafusion = "54" # OUR direct pin, in every crate that DEPENDS on it + # (lance-graph, -catalog, -callcenter, -python, holograph), + # and the sub-crates move with it: datafusion-common / -expr + # / -sql / -functions-aggregate are all "54". lance / + # lancedb / lance-index / lance-datafusion require 54 too. + # (Grep caveat: `lance-graph-planner/Cargo.toml` has + # `datafusion = []` under [features] — a FEATURE NAME, not a + # version pin. It is not a counter-example.) + # + # ⚠ BOTH MAJORS ARE REQUIRED — do NOT "fix" Cargo.lock to one. + # `datafusion 53.1.0` (+ -datasource, -physical-expr-adapter) + # also resolves because `deltalake-core 0.32.4` pins DF 53 + # upstream, and it backs the optional `delta` feature. Two + # semver majors legitimately coexist; collapsing them breaks + # `delta`. 53 is NOT a stale leftover — it is a live upstream + # constraint that lifts only when deltalake moves to DF 54. + # Probe: .claude/plans/lance9-datafusion54-upgrade-probe-v1.md +lance = "=9.0.0" # exact-pinned: lancedb 0.33.0 requires lance =9.0.0 +lance-linalg = "=9.0.0" +lance-index = "=9.0.0" +lancedb = "=0.33.0" # the lance-9 pairing. NOTE: lancedb 0.36 is the PyPI + # package, versioned independently — the Rust crate + # tops out at 0.33.0 (probe §1). +rust = "1.97.1" # rust-toolchain.toml is authoritative; see its bump log ndarray = { path = "../../../ndarray" } # AdaWorldAPI fork, default, optional fallback nom = "7.1" snafu = "0.8" diff --git a/rust-toolchain.toml b/rust-toolchain.toml index a276a290..104b51c7 100644 --- a/rust-toolchain.toml +++ b/rust-toolchain.toml @@ -1,19 +1,31 @@ [toolchain] channel = "1.97.1" -# Pinned to 1.95.0 (bumped from 1.94.1 in PR #367, 2026-05-13). Aligns -# with bevy (edition 2024 → 1.95 MSRV) and ndarray (bumped in parallel -# per LATEST_STATE 2026-05-13). 1.95 added clippy lints -# (`clippy::manual_checked_ops`, `clippy::unnecessary_sort_by`, -# `clippy::len_zero` cleanups) that PR #367 + #368 closed across the -# workspace. Never auto-track `stable` — bump explicitly when a future -# version is reviewed and workspace clippy passes clean. +# The pinned version is the `channel` line ABOVE — deliberately not restated +# in this comment. Restating it is what made this comment wrong twice: it said +# "1.94.1" until `10f87fb6` corrected it, then said "1.95.0" after the bump to +# 1.97.1, because a bump edits `channel` and leaves the prose behind. Keep the +# rationale here and the version there; append to the log, never re-narrate. +# +# Bump log (append one line per bump): +# 1.94.0 → 1.94.1 matches sibling repos (f638fa27) +# 1.94.1 → 1.95.0 bevy edition-2024 MSRV + ndarray in parallel (PR #367, +# 2026-05-13); #367 + #368 closed the lints 1.95 added +# (`manual_checked_ops`, `unnecessary_sort_by`, `len_zero`) +# 1.95.0 → 1.97.1 lance 9 / lancedb 0.33 / DataFusion 54 sweep (b2b08b07, +# 2026-08-05); toolchain leg measured ALONE on the old pins +# first, so a toolchain failure could not be mistaken for a +# dependency failure — +# `.claude/plans/lance9-datafusion54-upgrade-probe-v1.md` +# +# Never auto-track `stable` — bump explicitly when a future version is +# reviewed and workspace clippy passes clean. # # rustfmt + clippy are mandatory CI components and must be present # whenever the pinned channel is installed. Without this, `cargo fmt` # under `actions-rust-lang/setup-rust-toolchain@v1` resolves to the -# pinned `1.94.x` channel but rustfmt is only installed for `stable` +# pinned channel but rustfmt is only installed for `stable` # (the action's default), producing: -# error: 'cargo-fmt' is not installed for the toolchain '1.94.x-...' +# error: 'cargo-fmt' is not installed for the toolchain '-...' # Declaring components here makes rustup install them as part of the # toolchain itself, which is also developer-friendly: a fresh # `rustup show` in this workspace materializes a complete CI-equivalent