From d950b4ae705560ca1e0a6791d843b1b377812852 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 6 Aug 2026 09:37:52 +0000 Subject: [PATCH 1/2] Document object-store hydration doctrine and plan idle-flush eviction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An object store is not a runtime store, and the reason is architectural rather than about speed: Lance opens a network-scheme URI natively, so the wrong topology *runs* and only degrades — while deleting the mmap layer underneath it. A remote read lands in a freshly allocated buffer, one copy per read, with no page cache, so every zero-copy guarantee below that point becomes a claim about bytes that were copied into existence. Splitting the three jobs (object store hydrates / local directory IS the store / the volume only decides whether hydration repeats) makes the correctness requirement visible: a local directory, of any kind. The volume is an optimization on hydration frequency and must not be stated as a dependency. The feature-gate half is written down because it cost a session an hour. `lancedb` ships `default = []` while `lance-io` carries `aws` in its own defaults, so "the Lance stack does object storage by default" is true one layer down and false at the layer we depend on — the diagnosis goes wrong because the mental model is correct about the wrong crate. The resulting error names a *scheme*, before any credential is read, so no amount of endpoint or region debugging can help. That gives a mechanical rule worth keeping: a scheme-named error is a build problem, a credential-named error is a config problem. `docs/DATAFUSION-PERIMETER.md` gets a cross-reference because that document already catalogues this exact shape one crate over. The plan builds on the same lifecycle but is deliberately not an implementation. It records the two things a future reader would otherwise re-derive or get wrong: that the purpose is cost smoothing rather than capacity (so the budget is soft and never fails an operation), and that a lease/refcount protocol was considered and rejected as disproportionate at a three-day idle floor — with the revisit condition named, so it is not re-added later on the assumption it was overlooked. The race therefore owes a "does not corrupt" test rather than an impossibility proof, since the latter would be exactly the code-implied assertion the falsifiability rule forbids. Acceptance criteria are fire/silence pairs; because the trigger is a conjunction, the silence half splits in two, and the sharper of them is what a staleness-only policy would fail. Docs only. No crate, type, feature or test changed. Board hygiene in the same commit per CLAUDE.md: EPIPHANIES x2 (prepend), INTEGRATION_PLANS x1 (prepend), LATEST_STATE (prepend). Co-Authored-By: Claude Opus 5 --- .claude/board/EPIPHANIES.md | 38 ++ .claude/board/INTEGRATION_PLANS.md | 31 ++ .claude/board/LATEST_STATE.md | 9 + .claude/knowledge/s3-hydration-lifecycle.md | 206 +++++++++++ .../plans/idle-flush-dataset-eviction-v1.md | 331 ++++++++++++++++++ docs/DATAFUSION-PERIMETER.md | 25 ++ 6 files changed, 640 insertions(+) create mode 100644 .claude/knowledge/s3-hydration-lifecycle.md create mode 100644 .claude/plans/idle-flush-dataset-eviction-v1.md diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 1ee3977f3..12ebbfb26 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,41 @@ +## 2026-08-06 — E-OBJECT-STORE-HYDRATES-IT-DOES-NOT-STORE-1 — the object store is the hydration path; the local filesystem is the store; the volume only decides whether hydration repeats + +**Status:** FINDING (mechanism) + one reported single-observation measurement set. **Confidence:** High for the layer split and the feature-gate facts (manifest-verified in session); Medium for the endpoint numbers (one provider, one region, one point in time, not re-run); the flush/rehydrate lifecycle is labelled CONJECTURE in the doc with its falsifier stated. Documentation-only — no Rust changed. + +**The three layers, one job each.** Object store = hydration source (durable, versioned, shared; absent → fall back to whatever secondary source the consumer has). Local directory = **THE** store — zero-copy mmap reads, page cache, no network in the read path; **no fallback, always required**, and **any** local path satisfies it. Persistent volume = decides *which* local directory, chosen only because it survives redeploys; absent → hydrate every boot, still correct, merely slower. **The volume is an optimization on hydration frequency, not a component of the store** — a design that says "we need a volume or this doesn't work" has mis-assigned a job: what it needs is a directory. + +**Lance opens a network-scheme URI natively, and that is exactly the trap** — the wrong architecture *runs*, correctly, and only degrades. A local read is a mapped page a lens can borrow with a cast; a remote read is a range request landing in a freshly allocated buffer — one copy per read minimum, no page cache. So mounting the object store as the runtime store **deletes the mmap layer from the architecture**, and every zero-copy guarantee below it becomes a claim about buffers that were copied into existence (`zero-copy-lens-law.md`, one layer down). Review question: *where does the process open its dataset from?* A network scheme voids the zero-copy story regardless of type signatures. + +**The feature gate that costs an hour (manifest-verified this session, two releases apart).** `lancedb` ships `default = []`; its `aws` feature forwards to `lance/aws` + `lance-io/aws` (+ `object_store/aws`), and *that* is what registers the provider. `lance-io` DOES carry `aws` in its own defaults — so "the Lance stack supports object storage by default" is **true one layer down and false at the layer we depend on**; `lancedb` is the layer that opts out, which is why the diagnosis goes wrong (the mental model is correct about the wrong crate). Without it the URI fails at provider lookup **by scheme**, before any credential/endpoint/region is read — no amount of endpoint or quoting debugging can help, because that code is not in the binary. **Mechanical rule: an error naming a *scheme* is a BUILD problem; an error naming a *credential/host/bucket/region/signature* is a CONFIG problem. Read the error's noun before touching an env var.** + +**Ratios, not numbers, are the finding** (single observation, provider- and region-dependent, deliberately unnamed): a small-object round trip ≈ **2.5 million×** slower than RAM and **~2500×** slower than NVMe ⇒ **NOT viable as swap, NOT viable as a page-fault backing store** (which is the mmap argument restated in numbers — mounting it as the store puts that latency under every read); a cold re-open + full count of a tens-of-MB dataset is boot-viable, while writing the same dataset costs ~5× that and carries object-per-fragment overhead on top of raw throughput ⇒ **writes are an ops step, never a boot path**. One sentence generalizes every verdict: **the object store is fine when the object count is small and the objects are large; unusable when the access count is large and the accesses are small.** + +**Lifecycle (CONJECTURE, falsifier stated in the doc):** absent → hydrated → dirty → flushed → rehydrate. The single load-bearing rule: **flush is legal only from `hydrated`, never from `dirty`** — the `dirty → flushed` edge is data loss with **no error**, so it must be an asserted condition rather than an assumption. + +Doc: `.claude/knowledge/s3-hydration-lifecycle.md` (READ BY header + per-claim evidence table). Cross-ref section: `docs/DATAFUSION-PERIMETER.md` §9a — the same "capability behind a default-off feature, diagnosed at the wrong layer" shape that document's §9 already catalogues for a different dependency. Follow-on plan (PROPOSAL, unimplemented): `.claude/plans/idle-flush-dataset-eviction-v1.md`. + +## 2026-08-06 — E-IDLE-FLUSH-IS-COST-SMOOTHING-NOT-CAPACITY-AND-THE-3-DAY-FLOOR-PRICES-OUT-A-LEASE-PROTOCOL-1 — plan v1 for a feature-gated idle-flush: operator-set policy defaults, and the guards that must each fire AND stay silent + +**Status:** PROPOSAL — **nothing implemented, nothing measured.** The policy defaults and the scope ruling on the race are **OPERATOR-SET**, not findings; the rest is CONJECTURE. **Confidence:** Medium for the design shape (argued from the named failure modes of each rejected alternative); the dirty-detector carries an explicit unclosed verification gate. Documentation-only. + +**The purpose is COST SMOOTHING, not capacity** (operator framing). This is **not** a mechanism for fitting a working set into a disk that is too small, and the plan explicitly does not justify itself with "otherwise you run out of disk." Local disk is billed **continuously for capacity provisioned**; object storage is billed for **what is kept** — so a dataset touched once and retained forever pays a standing charge for a one-time access, and eviction flattens that curve into one transfer plus storage-at-rest. The binding consequence: **never fail an operation to hold the number** — a smoothing mechanism that breaks a workload has traded a billing improvement for an outage. + +**The default policy (OPERATOR-SET, heuristic, both config): TWO conditions, both required.** Age — idle **> 3 days** ⇒ a *candidate*. Budget — total local footprint **> ~300 MB** ⇒ eviction *engages at all*. The trigger is **pressure-driven and age-ordered**: under budget **nothing is ever evicted no matter how stale** (a small deployment pays nothing, not even churn); over budget the **stalest go first** until back under. **Q1** resolves to age measured from **last USE (read or write)** — either touch is working-set evidence; dirtiness stays the separate axis governing whether flushing is *legal*, never whether it is *desirable*. **Q2** resolves to the watermark over both bare alternatives (a bare timer works on datasets nobody is asking about and pays when disk is abundant; a bare allocation-failure signal acts only **inside a request**, charging eviction *and* rehydration to a waiting caller). A `bytes × idle_seconds` ranking is **deferred, not rejected** — it would reach the budget in fewer evictions but preferentially evicts what is most expensive to get back, since rehydration cost is *also* size-proportional; revisit only with measured access-pattern data. + +**"Soft spot" is load-bearing.** ~300 MB is where pressure **BEGINS**, never a hard cap: no operation may fail because the budget is exceeded (no admission control, no back-pressure), an in-use dataset larger than the whole budget **stays resident** (**correctness beats the watermark, always**), and the sweep may therefore **fail to reach the target** — a legitimate steady state, not an error. Which forces the observability requirement: **a deployment silently over budget must not look identical to one under it**, so a no-target sweep reports footprint-vs-budget AND distinguishes *"no candidate was old enough"* from *"every candidate was in use"* — different situations, different responses, must not collapse into one line. **Over-budget-but-nothing-stale is INTENDED** (the age floor protecting a hot working set from being thrashed by budget pressure); if that line persists the working set genuinely exceeds the budget — a **capacity finding**, not a policy failure. + +**Q3 — dirty detection is the Lance dataset VERSION, never a hash** (hashing tens of MB to answer a boolean is the wrong cost class and scales with the thing being avoided): record `version_at_hydration`, dirty ⇔ current ≠ recorded — an integer compare against a generation counter the storage layer already maintains, correct across append and compaction alike. mtime is a cheap pre-check but **never the authority** (unreliable across filesystems/maintenance); on disagreement version wins **and the disagreement is logged**. **VERIFICATION GATE, explicitly unclosed:** that a cheap local version read exists has **not** been checked against the API — if it fails this is a BLOCKER needing a different detector, and the honest response is to say so, not substitute a heuristic. + +**Q4 — SCOPE REDUCED by operator correction; a lease/refcount protocol was CONSIDERED AND REJECTED.** *"Given 3 days not used it's just flattening the payment curve and hardly attributing to race conditions."* An earlier draft called the flush/read race "the part most likely to be subtly wrong" and specified a refcounted guard type — **walked back as disproportionate.** A dataset untouched for three days is not plausibly under an active mmap at the instant the sweeper picks it, so the design must be **safe if it happens, not engineered around the possibility**; the protocol is priced for a hot cache and this is not one (it would put a guard in every read signature, an atomic state-machine step, and a second gate-off code path onto **every** read to close a window the age floor already makes negligible). **Instead: cheap check-then-act** — skip the flush if a read is in flight (a non-authoritative check suffices, because losing the race is not harmful); if a read begins mid-flush, **let it rehydrate rather than block** (the `absent → hydrated` path is idempotent and already the recovery mechanism). **The bar is that the failure mode is RECOVERABLE, never CORRUPTING** — worst case a wasted rehydration (~1.4 s at the reported scale), never a torn read or bytes removed from under a live mapping. **The rejection is recorded rather than left silent so a future reader does not re-add it believing it was overlooked. Revisit condition: if the threshold ever drops from DAYS to HOURS, the calculus changes and the protocol question reopens.** + +**Q5 — the thrash falsifier is the gating acceptance criterion**, because a thrashing policy makes the system strictly worse than not having it *while appearing to function*: over a window, `rehydrations / distinct_datasets_accessed > 1.0` means a dataset was evicted and re-fetched **within its own working set**; a second hydration of the same dataset inside one `T_min` window is the sharper single-dataset signal. Supporting: `total_hydration_seconds / total_read_seconds` (the amortization ratio). **Both must be instrumented before the policy is enabled anywhere, or a thrashing deployment is indistinguishable from a working one.** + +**Q6 — off by default, and "nothing" is literal, not "a cheap timer":** gate-off removes the sweep task and its wakeups, the size/last-read accounting, the key computation, the watermark checks, and the push-back path. **The one place a gate can silently change semantics is the lease guard** (`I-LEGACY-API-FEATURE-GATED` shape) — either keep it present-but-inert (one code path) or compile it out and **owe a proof of equivalence**; the same function name must not mean different things under different gate states. Orthogonality: idle-flush **requires** the object-store feature but does not **imply** it — enabling it without the provider must be a build-time or startup-time refusal, never a runtime surprise on the first eviction (the scheme-error trap one layer up). + +**Per the P0 falsifiability rule the plan enumerates the acceptance tests as fire/silence PAIRS** — an eviction policy that never evicts and one that evicts everything are equally useless and both pass a naive assertion. Because the trigger is a **conjunction**, the silence half splits: **under budget + a very stale dataset ⇒ nothing evicted** (the sharpest test — *a policy that evicted on staleness alone would pass the can-fire test and still be wrong*), and **over budget + everything fresh ⇒ nothing evicted, and the sweep says so**. Plus: age-ordering is load-bearing (stalest first among candidates), age-floor and budget **inertness** (raising each must silence what lowering admits), the budget is **soft** (an in-use over-budget dataset stays resident and no operation fails), the dirty detector discriminates, the in-flight check both skips **and then releases**, and the thrash detector itself both fires and stays silent. **The race test is deliberately shaped as a corruption test, not an impossibility proof** — force the interleaving and assert the reader gets a correct complete dataset via rehydration; asserting the race "cannot happen" would be exactly the vacuous, code-implied assertion the P0 rule forbids. The two policy numbers are **operator-set defaults, not derived** — the access-pattern distribution that would justify them has not been measured, which is precisely why they are config. + +Plan: `.claude/plans/idle-flush-dataset-eviction-v1.md` (five open items listed there, incl. multi-process access and whether a sweep may *initiate* push-back — currently assumed **skip**, the conservative and possibly wrong choice). + ## E-A-FEATURE-CONDITIONAL-CLASSID-SILENTLY-EMPTIED-A-FIXTURE-1 (2026-08-05, measured) **A test that hardcodes a classid the domain now selects BY FEATURE does not diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 50a109ee9..280eca4c1 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,34 @@ +## 2026-08-06 — idle-flush-dataset-eviction v1 — PROPOSAL (not scheduled; nothing implemented, nothing measured) + +**Plan:** `.claude/plans/idle-flush-dataset-eviction-v1.md` +Feature-gated (**off by default**) eviction of a Lance dataset's LOCAL copy after +an idle period, pushed back to the object store first if dirty, rehydrating on +next access. **Purpose is cost smoothing, not capacity** (operator framing): the +win is the shape of the bill — local disk bills continuously for capacity +provisioned, object storage for what is kept — so the plan explicitly does NOT +justify itself with "otherwise you run out of disk", and never fails an +operation to hold a number. **Operator-set defaults (heuristic, both config):** +idle **> 3 days** AND footprint **> ~300 MB**, pressure-driven and age-ordered; +under budget nothing is ever evicted however stale, over budget the stalest go +first until back under. **The budget is a SOFT spot** — an in-use dataset larger +than the whole budget stays resident, a sweep may legitimately reach no target, +and that state must be *observable* ("no candidate old enough" vs "every +candidate in use" are different findings). Dirty detection = the Lance dataset +**version**, never a hash — carrying an **unclosed verification gate** (a cheap +local version read is assumed, not checked; a BLOCKER if it fails). +**A lease/refcount/guard-type protocol was CONSIDERED AND REJECTED** as +disproportionate at a 3-day floor (operator scope correction) — cheap +check-then-act instead, with the bar set at **"does not corrupt"** (worst case a +wasted rehydration) rather than "cannot occur"; the rejection is recorded so it +is not re-added, and the revisit condition (threshold dropping from days to +hours) is named. Acceptance criteria are **fire/silence pairs** per the P0 +falsifiability rule — including the conjunction-splitting silence tests that a +staleness-only policy would fail, and a race test deliberately shaped as a +corruption test rather than an impossibility proof. Five open items, incl. +multi-process access and whether a sweep may *initiate* push-back (currently +assumed **skip**). Prerequisite reading: +`.claude/knowledge/s3-hydration-lifecycle.md`. + ## 2026-08-05 — measure-64k-axes v3 — ACTIVE (the three arms Stage A0 earned; M+O build lane dispatched) **Plan:** `.claude/plans/measure-64k-axes-v3.md` diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index ecea4b205..2b9bb72cc 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,12 @@ +## 2026-08-06 — branch `claude/s3-hydration-lifecycle` — object-store hydration doctrine + idle-flush plan v1 (docs only, no Rust) + +**Documentation-only.** No crate, type, feature or test changed; nothing in `Cargo.toml` touched. + +- **NEW knowledge doc `.claude/knowledge/s3-hydration-lifecycle.md`** (`READ BY:` header + per-claim evidence table, every row graded). The three-layer split — **object store hydrates / local directory IS the store / persistent volume only decides whether hydration repeats**. Lance opens a network-scheme URI natively and *that is the trap*: the wrong architecture runs and only degrades, while **deleting the mmap layer** (a remote read lands in a fresh buffer — one copy per read, no page cache), so every zero-copy guarantee under it becomes a claim about copied bytes (`zero-copy-lens-law.md`, one layer down). **Any** local directory satisfies zero-copy — a volume is an optimization on hydration *frequency*, never a correctness requirement. Carries the feature-gate diagnosis (**manifest-verified in session**: `lancedb` `default = []`; `aws` forwards to `lance/aws` + `lance-io/aws` (+ `object_store/aws`); `lance-io` carries `aws` in its OWN defaults, so the layer that opts out is `lancedb` — the reason the diagnosis goes wrong is that the mental model is correct about the wrong crate), the mechanical rule (**scheme-named error = BUILD problem; credential/host/region-named error = CONFIG problem**), the four-state lifecycle (absent/hydrated/dirty/flushed) with **flush legal only from `hydrated`** (the `dirty → flushed` edge is data loss with no error), and one reported single-observation endpoint measurement set graded as ratios-generalize / absolutes-do-not: **NOT viable as swap or as a page-fault backing store; VIABLE for hydration and build caches.** +- **NEW plan `.claude/plans/idle-flush-dataset-eviction-v1.md`** — **PROPOSAL, nothing implemented, nothing measured.** Feature-gated (off by default) idle-flush eviction: a dataset idle past a floor has its local copy dropped (pushed back first if dirty), rehydrating on next access. **Purpose is COST SMOOTHING, not capacity** (operator framing — the win is the shape of the bill; local disk bills continuously for capacity provisioned, object storage for what is kept). **Operator-set default policy:** age **> 3 days** AND footprint **> ~300 MB**, pressure-driven and age-ordered — under budget nothing is ever evicted however stale; over budget the stalest go first. **~300 MB is a SOFT spot**: no operation may ever fail to hold the number, an in-use dataset larger than the whole budget stays resident (**correctness beats the watermark**), and a sweep that reaches no target is a legitimate steady state — which forces the observability requirement that *"no candidate old enough"* and *"every candidate in use"* be distinguishable. Dirty detection = the **Lance dataset version**, never a hash, with an explicit **unclosed verification gate** (a cheap local version read is *assumed*, not checked — a BLOCKER if it fails). **A lease/refcount/guard protocol was CONSIDERED AND REJECTED** as disproportionate at a 3-day floor (operator scope correction): cheap check-then-act, and the bar is **"does not corrupt"** (worst case a wasted rehydration) rather than **"cannot occur"** — recorded rather than left silent so it is not re-added, with the revisit condition named (threshold dropping from days to hours). Acceptance criteria written as **fire/silence pairs** per the P0 rule, including the conjunction-splitting silence tests (under-budget-but-stale, over-budget-but-fresh) that a staleness-only policy would fail. +- **`docs/DATAFUSION-PERIMETER.md` §9a (NEW section)** — cross-reference: the object-store provider is the *same class of fact* that document already catalogues, one crate over (a capability behind a default-off feature, diagnosed at the wrong layer). +- Board: `.claude/board/EPIPHANIES.md` PREPEND ×2 (`E-OBJECT-STORE-HYDRATES-IT-DOES-NOT-STORE-1`, `E-IDLE-FLUSH-IS-COST-SMOOTHING-NOT-CAPACITY-AND-THE-3-DAY-FLOOR-PRICES-OUT-A-LEASE-PROTOCOL-1`); `.claude/board/INTEGRATION_PLANS.md` PREPEND ×1. + ## 2026-08-05 — lance 9 / lancedb 0.33 / DataFusion 54 / Rust 1.97.1 — the ecosystem bump, MEASURED then LANDED across 9 repos **Current pins:** `lance =9.0.0`, `lancedb =0.33.0`, `datafusion 54`, `arrow 58` (unmoved), `object_store 0.13.2` (unmoved), toolchain **1.97.1**. Plan: `.claude/plans/lance9-datafusion54-upgrade-probe-v1.md`. diff --git a/.claude/knowledge/s3-hydration-lifecycle.md b/.claude/knowledge/s3-hydration-lifecycle.md new file mode 100644 index 000000000..259092807 --- /dev/null +++ b/.claude/knowledge/s3-hydration-lifecycle.md @@ -0,0 +1,206 @@ +# S3 is the hydration path, never the store — the on-demand Lance dataset lifecycle + +> **READ BY:** any session that opens a Lance dataset from a URI, wires an +> object-store backend, sizes a persistent volume, debugs a +> `"No object store provider found for scheme"` error, plans a rebake input, or +> proposes putting a dataset "in S3" as a runtime store. Also the +> `integration-lead` / `layer-boundary-warden` cards when a deployment topology +> question arrives. +> +> **Companions:** `.claude/knowledge/zero-copy-lens-law.md` (why a local +> filesystem is not a preference but the precondition — the lens needs mapped +> bytes, and a network fetch has none to lend) · +> `.claude/knowledge/ephemeral-warm-cold-lifecycle.md` (the *reasoning* tier +> ladder; this doc is the *bytes-on-disk* ladder and does not touch it) · +> `docs/DATAFUSION-PERIMETER.md` §11 (the feature-closure half of §3 below). + +## The one-line statement + +> **The object store hydrates; the local filesystem stores; the volume only +> decides whether hydration repeats.** Three layers, one job each. Collapsing +> any two of them is the failure this doc exists to prevent. + +## Evidence status (per the workspace rule: label everything) + +| claim | status | evidence | +|---|---|---| +| `lancedb` ships `default = []`; its `aws` feature forwards to `lance/aws` + `lance-io/aws` (+ `object_store/aws` directly in newer releases, transitively via `lance-io` in older ones) | **FINDING** — source-verified in this session | Read directly from the vendored `lancedb` manifests in the local registry, two releases apart; both show `default = []` and the same forwarding shape. | +| `lance-io` carries `aws` in its OWN defaults — so the opt-out is `lancedb`'s layer, not the stack's | **FINDING** — source-verified in this session | Same read: `lance-io`'s `[features] default` includes `aws`. | +| Without the feature, an `s3://` URI fails at provider registration — BEFORE any credential, endpoint or region is consulted | **FINDING** — reported measurement, not re-verified here | Reported by the session that hit it; the error text names the *scheme*, not a credential. Consistent with the manifest facts above (no provider is compiled in). Falsifier: build with the feature off and confirm the same URI fails identically with every env var unset AND with all of them set correctly. | +| Opening the object store as the runtime store makes every read a network fetch into a fresh buffer — no mmap, no page cache | **FINDING** (mechanism), *unbenchmarked here* | Structural: a remote range request has no mapped page to lend. Follows from the zero-copy law. **No A/B benchmark of remote-store vs local-store read paths has been run in this workspace.** | +| Any local directory satisfies zero-copy; a persistent volume is not a correctness requirement | **FINDING** (mechanism) | The store's requirement is a filesystem path, not a durable one. Persistence changes *how often you hydrate*, never *whether reads are zero-copy*. | +| The endpoint characteristics in §5 | **reported measurement, not re-verified in this session** | Measured once, against one S3-compatible endpoint, from one region, at one time. Provider- and region-dependent; treat the *ratios* as the finding and the absolute numbers as a single observation. | +| The flush/rehydrate lifecycle in §4 is the right shape for large single-use datasets | **CONJECTURE** | Argued from §5's ratios, not from a deployed instance. Falsifier stated inline at §4. **No probe has run.** | + +Nothing below is promoted past its row here. + +## 1. The three layers + +| layer | its ONE job | if it is absent | +|---|---|---| +| **object store** (S3-compatible) | **hydration source** — durable, versioned, shared between machines and between builds | fall back to whatever secondary source the consumer already has; the store still works, the dataset just has to come from somewhere else | +| **local directory** | **THE Lance store** — the path the process opens; zero-copy mmap reads, page cache, no network in the read path | **no fallback — always required.** But *any* local path satisfies it | +| **persistent volume** | decides **which** local directory — chosen only because it survives redeploys | hydrate on every boot; still correct, merely slower | + +Read the third row twice. The volume is an **optimization on hydration +frequency**, not a component of the store. A design that says "we need a volume +or this doesn't work" has mis-assigned a job: what it needs is a directory. + +## 2. Why the object store must not be the store — even though the URI works + +Lance opens an `s3://` URI natively. That is exactly what makes this trap +easy to fall into: the wrong architecture **runs**, correctly, and only +degrades. + +The reason it is wrong is the same reason the zero-copy law exists one layer +down. A local dataset read is a mapped page — the kernel hands you bytes that +are already resident, and a lens over them costs a cast. A remote object read +is a range request that lands in a **freshly allocated buffer**: one copy per +read, minimum, plus a round trip, and no page cache to make the second read of +the same bytes free. + +So the failure is not "S3 is slow." It is that **mounting the object store as +the runtime store deletes the mmap layer from the architecture** — every +downstream zero-copy guarantee is then a claim about buffers that were copied +into existence. The lens has nothing to borrow from. + +> **The review question:** *where does the process open its dataset from?* If +> the answer is a URI with a network scheme, the zero-copy story below it is +> already void, regardless of what any type signature promises. + +**Corollary — the local directory has no minimum quality.** An ephemeral +container path is functionally correct: mmap works, the page cache works, the +lens works. Losing that directory on redeploy costs a re-hydration, not a +correctness property. This is why §1's third row is an optimization and not a +requirement. + +## 3. The feature gate that costs an hour if you don't know it + +**`lancedb` ships `default = []`.** Its `aws` feature is what forwards to +`lance/aws` + `lance-io/aws` (+ `object_store/aws`), and *that* forwarding is +what registers the S3 provider. + +Two consequences, both non-obvious: + +1. **Without the feature, an `s3://` URI fails at provider lookup — before any + credential, endpoint, or region is read.** The error names the *scheme*. + That means **no amount of endpoint/region/credential/quoting debugging can + possibly help**, because none of that code has been reached. Every minute + spent on env vars is spent on a code path that does not exist in the binary. +2. **`lance-io` DOES carry `aws` in its own defaults.** So the intuition "the + Lance stack supports S3 by default" is *true one layer down* and false at + the layer you depend on. `lancedb` is the layer that opts out. That mismatch + is the whole reason the diagnosis goes wrong: the mental model is correct + about the wrong crate. + +**The diagnostic rule, mechanical:** an object-store error that names a +**scheme** is a *build* problem (a feature is off). An object-store error that +names a **credential, host, bucket, region, or signature** is a *config* +problem. Never debug the second when you are looking at the first. Read the +error's noun before touching an env var. + +*(Env var **names** — `AWS_ENDPOINT_URL`, `AWS_REGION`, and the standard +credential pair — are the config surface for the second class only. They are +inert against the first.)* + +## 4. The lifecycle — four states, and what each transition costs + +The actual operational ask: **large, single-use datasets** (rebake inputs, +one-off derivations, build artifacts) should not occupy local disk permanently. +They hydrate when needed, get pushed back if mutated, and their local copy is +reclaimed. + +| state | what exists where | invariant | +|---|---|---| +| **absent** | object store only | reads are impossible; the store is not open | +| **hydrated** | object store + local dir, identical | reads are zero-copy; this is the only readable state | +| **dirty** | local dir has diverged (written/appended/compacted) | **the local copy is now the only truth** — flushing here destroys data | +| **flushed** | object store only, local reclaimed | ≡ *absent*, but reached deliberately after a push | + +Transitions and their costs: + +| transition | cost | gate | +|---|---|---| +| absent → hydrated | one large sequential read (§5: sustained, amortized) + one connect | none; safe to repeat, it is idempotent | +| hydrated → dirty | a local write; free | — | +| dirty → hydrated | **push back** — the expensive direction (§5: writes are ~½ read throughput and pay per-fragment object overhead) | must complete before flush, or the divergence is lost | +| hydrated → flushed | a local delete; frees disk | **only legal from `hydrated`, never from `dirty`** | +| flushed → hydrated | same as absent → hydrated | — | + +**The one rule that matters:** *flush is legal only from `hydrated`, never from +`dirty`.* The state machine exists to make that a checkable condition rather +than an assumption. A `dirty → flushed` edge is data loss with no error. + +**Why writes are an ops step and never a boot path:** the push-back direction +pays object-per-fragment overhead on top of raw throughput (§5), so its cost +scales with fragmentation as well as bytes. Hydration is boot-viable; the return +trip is not. Any design that puts a write-back on a startup path has put the +slowest, most fragmentation-sensitive operation in front of the readiness check. + +*Falsifier for the CONJECTURE row:* run the full cycle on a representative +dataset and confirm (a) hydrate wall-clock stays inside the boot budget, (b) a +`dirty → flushed` attempt is refused rather than silently accepted, and (c) a +rehydrate after flush reads back byte-identically. Until that runs, §4 is a +design, not a result. + +## 5. Measured characteristics of one S3-compatible endpoint + +> **Single observation.** One provider, one region, one point in time, +> deliberately unnamed. **Provider- and region-dependent.** The *ratios* are +> what generalize; the absolute numbers do not. Reported by the session that +> measured them; **not re-run here.** + +| operation | observed | what it settles | +|---|---|---| +| small-object round trip | **~250 ms** | | +| large sequential read | **~21 MiB/s** | | +| large sequential write | **~11 MiB/s** (≈ ½ the read rate) | | +| store connect | **~730 ms** | | +| cold re-open + full count, dataset in the tens-of-MB range (~69k rows, ~35 MB) | **~1.4 s** | boot-viable | +| write of that same dataset | **~7.7 s** (≈ 4.4 MiB/s effective — below the raw write rate, the gap being object-per-fragment overhead) | ops step, not a boot path | + +**The round-trip number is the load-bearing one.** At ~250 ms per small object, +the object store is roughly **~2.5 million×** slower than RAM and **~2500×** +slower than NVMe. Those two ratios are the whole argument: + +- **NOT viable as swap.** A page fault backed by a ~250 ms fetch is not a + memory hierarchy; it is a hang with a progress bar. +- **NOT viable as a page-fault backing store**, for the same reason — and this + is precisely §2 restated in numbers: mounting it as the runtime store puts + that latency *under every read*. +- **VIABLE for hydration.** One large sequential transfer amortizes the round + trip across the whole dataset; the effective cost is the ~21 MiB/s line plus + one connect. +- **VIABLE for build caches** — same shape: large objects, few of them, latency + amortized. + +The rule that falls out: **the object store is fine when the object count is +small and the objects are large; it is unusable when the access count is large +and the accesses are small.** Every viable/non-viable verdict above is that one +sentence applied twice. + +## 6. Consequences for new work + +- **Never open a network-scheme URI as the runtime store.** Hydrate to a local + path, open the local path. If a design opens the remote URI directly, the + finding is not "this is slow" — it is that the zero-copy layer has been + removed. +- **Do not require a persistent volume for correctness.** Require a *directory*. + State the volume as an optimization with a named cost (one hydration per + boot), so a deployment without one is a known trade rather than a bug report. +- **Gate the object-store feature explicitly, and say so where the URI is + parsed.** A scheme-named error must lead the reader to the manifest, not to + the credentials. +- **Never place a write-back on a startup path.** Push-back is an operational + step with its own trigger. +- **Any flush path must assert `hydrated`, not assume it.** The `dirty → flushed` + edge fails silently by construction; only an explicit check catches it. + +## Cross-refs + +`.claude/knowledge/zero-copy-lens-law.md` (the law this doc is the storage-siting +corollary of) · `.claude/knowledge/ephemeral-warm-cold-lifecycle.md` (the +reasoning-tier ladder — orthogonal; do not conflate its cold tier with this +doc's flushed state) · `docs/DATAFUSION-PERIMETER.md` §11 (feature-closure half) +· ADR-022/023 (the Firewall — no serialization in the hot path; a remote read +path is that violation arriving through the storage layer). diff --git a/.claude/plans/idle-flush-dataset-eviction-v1.md b/.claude/plans/idle-flush-dataset-eviction-v1.md new file mode 100644 index 000000000..85b2e175b --- /dev/null +++ b/.claude/plans/idle-flush-dataset-eviction-v1.md @@ -0,0 +1,331 @@ +# Idle-flush dataset eviction — plan v1 + +> **Status:** PROPOSAL. Nothing here is implemented; nothing here is measured. +> **Scope:** design + acceptance criteria for a feature-gated local-copy +> eviction policy over Lance datasets. **This plan does not authorize the +> implementation** — it states what the implementation would owe. +> +> **Prerequisite reading:** `.claude/knowledge/s3-hydration-lifecycle.md` — the +> three-layer model (object store hydrates / local dir stores / volume only +> decides whether hydration repeats), the four lifecycle states, and the one +> reported measurement this plan's cost model rests on. This plan **extends** +> that lifecycle with an automatic `hydrated → flushed` trigger; it restates +> none of it. + +## 0. Evidence grading (workspace rule: label everything) + +| claim | status | +|---|---| +| The four-state lifecycle and its legal transitions | **FINDING** (mechanism) — see the knowledge doc | +| Rehydration of a tens-of-MB dataset is ~1.4 s | **reported measurement**, single observation, provider- and region-dependent, not re-run here | +| The default policy: age floor **3 days** + soft budget **~300 MB**, pressure-driven and age-ordered (§2) | **OPERATOR-SET POLICY** — a heuristic starting point, explicitly **not measured**. Both are config; these are defaults. | +| Age-ordering (rather than a `size × idleness` key) is the right default (§2) | **CONJECTURE** — argued (rehydration cost is also size-proportional), not measured; the size-weighted variant is deferred, not rejected | +| A watermark-driven sweep dominates both a bare timer and a bare allocation-failure signal (§3) | **CONJECTURE** — argued from the two failure modes, no deployed instance | +| The Lance dataset version is a sufficient dirty-detector (§4) | **CONJECTURE**, with a named verification gate that must close before implementation | +| At a 3-day floor the flush/read race is negligible, so check-then-act suffices and a lease protocol is disproportionate (§5) | **OPERATOR-SET SCOPE RULING** — the requirement is *does not corrupt*, not *cannot occur*; revisit if the threshold drops to hours | + +**No probe has run for any row marked CONJECTURE.** The falsifiers are §7. + +## 1. What this buys — cost SMOOTHING, not capacity + +**The win is the shape of the bill, not a capacity ceiling.** This is not a +mechanism for fitting a working set into a disk that is too small, and the plan +does **not** justify itself with "otherwise you run out of disk." It exists so +that datasets nobody is reading stop being paid for continuously. + +The economics are structural: **local disk is billed continuously for capacity +provisioned; object storage is billed for what is kept.** A dataset touched once +and then retained forever pays the continuous rate for a one-time access. +Eviction flattens that curve — a one-time transfer plus storage-at-rest, instead +of a standing charge for bytes nobody reads. The population this targets is +large single-use material: one-off corpora, rebake inputs, derivations touched +once and never again. + +Two consequences of the framing, both binding on the design: + +- **Never fail an operation to hold the number.** The budget is a soft + watermark (§2); there is no admission control, because a smoothing mechanism + that breaks a workload has traded a billing improvement for an outage. +- **The trade is explicit:** a flushed dataset's next access costs a + rehydration (~1.4 s at the reported tens-of-MB scale — single observation, + §0). Worthwhile exactly when access is sparse enough that this is paid rarely. + When it is paid often the policy is *worse* than doing nothing, which is why + §7's thrash falsifier is a gating acceptance criterion rather than an + afterthought. + +## 2. The shipped default policy (operator-set) — answers Q1 and Q2 + +**Two conditions, BOTH required before anything is evicted:** + +| condition | default | is | +|---|---|---| +| **Age** — dataset idle since last use | **> 3 days** | a flush *candidate* | +| **Budget** — total local footprint across all datasets | **> ~300 MB** | eviction *engages at all* | + +**The trigger is pressure-driven and age-ordered.** Under the budget, **nothing +is ever evicted, no matter how stale** — a small deployment pays nothing, not +even churn. Over the budget, the **stalest candidates go first** until the +footprint is back under. + +**Both numbers are configuration; these are the defaults.** They are +**operator-set, not measured** — grade them as policy, never as findings. `~300 MB` +in particular is a heuristic starting point. + +### "Soft spot" is load-bearing + +**~300 MB is the point where pressure BEGINS, not a hard cap that must never be +exceeded.** Three consequences, and each is a design constraint rather than a +nicety: + +1. **No operation may ever fail because the budget is exceeded.** The sweep + evicts what it can and carries on. There is no admission control, no + back-pressure onto callers, no error path keyed to the watermark. +2. **A dataset actively in use, larger than the whole budget, stays resident.** + **Correctness beats the watermark, always.** The budget cannot evict a live + in-use dataset (§5), and the single-dataset-over-budget case is an ordinary + steady state rather than a special case to handle. +3. **Eviction may therefore fail to reach the target**, when everything resident + is in use or nothing is old enough. That is a **legitimate steady state, not + an error** — see the observability requirement below. + +### Q1 — what the age is measured from **ANSWERED** + +**Age is time since last USE — read or write.** Either touch is evidence the +dataset is in the working set. (Dirtiness is a *separate* axis: a write also +makes the dataset dirty, and dirtiness governs whether flushing is **legal** +(§4, §7) — never whether it is **desirable**.) + +**Ordering among candidates is by age: stalest first.** The 3-day floor is what +does the real work of keeping a hot working set intact, and it does so +*regardless of size* — which is why the default policy does not need a +size-weighted key. + +> **Deferred refinement (CONJECTURE, not the default):** ranking candidates by +> `bytes_on_disk × idle_seconds` instead of age alone would evict the largest +> disk-seconds first and reach the budget in fewer evictions. It is **not** the +> shipped policy because it cuts both ways: rehydration cost is *also* +> proportional to size, so a size-weighted key preferentially evicts what is +> most expensive to get back. Age-ordering under a hard age floor is the +> conservative choice. Revisit only with measured access-pattern data — which +> nobody has. + +### Observability — the requirement that keeps the soft watermark honest + +**A deployment silently over budget must not look identical to one under it.** +The sweep therefore reports, every time it runs and reaches no target: + +- current footprint vs budget, and **why the sweep stopped** — distinguishing + **"no candidate was old enough"** from **"every candidate was in use"**. These + are different operational situations with different responses and must not + collapse into one "could not evict" line. + +**The over-budget-but-nothing-stale case is INTENDED, not a bug.** If the +footprint exceeds ~300 MB but nothing has been idle for 3 days, **nothing is +evicted** — that is precisely the age floor protecting a hot working set from +being thrashed by budget pressure. What an operator sees in that case is the +first reason above: *over budget, zero candidates past the age floor.* If that +line persists, the working set genuinely exceeds the budget, and the response is +to raise the budget or reduce the working set — a **capacity finding**, not a +policy failure. + +## 3. Q2 — who triggers the flush? **ANSWERED (watermark, per the policy above)** + +Neither a bare timer nor a bare pressure signal. Both fail in a stated way: + +| trigger | failure mode | +|---|---| +| bare timer | wakes and does work on datasets nobody is asking about, and pays the sweep cost even when disk is abundant | +| bare allocation-failure signal | only acts when it must — but by then it is **inside a request**, so the eviction *and* any subsequent rehydration are charged to a caller who is waiting | + +**The design is a watermark-driven background sweep**, which is what the +operator policy selects. The sweep task exists but **does nothing while the +footprint is under budget** — that removes the bare timer's cost, because the +common case is a no-op check. Over budget, it evicts age-ordered candidates past +the floor, off the request path, until the footprint is back under **or it runs +out of candidates** (§2 — a legitimate stop, reported). + +**A last-resort synchronous fallback on genuine allocation failure may exist, +but it is not part of the budget mechanism** and must carry its own counter. +Because the watermark is soft and never fails an operation, this path is a +disk-actually-full condition, not a budget condition — conflating the two would +smuggle admission control in through the back door. + +## 4. Q3 — dirty detection without hashing **ANSWERED, with a verification gate** + +**Do not hash.** Hashing 35 MB to answer a boolean is the wrong cost class and +scales with the thing being avoided. + +**The mechanism is the Lance dataset version.** Lance datasets are versioned; +record `version_at_hydration` when the local copy is established, and define +**dirty ⇔ `current_local_version != version_at_hydration`**. This is an integer +comparison against a generation counter the storage layer already maintains for +its own reasons — no new bookkeeping, no content scan, and it is correct across +compaction and append alike because those are what bump it. + +**mtime is a corroborating signal, never the authority.** It is cheap and can +serve as a fast pre-check, but it is not reliable enough across filesystems and +maintenance operations to gate a destructive action. If mtime and version +disagree, **version wins and the disagreement is logged** — a disagreement is +itself worth seeing. + +> **VERIFICATION GATE (must close before implementation, not assumed):** +> confirm that the current local dataset version can be read **cheaply and +> without a full dataset open**, or that a local open is cheap enough to run per +> sweep candidate. If neither holds, this mechanism is a **BLOCKER**, not a +> design — and the honest response is to say so and stop, not to substitute a +> heuristic. Stating it plainly: *this plan assumes a cheap local version read +> exists; that assumption has not been checked against the API.* + +## 5. Q4 — the flush/read race: cheap check-then-act, NOT a lease protocol + +> **Scope correction (operator).** An earlier draft called this "the part most +> likely to be subtly wrong" and specified a refcounted guard type. **That is +> walked back as disproportionate.** *"Given 3 days not used it's just +> flattening the payment curve and hardly attributing to race conditions."* + +**At a 3-day idle threshold the race is vanishingly rare.** A dataset untouched +for three days is not plausibly under an active mmap at the moment the sweeper +decides to flush it. The design must therefore be **safe if it happens**, not +**engineered around the possibility**. + +**A lease / refcount / guard-type protocol was CONSIDERED AND REJECTED** — and +that is recorded here rather than left silent, so a future reader does not +re-add it believing it was overlooked. Reason: it puts a permanent cost (a guard +in every read signature, an atomic state-machine step, a second gate-off code +path per §6) on **every** read, to close a window that the age floor already +makes negligible. The protocol is priced for a hot cache; this is not one. + +**What the design does instead — cheap check-then-act:** + +- **Skip the flush if a read is in flight.** A cheap, non-authoritative check is + sufficient; it does not need to be race-free, because losing the race is not + harmful (below). +- **If a read begins mid-flush, let it rehydrate.** Do not block it, do not + coordinate with the sweeper. The reader's own hydration path is already the + recovery mechanism — `absent → hydrated` is idempotent and safe to repeat. +- **The failure mode must be RECOVERABLE, never CORRUPTING.** Worst case is a + **wasted rehydration** (~1.4 s at the reported scale). Never a torn read, + never a partially-visible dataset, never bytes removed from under a live + mapping observed as corruption. + +The bar the implementation must clear is therefore **"does not corrupt"**, not +**"cannot occur"** — and §8 asserts it that way. + +> **Revisit condition, stated so the decision is falsifiable rather than +> permanent:** if the idle threshold ever drops from **days to hours**, the +> calculus changes — the race stops being negligible and the protocol question +> **reopens**. Tie any such threshold change to a re-read of this section. + +## 6. Q6 — why a feature gate, and what "off" excludes **ANSWERED** + +**Off by default.** A consumer with ample local disk must pay *nothing* — and +"nothing" is meant literally, not "a cheap timer". + +With the gate off, the following do not exist in the binary: the sweep task and +its wakeups, the per-dataset size and last-read accounting, the eviction key +computation, the watermark checks, and the object-store push path used for +`dirty → hydrated`. The store opens the local path exactly as it does today. + +**Because §5 rejects the guard type, the read path does not change shape with +the gate** — which removes the `I-LEGACY-API-FEATURE-GATED` hazard the earlier +draft had to design around. The in-flight check is a cheap, non-authoritative +read of the same bookkeeping the sweep uses; when the gate is off that +bookkeeping does not exist and neither does the check. + +**The constraint that remains: the same function name must not mean different +things under different gate states.** Gate-off must be *inert*, never *subtly +different* — asserted by T12. + +**Orthogonality note:** idle-flush *requires* the object-store feature (§3 of +the knowledge doc) but does not *imply* it. Two independent gates; enabling +idle-flush without the object-store provider must be a **build-time or +startup-time refusal**, never a runtime surprise on the first eviction — the +scheme-error diagnosis trap, one layer up. + +## 7. Q5 — what it costs when wrong, and the falsifiers **ANSWERED** + +Rehydration is ~1.4 s at the reported tens-of-MB scale (single observation). **A +thrashing policy converts that from a rare cost into a per-request cost** — the +policy then makes the system strictly worse than not having it, while appearing +to function. + +**The thrash falsifier — the gating acceptance criterion:** + +> Over a measurement window, compute +> **`rehydrations / distinct_datasets_accessed`**. +> A value **> 1.0** means at least one dataset was evicted and re-fetched +> *within its own working set* — the definition of thrash. +> A **second hydration of the same dataset inside one age-floor window** is the +> same finding at single-dataset granularity, and is the sharper signal. + +Supporting measurement: **`total_hydration_seconds / total_read_seconds`**. This +is the amortization ratio; if hydration time approaches read time, the feature +is paying for itself with the thing it was supposed to make cheaper. + +**Both must be instrumented before the policy is enabled anywhere**, or a +thrashing deployment is indistinguishable from a working one — the observation +that motivates this whole section. + +## 8. Acceptance criteria — the tests the implementation owes + +Per the workspace P0 falsifiability rule, **every guard owes a can-fire test AND +a can-stay-silent test, both on non-trivial inputs.** An eviction policy that +never evicts and one that evicts everything are equally useless and both pass a +naive assertion. Enumerated: + +| # | test | what it proves | +|---|---|---| +| T1 | **Eviction CAN fire** — over budget **and** a past-the-floor stale candidate present ⇒ something is evicted | the policy acts, and only when BOTH conditions hold | +| T2a | **Silent UNDER BUDGET** — footprint under the budget with a **very stale** dataset present ⇒ **nothing** is evicted | staleness alone must NOT evict; a policy that evicted on staleness alone would pass T1 and be wrong | +| T2b | **Silent OVER BUDGET, all fresh** — footprint over budget but no candidate past the age floor ⇒ **nothing** is evicted, and the sweep reports *zero candidates past the floor* | the age floor genuinely protects a hot working set, and the intended no-op is **observable** | +| T2c | **Discrimination within one sweep** — a within-floor dataset is spared *while* a past-the-floor one in the same sweep IS evicted | the policy discriminates; the paired candidate is what makes it non-vacuous (a silence test on an empty candidate set proves only that emptiness is handled) | +| T3 | **Age ordering is load-bearing** — among several past-the-floor candidates, the stalest is evicted first | ordering is by age, per the default policy | +| T4 | **Age-floor inertness** — raising the floor silences an eviction a lower value admits; lowering it admits one a higher value silences | the parameter is not decoration | +| T5 | **Budget inertness** — raising the budget above the current footprint silences a sweep that a lower budget engages; the sweep stops as soon as the footprint is back under | the budget is the trigger, and it is a threshold rather than decoration | +| T5b | **The budget is SOFT** — a single in-use dataset larger than the whole budget stays resident and **no operation fails**; the sweep reports *every candidate in use* | correctness beats the watermark; there is no admission control | +| T6 | **`dirty → flushed` is REFUSED** — a dirty dataset offered to the flush path is rejected, not silently accepted | the destructive edge is checked, not assumed (the knowledge doc's one rule) | +| T7 | **Dirty is DETECTED** — a mutation makes the version differ and the dataset reads dirty; **and an unmutated dataset reads clean after a full sweep** | the detector discriminates rather than always-firing or never-firing | +| T8 | **In-flight read is skipped** — a dataset with a read in flight is not flushed; **and the same dataset IS flushed once the read completes** | the cheap check discriminates in both directions (not always-skip, not never-skip) | +| T9 | **A LOST race does not corrupt** — force the interleaving (read begins mid-flush) and assert the reader gets a **correct, complete dataset** via rehydration. The cost may be a wasted rehydration; the result may **never** be a torn or partial read | §5's actual bar: *does not corrupt*, NOT *cannot occur* — this test deliberately makes the race happen rather than proving it impossible | +| T10 | **Rehydrate is byte-identical** — flush → rehydrate → read equals the pre-flush read | the round trip is lossless | +| T11 | **Thrash detector CAN fire** — a synthetic access pattern designed to thrash produces a ratio > 1.0; **and a well-behaved pattern produces ≤ 1.0** | §7's metric discriminates — a detector that fires on everything carries no information | +| T12 | **Gate-off is inert** — with the feature off, no sweep runs, no accounting is kept, and the read path is unchanged (per §6, either one code path or a proven-equivalent second one) | the gate costs nothing when off | + +**T2a/T2b, T7, T8 and T11 are the ones that matter most** — each is a paired +fire/silence test on the exact guard whose degenerate always-on or always-off +form would otherwise pass review. **T2a is the sharpest:** a policy that evicted +on staleness alone would pass T1 and still be wrong. + +**T9 is deliberately shaped as a corruption test, not an impossibility proof.** +Asserting the race "cannot happen" would be exactly the vacuous assertion the P0 +rule forbids — implied by the code, falsifiable by nothing. + +## 9. Open items (explicitly NOT answered) + +1. **The §4 verification gate.** Cheap local version read — assumed, unchecked. + Closing this is the first task; if it fails, the plan needs a different + dirty-detector and this document is wrong rather than incomplete. +2. **Whether the default values are right.** The 3-day floor and ~300 MB budget + are operator-set starting points, not derived from a measured access-pattern + distribution. They are config precisely because that distribution is unknown; + the deferred size-weighted ranking (§2) should be revisited only once it is. +3. **Multi-process access to one local directory.** The in-flight check is + in-process. Two processes over one directory is out of scope and would need a + different mechanism — named here so it is not assumed handled. (Note it does + not change §5's bar: the worst case stays a wasted rehydration.) +4. **Partial hydration.** Whether a subset (fragment / column range) can be + hydrated instead of a whole dataset is unexplored. It would change the size + term of the eviction key, so it is a policy question, not just an I/O one. +5. **Interaction with the push-back direction.** `dirty → hydrated` is the + expensive direction and is currently an operational step. Whether a sweep + may *initiate* a push-back (making eviction possible) or only skip dirty + candidates is undecided; the plan currently assumes **skip**, which is the + conservative choice and possibly the wrong one for the target workload. + +## Cross-refs + +`.claude/knowledge/s3-hydration-lifecycle.md` (the lifecycle, the three-layer +model, the reported measurements) · `.claude/knowledge/zero-copy-lens-law.md` +(why the local copy is not optional — the mapped bytes that law depends on are +what §5's "does not corrupt" bar protects) · `I-LEGACY-API-FEATURE-GATED` (§6's +same-name-different-semantics constraint). diff --git a/docs/DATAFUSION-PERIMETER.md b/docs/DATAFUSION-PERIMETER.md index b9ba25902..50931c58e 100644 --- a/docs/DATAFUSION-PERIMETER.md +++ b/docs/DATAFUSION-PERIMETER.md @@ -325,6 +325,31 @@ it, **before** §5 was written with a same-identifier grep. Documenting an anti-pattern does not immunise the document against it. The defence is mechanical — resolve imports, leave the workspace, count edges — not vigilance. +## 9a. The object-store provider is the same class of fact — and the same trap + +Cross-reference, because it is the identical failure shape one crate over: +a capability that "obviously exists" is behind a **feature that is off by +default**, and the resulting error is diagnosed at the wrong layer. + +Source-verified from the vendored manifests (two releases apart, both agree): +**`lancedb` ships `default = []`**, and its `aws` feature is what forwards to +`lance/aws` + `lance-io/aws` (+ `object_store/aws` directly in the newer +release, transitively via `lance-io` in the older). Meanwhile **`lance-io` +carries `aws` in its OWN defaults** — so the intuition "the Lance stack does +object storage by default" is true one layer down and false at the layer we +depend on. `lancedb` is the layer that opts out. + +Consequence, and it is the §9 lesson restated: without the feature an +object-store URI fails at **provider lookup by scheme**, before any credential, +endpoint or region is consulted — so credential and endpoint debugging is +spent on code that is not in the binary. **An error naming a *scheme* is a +build problem; an error naming a *credential/host/region* is a config +problem.** Read the error's noun before touching configuration. + +Full treatment — the three-layer hydration model, why the object store must +not be the runtime store, and the flush/rehydrate lifecycle: +`.claude/knowledge/s3-hydration-lifecycle.md`. + ## 10. Open questions, in decision order 1. **What does this repo actually use from `lance` that `lance-table` lacks?** From ea33f3ba1c2a5a96ec3728de9b805d1473f4022a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 6 Aug 2026 13:06:54 +0000 Subject: [PATCH 2/2] =?UTF-8?q?docs(hydration):=20review=20round=20on=20PR?= =?UTF-8?q?=20#901=20=E2=80=94=20six=20corrections,=20one=20load-bearing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every review comment evaluated against the code and the manifests before acting; the corrections below are the ones that survived that check. THE LOAD-BEARING ONE. Four comments converged on the same gap, and they were right: the plan's §5 rejects a lease protocol and asserts a lost flush/read race costs at worst a wasted rehydration — resting entirely on "absent → hydrated is idempotent". A Lance dataset is a multi-file DIRECTORY, so a repeatable TRANSFER is not idempotence over the DESTINATION. A partial fetch leaves debris a retry merges into; a fetch against a different source version mixes two snapshots; a concurrent reclaim can delete what a hydration just wrote or expose a directory that is neither present nor absent. Two of those are torn reads and one is a silent wrong answer. Rarity was being used where recoverability was claimed. Fix: hydrate aside, publish by rename, retire by rename (knowledge §4a, plan §5a) — a filesystem-atomicity boundary, NOT the protocol the operator walked back. It costs the sweeper a rename and the reader nothing: no guard in a signature, no per-read atomic step, no refcount, no lease, no second gate-off path. Every objection that priced out the lease protocol leaves it untouched, so the operator's scope ruling stands and the hole closes. T9 sharpened to three enumerated interleavings; T9b added for the boundary itself. THE DOC FELL INTO ITS OWN TRAP. §3 teaches "diagnose the feature on the crate that opens the URI", then diagnosed `lancedb`. Verified: this repo opens datasets through `lance` — direct, non-optional, default features, and `lance`'s own default includes `aws`; `lancedb` is optional, `default-features = false`, behind its own flag, and no production path uses it for these reads. Corrected as §3a with probe commands, output and an explicit promotion decision. Same correction applied to docs/DATAFUSION-PERIMETER.md §9a, which had it too. A CATEGORICAL RULE CONDEMNED SHIPPED API. §6's "never open a network-scheme URI" reads as invalidating the public, tested `VersionedGraph::{s3,azure,gcs}` with no replacement offered. Scoped in §6a to the hot zero-copy substrate; the missing hydrating counterpart recorded as ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE rather than implied. Not a deprecation — nothing is removed. THREE OVER-CLAIMS. "Any local path" → a supported, mmap-capable local filesystem (a network mount presents an ordinary directory while changing page-cache, consistency and lock semantics); durability stays an optimization, the filesystem is the correctness axis. "VIABLE for hydration" is a verdict about access SHAPE, not size — ~1 GiB is ~49 s at the observed rate, so boot-viability is a per-dataset question and only the tens-of-MB case was measured; the RAM/NVMe ratios are conventional figures, not measurements taken here. The cost model priced only retained bytes — request/retrieval/transfer/storage-management named, storage-class assumption stated, and the OPERATOR-SET grading of the two defaults explicitly protected from drifting toward FINDING. THE THRASH METRIC COULD NOT DO ITS JOB, three ways, all real: an unattributed numerator (restarts, retries, invalidation, version reloads all counted), an unbounded window that false-alarmed on correct sparse usage, and a `> 1.0` threshold that could not fire when one dataset among many thrashes. Redefined as `eviction_caused_rehydrations` (sweeper-stamped at reclaim, so first hydrations are excluded by construction) over a window bounded to the age floor, threshold `> 0`, ratio retained as severity. T11 rewritten, T11b added to assert the attribution itself. A CONTRADICTION BETWEEN THE PLAN AND ITS OWN SUMMARIES about dirty candidates — resolved, not papered over: the sweep does CLEAN EVICTION ONLY (§9a) and never initiates push-back. It follows from the one hard rule (flush is legal only from `hydrated`): a sweep that pushed first would be manufacturing the precondition for its own destructive step, unattended, on a timer. "Every candidate dirty" becomes a third distinct stop reason; T6b asserts both halves. The two board one-liners that said "pushed back first if dirty" are corrected in place with the correction itself recorded — they are this PR's own unmerged text, not merged history. Board hygiene: EPIPHANIES PREPEND (E-A-REPEATABLE-TRANSFER-IS-NOT-IDEMPOTENCE-OVER-A-MULTI-FILE-DIRECTORY-1 — the two entries it corrects are left as written so the first draft's claims stay visible), ISSUES PREPEND ×1, LATEST_STATE review-round bullet. Still documentation-only: no crate, type, feature or test changed; nothing in any Cargo.toml touched. --- .claude/board/EPIPHANIES.md | 20 ++ .claude/board/INTEGRATION_PLANS.md | 8 +- .claude/board/ISSUES.md | 31 +++ .claude/board/LATEST_STATE.md | 3 +- .claude/knowledge/s3-hydration-lifecycle.md | 200 +++++++++++++++++- .../plans/idle-flush-dataset-eviction-v1.md | 194 ++++++++++++++++- docs/DATAFUSION-PERIMETER.md | 21 +- 7 files changed, 452 insertions(+), 25 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 12ebbfb26..2ab41f757 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,23 @@ +## 2026-08-06 — E-A-REPEATABLE-TRANSFER-IS-NOT-IDEMPOTENCE-OVER-A-MULTI-FILE-DIRECTORY-1 — the PR #901 review round: four corrections, one of them load-bearing on a safety claim + +**Status:** FINDING (review round, PR #901) — corrections to the two entries below, which stay as written so the first draft's claims remain visible. **Confidence:** High for the manifest correction (probe re-run, recorded in the knowledge doc's evidence table) and for the atomicity gap (it follows from Lance datasets being multi-file directories, which is not in dispute); the eviction plan remains a PROPOSAL and nothing in it is measured. Documentation-only. + +**1. "Safe to repeat, it is idempotent" was doing work it could not support — the load-bearing one.** The plan's §5 rejects a lease protocol and asserts the worst case of a lost flush/read race is a wasted rehydration. That assertion rested on `absent → hydrated` being idempotent. **A Lance dataset is a multi-file DIRECTORY**, so a repeatable *transfer* is not idempotence over the *destination*: a partial fetch leaves debris that a retry merges into, a fetch against a different source version mixes two snapshots into one directory, and a concurrent reclaim can delete files a hydration just wrote or expose a reader to a directory that is neither present nor absent. Two of those are torn reads and one is a silent wrong answer — **none is a wasted rehydration**, which is the bar §5 set for itself. The age floor makes them *rare*; rarity is not the property that was claimed. + +**The fix is a filesystem boundary, NOT the protocol the operator walked back — and the distinction is the whole point.** *Hydrate aside, publish by rename, retire by rename*: fetch into a private temporary directory, make it visible with one atomic directory rename, and reclaim by renaming away first and deleting afterwards. A reader resolving the published name sees either absent or a complete dataset, always. This costs the **sweeper** a rename and the **reader** nothing — no guard in a signature, no per-read atomic step, no refcount, no lease, no second gate-off code path. Every objection that priced out the lease protocol in §5 leaves this untouched, so the operator's scope ruling stands *and* the corruption hole closes. Knowledge doc §4a; plan §5a; new tests T9 (three enumerated interleavings, not one convenient one) and T9b. + +**2. The document fell into its own trap, one layer out.** §3 teaches "diagnose the feature on the crate that actually opens the URI", and the first draft then diagnosed **`lancedb`** — while this repository opens datasets through **`lance`**, taken as a direct, non-optional dependency **with default features**, whose own `default` includes the object-store feature. `lancedb` here is `optional = true, default-features = false` behind its own flag and no production path uses it for these reads. So the §3 story was true about the wrong crate *for this consumer*, which is precisely the failure §3 exists to name. Corrected as §3a with the probe commands, their output, and an explicit promotion decision (the manifest rows stay FINDING; the *inference* the first draft drew from them is corrected, not promoted). **The generalizable form: a document that states a diagnostic rule is not thereby immune to it — the defence is running the rule on yourself, mechanically.** + +**3. A categorical rule silently condemned shipped public API.** §6's "never open a network-scheme URI as the runtime store" reads as invalidating `VersionedGraph::{s3, azure, gcs}` — public, tested, and passing the URI straight to the dataset open — while offering no replacement. Scoped in §6a: the rule binds the **hot zero-copy substrate**, where a network scheme *voids* rather than degrades the guarantee; it does not bind occasional non-hot access making no zero-copy claim. The constructors are **usable and unmigrated**, and the missing hydrating counterpart is recorded (`ISSUES.md` `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE`) rather than implied. **A doctrine that condemns an API it does not replace produces silent non-compliance, not migration.** + +**4. Three smaller corrections, each a real over-claim.** (a) *"Any local path satisfies it"* → **a supported, mmap-capable local filesystem**: a network mount or FUSE layer presents an ordinary-looking directory while changing page-cache, consistency and lock semantics — the same trap as putting the network in the read path, wearing a directory's clothes. Durability stays purely an optimization; the *filesystem* is the correctness axis. (b) *"VIABLE for hydration"* is a verdict about the access **shape**, not about any size: at the observed sequential rate ~1 GiB is **~49 s**, so boot-viability is a size question answered per dataset, and only the tens-of-megabytes case was measured. The RAM/NVMe ratios are conventional figures, not measurements taken here. (c) The cost model priced only **retained bytes**; object stores also bill request count, retrieval, transfer and storage-management, and every one of those falls on the side this policy *increases* — so the honest claim is that the retained-byte saving must exceed the rehydration cost, which is the same quantity the thrash criterion already gates on. + +**5. The thrash metric could not do its job, in three separate ways, and all three were real.** `rehydrations / distinct_datasets_accessed > 1.0` (a) counted restarts, retries, manual invalidation and version reloads in the numerator — a metric that cannot attribute cannot falsify; (b) false-alarmed on *correct* sparse usage whenever the window outran the age floor, since a legitimately-evicted dataset starting a new working-set interval later looks identical to thrash; and (c) could not fire at the granularity that matters, because one thrashing dataset among many never reaches 1.0. Redefined as **`eviction_caused_rehydrations` (stamped by the sweeper at reclaim, so first hydrations are excluded by construction) over a window bounded to the age floor, with the threshold at `> 0`** and the ratio retained as a severity measure. New T11b asserts the attribution itself — that a restart, a failed retry and a manual invalidation each fail to increment it — because without that the metric is a hydration counter wearing a thrash label. + +**6. The plan and its own board summaries disagreed about dirty candidates** — the summaries said push-back-then-evict, the plan's open item said skip. Different data-integrity contracts, so it is now **decided: the sweep does CLEAN EVICTION ONLY** (plan §9a). A dirty candidate is skipped and reported. The reasoning is already in the material: the one hard rule is *flush is legal only from `hydrated`*, and a sweep that pushed first would be **manufacturing the precondition for its own destructive step, unattended, on a timer** — inverting a safety check into a workflow. Consequence made observable: *"every candidate was dirty"* is a **third** distinct stop reason, never collapsed into "none old enough" or "every candidate in use", because a deployment stuck permanently over budget on dirty data is exactly where a human should enter the loop. T6b asserts both halves. + +**Triage note (method).** 19 review comments across the two PRs; several were the same finding reached from different angles, which is signal rather than noise — the four comments converging on the atomicity gap are what made it obvious the claim, not the wording, was wrong. One class was **not** accepted: the request to obtain a sanctioned home for the sibling PR's carving before publishing is not an oversight but an open operator question already recorded. **Comment count is not defect count, and neither is it a compliance quota.** + ## 2026-08-06 — E-OBJECT-STORE-HYDRATES-IT-DOES-NOT-STORE-1 — the object store is the hydration path; the local filesystem is the store; the volume only decides whether hydration repeats **Status:** FINDING (mechanism) + one reported single-observation measurement set. **Confidence:** High for the layer split and the feature-gate facts (manifest-verified in session); Medium for the endpoint numbers (one provider, one region, one point in time, not re-run); the flush/rehydrate lifecycle is labelled CONJECTURE in the doc with its falsifier stated. Documentation-only — no Rust changed. diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 280eca4c1..c718e4afb 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -2,8 +2,12 @@ **Plan:** `.claude/plans/idle-flush-dataset-eviction-v1.md` Feature-gated (**off by default**) eviction of a Lance dataset's LOCAL copy after -an idle period, pushed back to the object store first if dirty, rehydrating on -next access. **Purpose is cost smoothing, not capacity** (operator framing): the +an idle period, **skipped (never pushed back) if dirty**, rehydrating on +next access. **The sweep performs CLEAN EVICTION ONLY** — push-back is a separate +operation with its own trigger, and a background sweep must not manufacture the +precondition for its own destructive step (plan §9a; the first draft of this entry +said "pushed back first if dirty", which contradicted the plan's own open item and +was corrected in the PR #901 review round). **Purpose is cost smoothing, not capacity** (operator framing): the win is the shape of the bill — local disk bills continuously for capacity provisioned, object storage for what is kept — so the plan explicitly does NOT justify itself with "otherwise you run out of disk", and never fails an diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 2fc68c83f..48e00bfbb 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,5 +1,36 @@ # Issues Log — Open + Resolved (double-entry, append-only) +## ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE (2026-08-06) — OPEN, SURFACED BY REVIEW ON PR #901 + +`crates/lance-graph/src/graph/versioned.rs` ships `VersionedGraph::{s3, azure, gcs}`. +Each stores a network URI as `base_path`, and the read methods pass it straight to +the dataset open — so a caller using them opens the object store **as the runtime +store**, which is exactly the pattern `.claude/knowledge/s3-hydration-lifecycle.md` +§2/§6 argues against. The constructors are public and tested; the tests explicitly +preserve the remote paths. + +**Review on PR #901 was right that the first draft of §6 stated its rule +categorically** and thereby declared those flows architecturally invalid while +offering no replacement. That has been scoped (§6a: the rule binds the hot +zero-copy substrate, not occasional non-hot access), which resolves the +*documentation* defect. It does not resolve the underlying gap. + +**The gap:** there is no hydrating counterpart. A caller who *does* have a +zero-copy story and *does* hold a remote URI has nowhere to go except hand-rolling +the fetch. Something of the shape `hydrate_from(remote) -> VersionedGraph` (local +path, published per the plan's §5a rename boundary) is the missing piece. + +**Deliberately NOT done in PR #901**, which is documentation-only: adding it is a +new public API on a shipped type, and the eviction plan it would share machinery +with is still a PROPOSAL with an unclosed verification gate (plan §4). Building +the hydration API before that gate closes risks shipping a surface shaped by an +assumption that has not been checked. + +**Not a deprecation.** Nothing here proposes removing the constructors, and they +remain correct for occasional non-hot access. The instruction until the gap closes +is in §6a: **choose by read shape, not by constructor availability.** + + ## ISS-CODEC-RESEARCH-MDCT-ASSERT (2026-08-05) — OPEN, PRE-EXISTING, DISCOVERED NOT CAUSED **The observation.** `cargo +1.97.1 test --manifest-path diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 2b9bb72cc..91e587ae8 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -3,8 +3,9 @@ **Documentation-only.** No crate, type, feature or test changed; nothing in `Cargo.toml` touched. - **NEW knowledge doc `.claude/knowledge/s3-hydration-lifecycle.md`** (`READ BY:` header + per-claim evidence table, every row graded). The three-layer split — **object store hydrates / local directory IS the store / persistent volume only decides whether hydration repeats**. Lance opens a network-scheme URI natively and *that is the trap*: the wrong architecture runs and only degrades, while **deleting the mmap layer** (a remote read lands in a fresh buffer — one copy per read, no page cache), so every zero-copy guarantee under it becomes a claim about copied bytes (`zero-copy-lens-law.md`, one layer down). **Any** local directory satisfies zero-copy — a volume is an optimization on hydration *frequency*, never a correctness requirement. Carries the feature-gate diagnosis (**manifest-verified in session**: `lancedb` `default = []`; `aws` forwards to `lance/aws` + `lance-io/aws` (+ `object_store/aws`); `lance-io` carries `aws` in its OWN defaults, so the layer that opts out is `lancedb` — the reason the diagnosis goes wrong is that the mental model is correct about the wrong crate), the mechanical rule (**scheme-named error = BUILD problem; credential/host/region-named error = CONFIG problem**), the four-state lifecycle (absent/hydrated/dirty/flushed) with **flush legal only from `hydrated`** (the `dirty → flushed` edge is data loss with no error), and one reported single-observation endpoint measurement set graded as ratios-generalize / absolutes-do-not: **NOT viable as swap or as a page-fault backing store; VIABLE for hydration and build caches.** -- **NEW plan `.claude/plans/idle-flush-dataset-eviction-v1.md`** — **PROPOSAL, nothing implemented, nothing measured.** Feature-gated (off by default) idle-flush eviction: a dataset idle past a floor has its local copy dropped (pushed back first if dirty), rehydrating on next access. **Purpose is COST SMOOTHING, not capacity** (operator framing — the win is the shape of the bill; local disk bills continuously for capacity provisioned, object storage for what is kept). **Operator-set default policy:** age **> 3 days** AND footprint **> ~300 MB**, pressure-driven and age-ordered — under budget nothing is ever evicted however stale; over budget the stalest go first. **~300 MB is a SOFT spot**: no operation may ever fail to hold the number, an in-use dataset larger than the whole budget stays resident (**correctness beats the watermark**), and a sweep that reaches no target is a legitimate steady state — which forces the observability requirement that *"no candidate old enough"* and *"every candidate in use"* be distinguishable. Dirty detection = the **Lance dataset version**, never a hash, with an explicit **unclosed verification gate** (a cheap local version read is *assumed*, not checked — a BLOCKER if it fails). **A lease/refcount/guard protocol was CONSIDERED AND REJECTED** as disproportionate at a 3-day floor (operator scope correction): cheap check-then-act, and the bar is **"does not corrupt"** (worst case a wasted rehydration) rather than **"cannot occur"** — recorded rather than left silent so it is not re-added, with the revisit condition named (threshold dropping from days to hours). Acceptance criteria written as **fire/silence pairs** per the P0 rule, including the conjunction-splitting silence tests (under-budget-but-stale, over-budget-but-fresh) that a staleness-only policy would fail. +- **NEW plan `.claude/plans/idle-flush-dataset-eviction-v1.md`** — **PROPOSAL, nothing implemented, nothing measured.** Feature-gated (off by default) idle-flush eviction: a dataset idle past a floor has its local copy dropped — **and is SKIPPED, never pushed back, if dirty** (plan §9a: the sweep does clean eviction only; push-back is a separately-triggered operation, because a background sweep that pushed first would be manufacturing the precondition for its own destructive step, unattended). The first draft of this line said "pushed back first if dirty" and contradicted the plan; corrected in the PR #901 review round. Rehydrates on next access. **Purpose is COST SMOOTHING, not capacity** (operator framing — the win is the shape of the bill; local disk bills continuously for capacity provisioned, object storage for what is kept). **Operator-set default policy:** age **> 3 days** AND footprint **> ~300 MB**, pressure-driven and age-ordered — under budget nothing is ever evicted however stale; over budget the stalest go first. **~300 MB is a SOFT spot**: no operation may ever fail to hold the number, an in-use dataset larger than the whole budget stays resident (**correctness beats the watermark**), and a sweep that reaches no target is a legitimate steady state — which forces the observability requirement that *"no candidate old enough"*, *"every candidate in use"* and *"every candidate dirty"* be distinguishable (the third reason added with §9a). Dirty detection = the **Lance dataset version**, never a hash, with an explicit **unclosed verification gate** (a cheap local version read is *assumed*, not checked — a BLOCKER if it fails). **A lease/refcount/guard protocol was CONSIDERED AND REJECTED** as disproportionate at a 3-day floor (operator scope correction): cheap check-then-act, and the bar is **"does not corrupt"** (worst case a wasted rehydration) rather than **"cannot occur"** — recorded rather than left silent so it is not re-added, with the revisit condition named (threshold dropping from days to hours). Acceptance criteria written as **fire/silence pairs** per the P0 rule, including the conjunction-splitting silence tests (under-budget-but-stale, over-budget-but-fresh) that a staleness-only policy would fail. - **`docs/DATAFUSION-PERIMETER.md` §9a (NEW section)** — cross-reference: the object-store provider is the *same class of fact* that document already catalogues, one crate over (a capability behind a default-off feature, diagnosed at the wrong layer). +- **Review round (PR #901), all corrections additive.** 19 review comments across this PR and its sibling; several were one finding reached from different angles. **Accepted + fixed:** (1) *"safe to repeat, it is idempotent"* did not support the safety claim resting on it — a Lance dataset is a multi-file **directory**, so the fix is *hydrate aside / publish by rename / retire by rename* (knowledge §4a, plan §5a), a **filesystem-atomicity boundary that costs the reader nothing** and therefore leaves the operator's rejection of a lease protocol intact; (2) the doc fell into its own §3 trap — this repo opens datasets through **`lance`** (direct, non-optional, default features, `aws` ON), not `lancedb` (optional, `default-features = false`), corrected as §3a with a probe record; (3) §6's categorical rule silently condemned shipped `VersionedGraph::{s3,azure,gcs}` — scoped in §6a to the hot zero-copy substrate, gap recorded as `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE`; (4) *"any local path"* → **mmap-capable local filesystem**; (5) boot-viability is a **size** claim (~1 GiB ≈ 49 s at the observed rate), scoped to the measured tens-of-MB case; (6) the cost model priced only retained bytes — request/retrieval/transfer/storage-management named, with the storage-class assumption stated; (7) the thrash metric was unusable in three ways (unattributed numerator, unbounded window, threshold that could not fire at single-dataset granularity) — redefined as `eviction_caused_rehydrations` over an age-floor-bounded window at `> 0`, plus **T11b** asserting the attribution itself; (8) the plan and its board summaries disagreed on dirty candidates — **decided: clean eviction only, the sweep never initiates push-back** (§9a), with *"every candidate dirty"* as a third distinct stop reason and **T6b** asserting both halves. New acceptance tests: T6b, T9b, T11b; T9 sharpened to three enumerated interleavings. **Not accepted:** the sibling PR's carving-sanction request — an open operator question already recorded, not an oversight. - Board: `.claude/board/EPIPHANIES.md` PREPEND ×2 (`E-OBJECT-STORE-HYDRATES-IT-DOES-NOT-STORE-1`, `E-IDLE-FLUSH-IS-COST-SMOOTHING-NOT-CAPACITY-AND-THE-3-DAY-FLOOR-PRICES-OUT-A-LEASE-PROTOCOL-1`); `.claude/board/INTEGRATION_PLANS.md` PREPEND ×1. ## 2026-08-05 — lance 9 / lancedb 0.33 / DataFusion 54 / Rust 1.97.1 — the ecosystem bump, MEASURED then LANDED across 9 repos diff --git a/.claude/knowledge/s3-hydration-lifecycle.md b/.claude/knowledge/s3-hydration-lifecycle.md index 259092807..b7d86e45b 100644 --- a/.claude/knowledge/s3-hydration-lifecycle.md +++ b/.claude/knowledge/s3-hydration-lifecycle.md @@ -31,26 +31,71 @@ | Any local directory satisfies zero-copy; a persistent volume is not a correctness requirement | **FINDING** (mechanism) | The store's requirement is a filesystem path, not a durable one. Persistence changes *how often you hydrate*, never *whether reads are zero-copy*. | | The endpoint characteristics in §5 | **reported measurement, not re-verified in this session** | Measured once, against one S3-compatible endpoint, from one region, at one time. Provider- and region-dependent; treat the *ratios* as the finding and the absolute numbers as a single observation. | | The flush/rehydrate lifecycle in §4 is the right shape for large single-use datasets | **CONJECTURE** | Argued from §5's ratios, not from a deployed instance. Falsifier stated inline at §4. **No probe has run.** | +| **This repo's own object-store path goes through `lance` (default features, `aws` ON), not through `lancedb`** — so §3's `lancedb` gate is a *consumer-side* trap, not this crate's | **FINDING** — probe run, recorded in §3a | Raised by review on PR #901 and verified against the manifests + call sites; the probe command, its output and the promotion decision are in §3a. **This corrects the first draft of §3**, which stated the `lancedb` gate as if it were the gate on this repo's `s3://` reads. | Nothing below is promoted past its row here. +**Probe record for the two manifest rows above** (re-run any time; all three are +read-only and take seconds): + +```bash +# P1 — the feature declarations, read from the vendored manifests: +sed -n '/^\[features\]/,/^\[[a-z]/p' ~/.cargo/registry/src/*/lancedb-*/Cargo.toml +sed -n '/^\[features\]/,/^\[[a-z]/p' ~/.cargo/registry/src/*/lance-io-*/Cargo.toml +# P2 — which crate THIS repo opens datasets with, and how it is configured: +grep -nE '^(lance|lancedb) *=' crates/lance-graph/Cargo.toml +``` + +**Result (2026-08-06):** P1 — `lancedb` `default = []`, `aws = [...]`; `lance-io` +`default = ["aws", "azure", "gcp"]`. P2 — `lance` is a **direct, non-optional** +dependency taken **with default features**, and `lance`'s own +`default` includes `aws`; `lancedb` is `optional = true, default-features = +false` behind a separate feature. **Promotion decision:** the manifest rows stay +FINDING; the *inference* drawn from them in the first draft of §3 is **corrected** +by §3a rather than promoted. + ## 1. The three layers | layer | its ONE job | if it is absent | |---|---|---| | **object store** (S3-compatible) | **hydration source** — durable, versioned, shared between machines and between builds | fall back to whatever secondary source the consumer already has; the store still works, the dataset just has to come from somewhere else | -| **local directory** | **THE Lance store** — the path the process opens; zero-copy mmap reads, page cache, no network in the read path | **no fallback — always required.** But *any* local path satisfies it | +| **local directory** | **THE Lance store** — the path the process opens; zero-copy mmap reads, page cache, no network in the read path | **no fallback — always required.** Any path on a **supported, mmap-capable local filesystem** satisfies it (see the qualification below) | | **persistent volume** | decides **which** local directory — chosen only because it survives redeploys | hydrate on every boot; still correct, merely slower | Read the third row twice. The volume is an **optimization on hydration frequency**, not a component of the store. A design that says "we need a volume or this doesn't work" has mis-assigned a job: what it needs is a directory. +**The qualification on "any local path" (raised by review, PR #901).** What the +store needs is not merely *a path that is not a URI* — it is a filesystem that +actually delivers the mmap and locking semantics the zero-copy read depends on. A +network filesystem (NFS/EFS-class), a FUSE mount, or an overlay with unusual +caching presents a perfectly ordinary local-looking path while changing page-cache +behaviour, consistency, and lock semantics underneath it. Those are the cases +where "it is a local directory, therefore reads are zero-copy" stops being true. + +So the requirement is **a supported, mmap-capable local filesystem**, and the two +axes stay separate: + +- **correctness** — mmap-capable filesystem. Not negotiable, and not satisfied by + path *shape*. +- **hydration frequency** — durability/persistence. Purely an optimization, as + the third row says. + +An ephemeral container path on an ordinary local filesystem satisfies the first +and not the second, which is exactly the intended trade. A network mount may +satisfy the second and *not* the first, which is the trap — and it is the same +trap as §2 one level down, since a network filesystem reintroduces the network +into the read path while still looking like a directory. + ## 2. Why the object store must not be the store — even though the URI works -Lance opens an `s3://` URI natively. That is exactly what makes this trap -easy to fall into: the wrong architecture **runs**, correctly, and only -degrades. +Lance opens an `s3://` URI natively — **given the object-store feature its +provider registration needs** (§3, and §3a for which crate's feature that is in +any given consumer). That is exactly what makes this trap easy to fall into: with +the feature on, the wrong architecture **runs**, correctly, and only degrades. The +feature being off produces a different, louder failure and is §3's subject; this +section is about the case where it works. The reason it is wrong is the same reason the zero-copy law exists one layer down. A local dataset read is a mapped page — the kernel hands you bytes that @@ -68,11 +113,13 @@ into existence. The lens has nothing to borrow from. > the answer is a URI with a network scheme, the zero-copy story below it is > already void, regardless of what any type signature promises. -**Corollary — the local directory has no minimum quality.** An ephemeral -container path is functionally correct: mmap works, the page cache works, the -lens works. Losing that directory on redeploy costs a re-hydration, not a -correctness property. This is why §1's third row is an optimization and not a -requirement. +**Corollary — the local directory has no minimum *durability*.** An ephemeral +container path on an ordinary local filesystem is functionally correct: mmap +works, the page cache works, the lens works. Losing that directory on redeploy +costs a re-hydration, not a correctness property. This is why §1's third row is an +optimization and not a requirement. (It has no minimum durability; it does have a +minimum *filesystem* — see §1's qualification. "No minimum quality", as the first +draft put it, was too strong.) ## 3. The feature gate that costs an hour if you don't know it @@ -103,6 +150,37 @@ error's noun before touching an env var. credential pair — are the config surface for the second class only. They are inert against the first.)* +### 3a. …but diagnose the crate that actually opens YOUR uri — this repo's is `lance` + +**Correction, raised by review on PR #901 and verified (probe record in +§ Evidence status).** §3 above is true *about `lancedb`*, and the first draft +stated it as though it were the gate on this repository's object-store reads. It +is not. + +| | crate | how this repo takes it | `aws` in effect? | +|---|---|---|---| +| what `VersionedGraph::{s3,azure,gcs}` reads through | **`lance`** | direct, **non-optional**, **default features** | **YES** — `lance`'s own `default` includes `aws` | +| the optional SDK surface | `lancedb` | `optional = true`, **`default-features = false`**, behind its own feature | **NO**, unless that feature turns it on | + +So the mechanical rule in §3 stands, but its *first step changes*: **resolve +which crate opens the URI before you look at any manifest.** The §3 story — "the +mental model is correct about the wrong crate" — is exactly the trap this +subsection exists to stop this document from itself falling into, one layer +further out. + +Restated so it is checkable rather than remembered: + +1. Find the call that opens the URI, and name the crate it belongs to. +2. Read **that** crate's feature declarations, and how *this* manifest takes it + (a `default-features = false` on the dependency line overrides the upstream + default, and is easy to miss). +3. Only then decide whether a scheme-named error is a build problem here. + +A consumer that opens datasets through `lancedb` is squarely in §3's case. A +consumer that opens them through `lance` with default features is not — and for +that consumer, a scheme-named error means something else and the §3 diagnosis +would send it down the wrong path. + ## 4. The lifecycle — four states, and what each transition costs The actual operational ask: **large, single-use datasets** (rebake inputs, @@ -121,7 +199,7 @@ Transitions and their costs: | transition | cost | gate | |---|---|---| -| absent → hydrated | one large sequential read (§5: sustained, amortized) + one connect | none; safe to repeat, it is idempotent | +| absent → hydrated | one large sequential read (§5: sustained, amortized) + one connect | safe to repeat **within the boundary below** — not unconditionally | | hydrated → dirty | a local write; free | — | | dirty → hydrated | **push back** — the expensive direction (§5: writes are ~½ read throughput and pay per-fragment object overhead) | must complete before flush, or the divergence is lost | | hydrated → flushed | a local delete; frees disk | **only legal from `hydrated`, never from `dirty`** | @@ -131,6 +209,44 @@ Transitions and their costs: `dirty`.* The state machine exists to make that a checkable condition rather than an assumption. A `dirty → flushed` edge is data loss with no error. +### 4a. The idempotency boundary — `absent → hydrated` is NOT unconditionally safe to repeat + +**Correction, raised by review on PR #901.** The first draft's "safe to repeat, it +is idempotent" was too strong, and the strength was load-bearing: the eviction +plan leans on that word to argue a lost race costs only a wasted rehydration. A +Lance dataset is a **multi-file directory**, so: + +- a hydration that **fails part-way** leaves a partial directory, and a retry + that treats it as a destination rather than as debris merges two attempts; +- a hydration against a **different source version** than a previous one mixes + files from two snapshots into one directory — each file individually valid, the + directory as a whole not a dataset that ever existed; +- a **concurrent** reclaim (§4's `hydrated → flushed`) deleting files from that + same directory can remove what a hydration just wrote, or expose a reader to a + directory that is neither complete nor absent. + +None of these is prevented by the transfer being repeatable. So the property is +**conditional**, and the conditions are the contract: + +> `absent → hydrated` is idempotent **given (a) a pinned source version and (b) a +> destination that is empty and not concurrently mutated.** Outside those two +> conditions it is not idempotent, it is a merge. + +**The mechanism that makes both conditions hold — hydrate aside, publish by +rename.** Fetch into a private temporary directory, then make it visible with a +single atomic directory rename; retire by renaming *away* first and deleting the +renamed copy afterwards. A reader therefore only ever resolves a name that is +either absent or a complete dataset, never one mid-assembly or mid-removal. A +failed hydration leaves only an unpublished temporary directory, which is debris a +sweep can delete without consulting anything. + +This is a **filesystem-atomicity boundary, not a coordination protocol** — it adds +nothing to the read path, takes no lock, and holds no lease. That distinction +matters because the eviction plan explicitly rejects a lease/refcount protocol; +this requirement is compatible with that rejection, and is what makes its "worst +case is a wasted rehydration" claim actually true. See +`.claude/plans/idle-flush-dataset-eviction-v1.md` §5a. + **Why writes are an ops step and never a boot path:** the push-back direction pays object-per-fragment overhead on top of raw throughput (§5), so its cost scales with fragmentation as well as bytes. Hydration is boot-viable; the return @@ -179,6 +295,33 @@ small and the objects are large; it is unusable when the access count is large and the accesses are small.** Every viable/non-viable verdict above is that one sentence applied twice. +### 5a. "Boot-viable" is a claim about a SIZE, and the size is in the table + +**Correction, raised by review on PR #901.** "VIABLE for hydration" above is a +verdict about the *shape* of the access (few, large, sequential). It is **not** a +verdict about any dataset size, and the row it was measured on is in the +tens-of-megabytes range. Carried forward naively it becomes a boot-viability claim +for arbitrary datasets, which the same numbers refute: + +| dataset size | implied transfer at the observed sequential rate | plus connect | boot-viable? | +|---|---|---|---| +| the measured ~35 MB | ~1.7 s | + ~0.7 s | yes — matches the observed ~1.4 s | +| ~256 MB | ~12 s | + ~0.7 s | depends entirely on the boot budget | +| ~1 GiB | **~49 s** | + ~0.7 s | **no**, against any ordinary readiness deadline | + +The linear term dominates the moment the round trip stops being the cost, which is +almost immediately. So the honest statement is: **hydration is the right *shape* +at any size; whether it fits a boot budget is a size question that must be +answered against the actual dataset and the actual budget**, and only the +tens-of-megabytes case has been measured here. + +**What is NOT stated**, and should not be inferred: no RAM or NVMe baseline was +measured in this workspace — the "~2.5 million×" and "~2500×" ratios use +conventional figures for those tiers, not measurements taken here, and they are +order-of-magnitude arguments rather than benchmarks. Neither is the measurement +method recorded (single run vs. best-of-N, cold vs. warm client). Treat the whole +of §5 as one observation with a shape, not as a performance model. + ## 6. Consequences for new work - **Never open a network-scheme URI as the runtime store.** Hydrate to a local @@ -195,6 +338,43 @@ sentence applied twice. step with its own trigger. - **Any flush path must assert `hydrated`, not assume it.** The `dirty → flushed` edge fails silently by construction; only an explicit check catches it. +- **Hydrate aside and publish by rename** (§4a). "Repeatable transfer" is not + idempotence over a multi-file directory. + +### 6a. Scope — what this doctrine binds, and the shipped API it does NOT invalidate + +**Correction, raised by review on PR #901.** The first bullet above was written +categorically, and read that way it declares an existing, tested, public API +architecturally invalid while offering no replacement. That is not what it means, +and the scope belongs in the document rather than in the reader's judgement. + +`crates/lance-graph/src/graph/versioned.rs` ships `VersionedGraph::{s3, azure, +gcs}`. Each stores a network URI as `base_path` and the read methods pass it +straight through, so those constructors *are* the pattern §6's first bullet warns +about. They are **not deprecated by this document**, and nothing here removes +them. + +**What the rule binds:** the **hot zero-copy substrate** — any read path whose +correctness story includes mapped bytes, a lens over them, or a page-cache +assumption. There, opening a network-scheme URI does not degrade the guarantee, it +**voids** it, and the finding is structural rather than about speed. + +**What the rule does not bind:** occasional, non-hot access where no zero-copy +claim is being made — administrative reads, one-off inspection, a version listing, +a small metadata query. The remote constructors remain the correct tool for those, +and calling one is not a violation. + +**Where that leaves the constructors:** they are **usable and unmigrated**, which +is a known state rather than a silent one. The missing piece is a hydrating +counterpart (`hydrate_from(remote) → local`) so a caller that *does* have a +zero-copy story has somewhere to go; that does not exist yet and this +documentation-only change does not add it. Tracked as +`.claude/board/ISSUES.md` `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE`. + +Until it exists, the honest instruction to a caller is: **choose by read shape, +not by constructor availability.** If your reads are hot and zero-copy, hydrate to +a local path yourself and use `local()`. If they are occasional and you are +claiming nothing about mapped bytes, the remote constructor is fine. ## Cross-refs diff --git a/.claude/plans/idle-flush-dataset-eviction-v1.md b/.claude/plans/idle-flush-dataset-eviction-v1.md index 85b2e175b..db1000cbe 100644 --- a/.claude/plans/idle-flush-dataset-eviction-v1.md +++ b/.claude/plans/idle-flush-dataset-eviction-v1.md @@ -41,6 +41,40 @@ of a standing charge for bytes nobody reads. The population this targets is large single-use material: one-off corpora, rebake inputs, derivations touched once and never again. +### The cost model is incomplete as stated, and the omission has a direction + +**Raised by review on PR #901, and correct.** "Object storage is billed for what +is kept" reduces the object-store side to **bytes at rest**. Real object stores +bill several further dimensions, and every one of them is charged on the side of +the ledger this policy *increases*: + +- **request count** — each hydration is many requests, not one, since a dataset is + a multi-file directory; +- **retrieval** — non-hot storage classes bill per byte retrieved, separately from + storage; +- **data transfer / egress** — charged when the read crosses a boundary the + provider prices; +- **storage-management features** — inventory, versioning, lifecycle rules and + similar, where enabled. + +So the honest form of the argument is **not** "storage-at-rest is cheaper than +provisioned disk" but: *the retained-byte saving must exceed the request + +retrieval + transfer cost of the rehydrations the policy causes.* Both sides scale +with **how often eviction is paid**, which is exactly what §7's thrash criterion +measures — the two sections are the same question asked as money and as latency, +and it is worth noticing that the plan already gates on the right quantity even +though the first draft priced only one term of it. + +**The assumptions this leaves standing, named so they are checkable rather than +implied:** a single deployment and a single storage class, both unnamed and +neither varied; a hot/standard-tier assumption (a colder class trades storage +against a retrieval charge and can invert the conclusion); and no measured +request-count-per-hydration for a representative dataset. **The 3-day and ~300 MB +defaults are not derived from any of this** — §0 already grades them +OPERATOR-SET, and this subsection is the reason that grading must not drift toward +FINDING: the cost model they would have to be derived *from* is not complete +enough to derive them. + Two consequences of the framing, both binding on the design: - **Never fail an operation to hold the number.** The budget is a soft @@ -216,6 +250,66 @@ The bar the implementation must clear is therefore **"does not corrupt"**, not > calculus changes — the race stops being negligible and the protocol question > **reopens**. Tie any such threshold change to a re-read of this section. +## 5a. The atomic publish boundary — what actually makes §5's bar true + +**Added after review on PR #901; four review comments converged on this gap and +they were right.** §5 rejects a lease protocol and asserts the worst case is a +wasted rehydration. That assertion **did not follow from anything §5 stated**, and +the reason is in the knowledge doc's §4a: a Lance dataset is a **multi-file +directory**, so "the reader can just rehydrate" is only a recovery when the reader +can tell *hydrated* from *half-hydrated* — and neither a partial fetch nor an +in-progress reclaim is distinguishable from a complete dataset by opening the +directory and looking. + +Concretely, the interleavings §5 leaves open: + +1. reader resolves the path → sweeper begins deleting → reader opens a directory + that is losing files underneath it; +2. reader's hydration begins → sweeper's delete finishes → sweeper removes files + the hydration just wrote, leaving a directory that is neither; +3. hydration fails mid-transfer → the partial directory is later treated as a + present dataset rather than as debris. + +None of these is a wasted rehydration. (1) and (2) are torn reads; (3) is a silent +wrong answer. The age floor makes them **rare**; it does not make them +**recoverable**, and §5's own bar is *does not corrupt*. + +**The requirement (not a lease, and compatible with §5's rejection):** + +> **Hydrate aside; publish by rename; retire by rename.** A hydration fetches into +> a private temporary directory and becomes visible by **one atomic directory +> rename**. A reclaim renames the published directory **away first**, then deletes +> the renamed copy. A reader resolves the published name **once**, at open, and +> holds what it resolved. + +Consequences, and each is why this is the cheap answer rather than the protocol +§5 walked back: + +- **The read path does not change.** No guard in a signature, no atomic + state-machine step per read, no refcount, no lease, no second gate-off code path + — the objections that priced out the protocol in §5 do not apply, because this + costs the *sweeper* a rename and the *reader* nothing. +- **Every observable state is complete.** The published name is either absent or a + whole dataset. Interleaving (1) and (2) degrade to exactly what §5 claimed: + the reader sees *absent* and hydrates, at worst redundantly. +- **Failure debris is self-identifying.** An unpublished temporary directory was + never visible, so a sweep may delete it with no coordination and no risk of + removing live data — which closes (3) without a partial-state protocol. +- **`hydrated → flushed` gets its barrier for free.** The rename-away IS the + transition; the dirty check happens before it, and after it there is nothing + left to race against. +- **It does not fix multi-process.** A rename is atomic on one filesystem, so two + processes over one directory see consistent *published* state — but the + reclaim-then-rehydrate decision is still uncoordinated and can duplicate work. + §9.3 stays open; the bar it now inherits is the honest one (duplicated work, + not corruption), which is the claim §9.3 previously made without support. + +**Assumption this rests on, stated rather than assumed:** directory rename is +atomic on the filesystem in use. That is true of ordinary local filesystems and is +part of the "supported, mmap-capable local filesystem" requirement (knowledge doc +§1); it is **not** guaranteed on the network-mount cases that requirement already +excludes. Same excluded set, one more reason. + ## 6. Q6 — why a feature gate, and what "off" excludes **ANSWERED** **Off by default.** A consumer with ample local disk must pay *nothing* — and @@ -252,12 +346,43 @@ to function. **The thrash falsifier — the gating acceptance criterion:** > Over a measurement window, compute -> **`rehydrations / distinct_datasets_accessed`**. -> A value **> 1.0** means at least one dataset was evicted and re-fetched -> *within its own working set* — the definition of thrash. +> **`eviction_caused_rehydrations / distinct_datasets_accessed`**. +> A value **> 0** means at least one dataset was evicted and re-fetched — and +> because the window is bounded to the age floor (below), that re-fetch is +> necessarily *within its own working set*, which is the definition of thrash. > A **second hydration of the same dataset inside one age-floor window** is the > same finding at single-dataset granularity, and is the sharper signal. +**The metric's definition is load-bearing, and the first draft's was not +usable — review caught three independent defects in it, all real:** + +1. **The numerator counted the wrong events.** A bare `rehydrations` count also + includes first hydrations, process restarts, failed-hydration retries, manual + invalidation, and version-driven reloads. None of those is eviction, and a + metric that cannot attribute cannot falsify. **Fix: the sweeper stamps an + eviction generation on each dataset it reclaims, and only a hydration that + finds such a stamp increments `eviction_caused_rehydrations`.** First + hydrations are excluded by construction — there is no stamp to find. +2. **An unbounded window produced false alarms.** With a window longer than the + age floor, a dataset can be accessed, go correctly idle past the floor, be + correctly evicted, and later start a *new* working-set interval. That is the + policy working exactly as designed, and it would have registered as thrash. + **Fix: the window is bounded to the age floor.** Inside one floor-length + window, a correctly-evicted dataset cannot legitimately be re-accessed — + re-access within the floor is precisely what "the floor was too short for this + dataset" means. +3. **The `> 1.0` threshold could not fire at the granularity that matters.** With + the numerator correctly restricted to eviction-caused rehydrations, one + thrashing dataset among many gives a ratio well below 1.0 while being exactly + the condition the criterion exists to catch. **Fix: the threshold is `> 0` on + the corrected numerator**, and the ratio is retained as a *severity* measure + (how widespread), not as the trigger. + +**Datasets already resident when the window opens** carry no eviction stamp, so +they contribute to the denominator (they were accessed) and not to the numerator +until this policy actually evicts them. That is the intended asymmetry: the metric +measures what the policy *did*, never what the deployment inherited. + Supporting measurement: **`total_hydration_seconds / total_read_seconds`**. This is the amortization ratio; if hydration time approaches read time, the feature is paying for itself with the thing it was supposed to make cheaper. @@ -284,11 +409,14 @@ naive assertion. Enumerated: | T5 | **Budget inertness** — raising the budget above the current footprint silences a sweep that a lower budget engages; the sweep stops as soon as the footprint is back under | the budget is the trigger, and it is a threshold rather than decoration | | T5b | **The budget is SOFT** — a single in-use dataset larger than the whole budget stays resident and **no operation fails**; the sweep reports *every candidate in use* | correctness beats the watermark; there is no admission control | | T6 | **`dirty → flushed` is REFUSED** — a dirty dataset offered to the flush path is rejected, not silently accepted | the destructive edge is checked, not assumed (the knowledge doc's one rule) | +| T6b | **A dirty candidate is SKIPPED and SAID** — a past-the-floor, over-budget, dirty candidate is not reclaimed, no push-back is attempted, and the sweep reports *dirty* as a **distinct** stop reason (not folded into "in use" or "none old enough"); **and the same candidate IS evicted once clean** | §9a's clean-eviction-only decision, in both directions. Without the silence half the sweep could be refusing everything; without the report half a permanently-stuck deployment is invisible | | T7 | **Dirty is DETECTED** — a mutation makes the version differ and the dataset reads dirty; **and an unmutated dataset reads clean after a full sweep** | the detector discriminates rather than always-firing or never-firing | | T8 | **In-flight read is skipped** — a dataset with a read in flight is not flushed; **and the same dataset IS flushed once the read completes** | the cheap check discriminates in both directions (not always-skip, not never-skip) | -| T9 | **A LOST race does not corrupt** — force the interleaving (read begins mid-flush) and assert the reader gets a **correct, complete dataset** via rehydration. The cost may be a wasted rehydration; the result may **never** be a torn or partial read | §5's actual bar: *does not corrupt*, NOT *cannot occur* — this test deliberately makes the race happen rather than proving it impossible | +| T9 | **A LOST race does not corrupt** — force **each** of §5a's three interleavings (read resolves then reclaim begins; hydration overlaps a reclaim; hydration fails mid-transfer) and assert the reader observes the published name as **either absent or a complete dataset**, never a partial one. The cost may be a wasted rehydration; the result may **never** be a torn or partial read | §5's actual bar: *does not corrupt*, NOT *cannot occur* — this test deliberately makes the race happen rather than proving it impossible. Enumerating the three interleavings is what stops it degenerating into a single easy one | +| T9b | **Publish and retire are atomic at the name** — a reader that resolves the published name mid-reclaim holds a complete dataset; and a failed hydration leaves **no** visible published directory (only unpublished debris a sweep may delete) | §5a's boundary is asserted rather than assumed — without this, T9 could pass by timing luck | | T10 | **Rehydrate is byte-identical** — flush → rehydrate → read equals the pre-flush read | the round trip is lossless | -| T11 | **Thrash detector CAN fire** — a synthetic access pattern designed to thrash produces a ratio > 1.0; **and a well-behaved pattern produces ≤ 1.0** | §7's metric discriminates — a detector that fires on everything carries no information | +| T11 | **Thrash detector CAN fire** — inside one age-floor window, a synthetic pattern that re-accesses an evicted dataset produces `eviction_caused_rehydrations > 0`; **and a well-behaved sparse pattern — accessed, correctly idled past the floor, evicted, then re-accessed in a LATER window — produces `0`** | §7's corrected metric discriminates. The silence half is deliberately the case the first draft's definition got wrong: normal sparse usage must NOT read as thrash | +| T11b | **Attribution is real** — a restart, a failed-hydration retry and a manual invalidation each produce a hydration that does **not** increment `eviction_caused_rehydrations` | the numerator counts eviction, not hydration; without this the metric is a hydration counter wearing a thrash label | | T12 | **Gate-off is inert** — with the feature off, no sweep runs, no accounting is kept, and the read path is unchanged (per §6, either one code path or a proven-equivalent second one) | the gate costs nothing when off | **T2a/T2b, T7, T8 and T11 are the ones that matter most** — each is a paired @@ -316,11 +444,57 @@ rule forbids — implied by the code, falsifiable by nothing. 4. **Partial hydration.** Whether a subset (fragment / column range) can be hydrated instead of a whole dataset is unexplored. It would change the size term of the eviction key, so it is a policy question, not just an I/O one. -5. **Interaction with the push-back direction.** `dirty → hydrated` is the - expensive direction and is currently an operational step. Whether a sweep - may *initiate* a push-back (making eviction possible) or only skip dirty - candidates is undecided; the plan currently assumes **skip**, which is the - conservative choice and possibly the wrong one for the target workload. +5. ~~**Interaction with the push-back direction.**~~ **DECIDED after review on + PR #901 — see §9a below.** It was recorded here as undecided while the board + summaries described push-back as part of eviction; that contradiction is + resolved in favour of **skip**, and the open item is now only the narrower + question of whether a *separately triggered* push-back should exist. + +### 9a. The sweep NEVER initiates push-back — clean eviction only + +**Review on PR #901 found the plan and its two board summaries disagreeing** about +whether a dirty candidate is skipped or pushed back first. Those are different +data-integrity contracts, so the disagreement is resolved here rather than left to +the reader. + +**The decision: the sweep performs CLEAN EVICTION ONLY.** A dirty candidate is +**skipped** — reported, never reclaimed, never pushed. Push-back is a **separate +operation with its own trigger**, and the sweep does not invoke it. + +Two mechanisms, deliberately not one: + +| operation | what it does | who triggers it | +|---|---|---| +| **clean eviction** | reclaims the local copy of a dataset that is *identical to the object store* | the watermark sweep (§3) | +| **push-back** (`dirty → hydrated`) | uploads a diverged local copy | an operator/operational step — **never the sweep** | + +Why this and not the other choice — it follows from what the plan already says +rather than being a new preference: + +- The knowledge doc's §4 rule is *flush is legal only from `hydrated`*. A sweep + that pushes first would be **manufacturing** the precondition for its own + destructive step, on a background timer, unattended. That inverts a safety + check into a workflow. +- §4 of that same doc establishes push-back as the expensive, + fragmentation-sensitive direction and rules it "an ops step, never a boot path". + A background sweep is not a boot path, but it *is* unattended — which is the + property that made it an ops step in the first place. +- A failed push mid-sweep leaves the only copy of diverged data in an ambiguous + state, with nobody watching. Skipping has no such failure mode: the worst case + is that the sweep reaches no target, which §2 already establishes as a + **legitimate reported steady state**. + +**Consequence to make observable:** a deployment whose footprint is dominated by +*dirty* datasets will sit permanently over budget with the sweep unable to act. §2 +already requires the sweep to say *why* it stopped; **"every candidate was dirty" +is a third distinct reason** alongside "none old enough" and "every candidate in +use", and it must not be collapsed into either. It is the signal that a push-back +is owed — which is the correct place for a human to enter the loop. + +**Remaining open question** (narrowed from the original item 5): whether a +separately-triggered push-back operation should exist in this feature at all, or +whether it belongs entirely outside it. Undecided; it does not block the sweep, +which never calls it either way. ## Cross-refs diff --git a/docs/DATAFUSION-PERIMETER.md b/docs/DATAFUSION-PERIMETER.md index 50931c58e..621ec2ab6 100644 --- a/docs/DATAFUSION-PERIMETER.md +++ b/docs/DATAFUSION-PERIMETER.md @@ -346,9 +346,26 @@ spent on code that is not in the binary. **An error naming a *scheme* is a build problem; an error naming a *credential/host/region* is a config problem.** Read the error's noun before touching configuration. +**Correction (review round, PR #901) — and it is the §9 lesson landing on this +section itself.** The paragraphs above are true *about `lancedb`*, and the first +draft presented them as the gate on **this repository's** object-store reads. They +are not. `crates/lance-graph/Cargo.toml` takes `lance` as a **direct, +non-optional** dependency **with default features**, and `lance`'s own `default` +includes `aws`; `lancedb` is `optional = true, default-features = false` behind a +separate feature, and no production path here opens datasets through it. So for +this crate the provider **is** compiled in, and a scheme-named error would mean +something else entirely. + +The rule survives; its **first step** was missing: *resolve which crate opens the +URI, then read that crate's features — and check how this manifest takes it, since +a `default-features = false` on the dependency line overrides the upstream +default.* Diagnosing the right facts about the wrong crate is the §9 failure mode, +and this section had it. + Full treatment — the three-layer hydration model, why the object store must -not be the runtime store, and the flush/rehydrate lifecycle: -`.claude/knowledge/s3-hydration-lifecycle.md`. +not be the runtime store, the flush/rehydrate lifecycle, and the probe record +behind the correction above: +`.claude/knowledge/s3-hydration-lifecycle.md` (§3a for the crate-resolution step). ## 10. Open questions, in decision order