Document object-store hydration doctrine + plan idle-flush eviction

.claude/board/INTEGRATION_PLANS.md

@@ -1,3 +1,38 @@
+## 2026-08-06 — idle-flush-dataset-eviction v1 — PROPOSAL (not scheduled; nothing implemented, nothing measured)
+
+**Plan:** `.claude/plans/idle-flush-dataset-eviction-v1.md`
+Feature-gated (**off by default**) eviction of a Lance dataset's LOCAL copy after
+an idle period, **skipped (never pushed back) if dirty**, rehydrating on
+next access. **The sweep performs CLEAN EVICTION ONLY** — push-back is a separate
+operation with its own trigger, and a background sweep must not manufacture the
+precondition for its own destructive step (plan §9a; the first draft of this entry
+said "pushed back first if dirty", which contradicted the plan's own open item and
+was corrected in the PR #901 review round). **Purpose is cost smoothing, not capacity** (operator framing): the
+win is the shape of the bill — local disk bills continuously for capacity
+provisioned, object storage for what is kept — so the plan explicitly does NOT
+justify itself with "otherwise you run out of disk", and never fails an
+operation to hold a number. **Operator-set defaults (heuristic, both config):**
+idle **> 3 days** AND footprint **> ~300 MB**, pressure-driven and age-ordered;
+under budget nothing is ever evicted however stale, over budget the stalest go
+first until back under. **The budget is a SOFT spot** — an in-use dataset larger
+than the whole budget stays resident, a sweep may legitimately reach no target,
+and that state must be *observable* ("no candidate old enough" vs "every
+candidate in use" are different findings). Dirty detection = the Lance dataset
+**version**, never a hash — carrying an **unclosed verification gate** (a cheap
+local version read is assumed, not checked; a BLOCKER if it fails).
+**A lease/refcount/guard-type protocol was CONSIDERED AND REJECTED** as
+disproportionate at a 3-day floor (operator scope correction) — cheap
+check-then-act instead, with the bar set at **"does not corrupt"** (worst case a
+wasted rehydration) rather than "cannot occur"; the rejection is recorded so it
+is not re-added, and the revisit condition (threshold dropping from days to
+hours) is named. Acceptance criteria are **fire/silence pairs** per the P0
+falsifiability rule — including the conjunction-splitting silence tests that a
+staleness-only policy would fail, and a race test deliberately shaped as a
+corruption test rather than an impossibility proof. Five open items, incl.
+multi-process access and whether a sweep may *initiate* push-back (currently
+assumed **skip**). Prerequisite reading:
+`.claude/knowledge/s3-hydration-lifecycle.md`.
+
 ## 2026-08-05 — measure-64k-axes v3 — ACTIVE (the three arms Stage A0 earned; M+O build lane dispatched)
 
 **Plan:** `.claude/plans/measure-64k-axes-v3.md`

.claude/board/ISSUES.md

@@ -1,5 +1,36 @@
 # Issues Log — Open + Resolved (double-entry, append-only)
 
+## ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE (2026-08-06) — OPEN, SURFACED BY REVIEW ON PR #901
+
+`crates/lance-graph/src/graph/versioned.rs` ships `VersionedGraph::{s3, azure, gcs}`.
+Each stores a network URI as `base_path`, and the read methods pass it straight to
+the dataset open — so a caller using them opens the object store **as the runtime
+store**, which is exactly the pattern `.claude/knowledge/s3-hydration-lifecycle.md`
+§2/§6 argues against. The constructors are public and tested; the tests explicitly
+preserve the remote paths.
+
+**Review on PR #901 was right that the first draft of §6 stated its rule
+categorically** and thereby declared those flows architecturally invalid while
+offering no replacement. That has been scoped (§6a: the rule binds the hot
+zero-copy substrate, not occasional non-hot access), which resolves the
+*documentation* defect. It does not resolve the underlying gap.
+
+**The gap:** there is no hydrating counterpart. A caller who *does* have a
+zero-copy story and *does* hold a remote URI has nowhere to go except hand-rolling
+the fetch. Something of the shape `hydrate_from(remote) -> VersionedGraph` (local
+path, published per the plan's §5a rename boundary) is the missing piece.
+
+**Deliberately NOT done in PR #901**, which is documentation-only: adding it is a
+new public API on a shipped type, and the eviction plan it would share machinery
+with is still a PROPOSAL with an unclosed verification gate (plan §4). Building
+the hydration API before that gate closes risks shipping a surface shaped by an
+assumption that has not been checked.
+
+**Not a deprecation.** Nothing here proposes removing the constructors, and they
+remain correct for occasional non-hot access. The instruction until the gap closes
+is in §6a: **choose by read shape, not by constructor availability.**
+
+
 ## ISS-CODEC-RESEARCH-MDCT-ASSERT (2026-08-05) — OPEN, PRE-EXISTING, DISCOVERED NOT CAUSED
 
 **The observation.** `cargo +1.97.1 test --manifest-path

.claude/board/LATEST_STATE.md

@@ -1,3 +1,13 @@
+## 2026-08-06 — branch `claude/s3-hydration-lifecycle` — object-store hydration doctrine + idle-flush plan v1 (docs only, no Rust)
+
+**Documentation-only.** No crate, type, feature or test changed; nothing in `Cargo.toml` touched.
+
+- **NEW knowledge doc `.claude/knowledge/s3-hydration-lifecycle.md`** (`READ BY:` header + per-claim evidence table, every row graded). The three-layer split — **object store hydrates / local directory IS the store / persistent volume only decides whether hydration repeats**. Lance opens a network-scheme URI natively and *that is the trap*: the wrong architecture runs and only degrades, while **deleting the mmap layer** (a remote read lands in a fresh buffer — one copy per read, no page cache), so every zero-copy guarantee under it becomes a claim about copied bytes (`zero-copy-lens-law.md`, one layer down). **Any** local directory satisfies zero-copy — a volume is an optimization on hydration *frequency*, never a correctness requirement. Carries the feature-gate diagnosis (**manifest-verified in session**: `lancedb` `default = []`; `aws` forwards to `lance/aws` + `lance-io/aws` (+ `object_store/aws`); `lance-io` carries `aws` in its OWN defaults, so the layer that opts out is `lancedb` — the reason the diagnosis goes wrong is that the mental model is correct about the wrong crate), the mechanical rule (**scheme-named error = BUILD problem; credential/host/region-named error = CONFIG problem**), the four-state lifecycle (absent/hydrated/dirty/flushed) with **flush legal only from `hydrated`** (the `dirty → flushed` edge is data loss with no error), and one reported single-observation endpoint measurement set graded as ratios-generalize / absolutes-do-not: **NOT viable as swap or as a page-fault backing store; VIABLE for hydration and build caches.**
+- **NEW plan `.claude/plans/idle-flush-dataset-eviction-v1.md`** — **PROPOSAL, nothing implemented, nothing measured.** Feature-gated (off by default) idle-flush eviction: a dataset idle past a floor has its local copy dropped — **and is SKIPPED, never pushed back, if dirty** (plan §9a: the sweep does clean eviction only; push-back is a separately-triggered operation, because a background sweep that pushed first would be manufacturing the precondition for its own destructive step, unattended). The first draft of this line said "pushed back first if dirty" and contradicted the plan; corrected in the PR #901 review round. Rehydrates on next access. **Purpose is COST SMOOTHING, not capacity** (operator framing — the win is the shape of the bill; local disk bills continuously for capacity provisioned, object storage for what is kept). **Operator-set default policy:** age **> 3 days** AND footprint **> ~300 MB**, pressure-driven and age-ordered — under budget nothing is ever evicted however stale; over budget the stalest go first. **~300 MB is a SOFT spot**: no operation may ever fail to hold the number, an in-use dataset larger than the whole budget stays resident (**correctness beats the watermark**), and a sweep that reaches no target is a legitimate steady state — which forces the observability requirement that *"no candidate old enough"*, *"every candidate in use"* and *"every candidate dirty"* be distinguishable (the third reason added with §9a). Dirty detection = the **Lance dataset version**, never a hash, with an explicit **unclosed verification gate** (a cheap local version read is *assumed*, not checked — a BLOCKER if it fails). **A lease/refcount/guard protocol was CONSIDERED AND REJECTED** as disproportionate at a 3-day floor (operator scope correction): cheap check-then-act, and the bar is **"does not corrupt"** (worst case a wasted rehydration) rather than **"cannot occur"** — recorded rather than left silent so it is not re-added, with the revisit condition named (threshold dropping from days to hours). Acceptance criteria written as **fire/silence pairs** per the P0 rule, including the conjunction-splitting silence tests (under-budget-but-stale, over-budget-but-fresh) that a staleness-only policy would fail.
+- **`docs/DATAFUSION-PERIMETER.md` §9a (NEW section)** — cross-reference: the object-store provider is the *same class of fact* that document already catalogues, one crate over (a capability behind a default-off feature, diagnosed at the wrong layer).
+- **Review round (PR #901), all corrections additive.** 19 review comments across this PR and its sibling; several were one finding reached from different angles. **Accepted + fixed:** (1) *"safe to repeat, it is idempotent"* did not support the safety claim resting on it — a Lance dataset is a multi-file **directory**, so the fix is *hydrate aside / publish by rename / retire by rename* (knowledge §4a, plan §5a), a **filesystem-atomicity boundary that costs the reader nothing** and therefore leaves the operator's rejection of a lease protocol intact; (2) the doc fell into its own §3 trap — this repo opens datasets through **`lance`** (direct, non-optional, default features, `aws` ON), not `lancedb` (optional, `default-features = false`), corrected as §3a with a probe record; (3) §6's categorical rule silently condemned shipped `VersionedGraph::{s3,azure,gcs}` — scoped in §6a to the hot zero-copy substrate, gap recorded as `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE`; (4) *"any local path"* → **mmap-capable local filesystem**; (5) boot-viability is a **size** claim (~1 GiB ≈ 49 s at the observed rate), scoped to the measured tens-of-MB case; (6) the cost model priced only retained bytes — request/retrieval/transfer/storage-management named, with the storage-class assumption stated; (7) the thrash metric was unusable in three ways (unattributed numerator, unbounded window, threshold that could not fire at single-dataset granularity) — redefined as `eviction_caused_rehydrations` over an age-floor-bounded window at `> 0`, plus **T11b** asserting the attribution itself; (8) the plan and its board summaries disagreed on dirty candidates — **decided: clean eviction only, the sweep never initiates push-back** (§9a), with *"every candidate dirty"* as a third distinct stop reason and **T6b** asserting both halves. New acceptance tests: T6b, T9b, T11b; T9 sharpened to three enumerated interleavings. **Not accepted:** the sibling PR's carving-sanction request — an open operator question already recorded, not an oversight.
+- Board: `.claude/board/EPIPHANIES.md` PREPEND ×2 (`E-OBJECT-STORE-HYDRATES-IT-DOES-NOT-STORE-1`, `E-IDLE-FLUSH-IS-COST-SMOOTHING-NOT-CAPACITY-AND-THE-3-DAY-FLOOR-PRICES-OUT-A-LEASE-PROTOCOL-1`); `.claude/board/INTEGRATION_PLANS.md` PREPEND ×1.
+
 ## 2026-08-05 — lance 9 / lancedb 0.33 / DataFusion 54 / Rust 1.97.1 — the ecosystem bump, MEASURED then LANDED across 9 repos
 
 **Current pins:** `lance =9.0.0`, `lancedb =0.33.0`, `datafusion 54`, `arrow 58` (unmoved), `object_store 0.13.2` (unmoved), toolchain **1.97.1**. Plan: `.claude/plans/lance9-datafusion54-upgrade-probe-v1.md`.