perf(mem_wal)!: serve the shard manifest from the store that wrote it

[hamersaw]

Problem

ShardManifestStore::read_latest scanned the version space on every call: a GET for version_hint.json, a HEAD to confirm the hinted version, then batches of parallel HEADs until a whole batch 404s. The loop only exits on a full batch of misses, so the floor is 2 + manifest_scan_batch_size object-store requests even when the hint is exact — and nothing cached the result, so a reader paid it per request.

Profiling a WAL node, this was essentially all of its GET+HEAD traffic. Per fresh-tier read with N L0 generations: 5 + N requests, of which the 5 were manifest probing and only N were data.

Change

Serve the manifest from the store that wrote it, and make the invariant that permits this an enforced one rather than an assumed one.

Serving

A store keeps the manifest it durably wrote as its position and serves that from latest(). Soundness does not rest on holding the claim. Manifest versions are CAS-allocated and gap-free — every writer commits latest.version + 1 under PUT-IF-NOT-EXISTS — so a successful write at version N proves N was the tip: a peer cannot hold N+1 without N existing first.

The two reads now differ by whether they take a position, which is what the names say:

	goes to storage	adopts a position
latest()	only when it has no position	no
refresh_latest()	always	yes

A plain scan deliberately does not adopt, so a reader that polls keeps observing the writer rather than pinning the first manifest it saw. refresh_latest adopts because a claim reads uncached precisely to see another process, and the tip it finds is what its own write then builds on. check_fenced and claim_epoch use it for exactly that reason.

Enforcing

Nothing previously enforced the gap-free invariant the cache rests on: every caller hand-wrote current.version + 1 and nothing checked the result. Given a gap, a cached commit could be acknowledged behind the durable tip.

The invariant is also load-bearing well beyond the cache. find_latest_version stops at the first absent batch, so a gap wider than manifest_scan_batch_size combined with a lost best-effort hint makes even an uncached read misreport the tip.

The check belongs with whoever holds the predecessor, not with the store's position — that position is shared, and a peer's failed CAS clears it, so a commit can find itself judged against an empty position midway through. commit_update therefore validates the closure's output against the manifest the closure received, which is immutable and local. A version the caller did not intend is rejected, not silently corrected; ShardManifest::next_version() is what callers build with.

write keeps only what its own state can judge: a version at or below this store's position is reported as the collision it is, so callers retry.

The tailer

A store can only check contiguity against a position it holds, and WalTailer was the one writer that could never hold one: it claims no epoch, and its store must serve fresh reads for its own position hints — so no single piece of state could be both stable enough to validate against and fresh enough to hint from.

It turns out it never needed to write at all. Its per-entry manifest write maintained wal_entry_position_last_seen purely as a cursor hint for next_position(), whose only callers are tests; replay, the tailer's sole production user, derives the tip from its own read loop. The tailer now tracks the highest position it has read in memory, and publishing that cursor moves to the replay driver, which already holds the epoch, as an ordinary commit_update.

That removes one manifest write per replayed WAL entry and leaves every manifest writer an epoch holder.

Measured

Against a WAL node driving this code:

	before	after
read, N L0 generations	5 + N requests	N
flush (seal into L0)	132	24
compaction pass	35	10
replay, per unflushed WAL entry	8.81	4.18

The replay figure predates the tailer change, which removes a further manifest write per entry.

Writes are unchanged (one conditional PUT per entry).

Breaking changes

• ShardManifestStore::read_latest → latest
• ShardManifestStore::read_latest_uncached → refresh_latest
• ShardManifestStore::write is now crate-private. Callers reach it through commit_update, claim_epoch, or initialize_shard — the three entrances that derive a version from a manifest they just read. It was public from the commit that introduced MemWAL and never acquired a caller.

Downstream commit_update closures need no change: setting version: current.version + 1 is exactly what the check expects.

Tests

cargo test -p lance --lib dataset::mem_wal:: — 619 passed.

Covering the properties this is allowed to break:

• a peer's claim is visible to check_fenced through a held position
• a reader that polls latest() keeps observing the writer, and never adopts a position
• only refresh_latest adopts — reversing that would either pin pollers or reject valid claims
• commit_update recovers from a stale position instead of spinning on the version it lost
• a non-successor write is refused, and the refusal leaves storage untouched
• a taken version reads as a collision, and drops the position so the retry re-reads
• eight concurrent commits on one handle all land, and none is lost
• a tailer read writes no manifest

Note on the commit stack

The first two commits are pre-existing work from the flush-observer branch that has not landed upstream yet; read this PR as its final commit. Rebasing once those merge will reduce it to the single change.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[xuanyu-z]

some agent comments:

Stop swallowing HEAD errors in find_latest_version (manifest.rs:340, if let Ok(true) = result). A transient 503 makes the scan report an older version as the tip, and that answer now gets cached. Two-line fix, and it's the invariant this PR is about.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The manifest scan now fails closed on any non-NotFound HEAD error before adopting a cache position, so a transient storage failure cannot become a stale fencing decision. Missing versions still terminate the scan normally, while the gap-free and collision-retry behavior remains unchanged.

[westonpace]

Some review suggestions from Claude, I think 2 & 3 are valid but non-blocking. I'm not worried about 1 (WAL still too experimental to consider something like this as breaking)

---

I read the full diff, ran the mem_wal manifest suite on the PR head (20/20 pass), and instrumented the store to measure the contended path. CI is green across all 39 checks.

Review of PR #8640

The soundness argument holds up. I traced the fenced-writer paths specifically: a stale position always targets a version the peer already took (contiguity guarantees it), so the CAS collides, invalidate() fires, the retry refreshes, and the fence surfaces. Making the gap-free invariant enforced rather than assumed is the right call — and after the change, every production version: field derives from next_version(); the only remaining hand-written + 1 are in tests. The HEAD-error fix (8caa9df) resolves @xuanyu-z's comment, and the ProxyObjectStore test for it is a good one.

Three things worth addressing:

1. ShardWriter::manifest() changed contract but isn't listed as breaking — write.rs:2512

ShardWriter is publicly exported (pub use write::ShardWriter), so this is embedder-facing API that silently went from "the durable tip" to "our own last commit". The concrete hazard is ShardStatus::Sealed: drop-table 2PC seals from another process, so an embedder polling writer.manifest()?.status to notice a drop-in-flight will now never see it once its own cache is warm. Same for a peer's compaction results. claim_epoch is safe (it uses refresh_latest), and there's no in-repo reader of ShardStatus::Sealed outside manifest.rs — so this lands entirely on Sophon, which the PR body notes keys on that marker. The doc comment covers the general staleness but not this case. Worth listing under Breaking changes, and a ShardWriter::refresh_manifest() would be friendlier than making callers go through manifest_store().refresh_latest().

2. The retry classification is a substring match — manifest.rs:647

let is_version_conflict = e.to_string().contains("already exists");

This is pre-existing, but the PR adds a second producer of that string (the local pre-check) and encodes the coupling as a comment on version_taken rather than fixing it. An error-message edit turns a retryable conflict into a hard commit failure, silently. lance_core::Error already has RetryableCommitConflict { version } and VersionConflict { version } — either lets commit_update use matches!. Since version_taken() is new in this PR, the change is small and in scope. Related: manifest.rs:248 documents "Returns Error::AlreadyExists", which isn't a variant that exists — the function returns Error::io.

3. Under contention on one handle the scan comes back — measured, not speculative

I added a counter to scan_latest and ran concurrent_commits_on_one_handle_all_land: 24 full manifest scans for 8 concurrent commits. Every CAS loss invalidates the shared position, so siblings that were perfectly up to date also drop to a scan. The invalidate() calls are both correct — I tried removing the one in the local pre-check and it got worse (29 scans, more wasted PUTs), because tasks then reach the CAS before invalidating anyway. The fix would be serialization: a tokio::Mutex around commit_update would make each commit hand its position to the next, giving zero scans and zero wasted PUTs. Commits are already serialized at the object store by CAS, so there's no throughput to lose.

Whether this matters depends on whether the flush task, a compaction pass, and the new replay-cursor commit actually overlap on the one shared handle — and the new manifest_store() accessor encourages embedders to funnel everything through it. If they're effectively serialized in production, this is fine as-is; I'd just note the bound somewhere, since MAX_RETRIES = 10 also caps how many concurrent commits can survive on one handle.

Nit: read_latest_serves_the_written_manifest (manifest.rs:813) still carries the old method name.

The tailer change is clean — I confirmed next_position()'s manifest hint still resolves via replay_after_wal_entry_position (advanced by flush) so probe_forward_from doesn't degrade to a full LIST, and that WalAppender::open's own store is test-only, so the internal handles that matter all share one cache.

Want me to post this as a PR review?