feat(mem_wal): expose frozen memtable and backpressure stats

[hamersaw]

What

Two additive, read-only accessors on the MemWAL writer.

MemTableStats.frozen_count / .frozen_bytes — memtable_stats reports the active memtable only, so the backpressure threshold (max_unflushed_memtable_bytes, which meters active + frozen) has no observable numerator. Both are read under the read lock memtable_stats already holds.

Deliberately not in_memory_memtable_refs: that calls check_poisoned()? first, so it goes blank exactly when an operator most needs the numbers. memtable_stats is already documented as poison-tolerant for that reason.

The two fields have different denominators on purpose, and the doc comments say so: frozen_bytes drains the moment a flush commits (that is what backpressure meters), while the handle lingers in the read view for frozen_memtable_grace, so frozen_count does not drop to zero with it.

ShardWriter::backpressure_stats() — BackpressureController::stats() and BackpressureStats::snapshot() are already public, but the controller sits inside a private WriterMode variant with no accessor, so the counters are unreachable from outside the writer. Answers in both modes, so a caller never has to know which one it is in.

Why

LanceDB's WAL service polls these to publish four Prometheus metrics it cannot derive today: wal_unflushed_bytes, wal_sealed_memtables, wal_backpressure_waits_total, wal_backpressure_wait_seconds_total. Without them a WAL pod can be throttling, or sitting a hair under an OOM, with nothing on a dashboard to say so.

Tests

• test_memtable_stats_frozen_count_outlives_frozen_bytes — pins the differing-denominator semantics: after wait_for_flush_drain under a long grace, count is non-zero and bytes are zero.
• test_backpressure_stats_reachable_in_both_modes — rstest over enable_memtable true/false, since the point of the accessor is that the private-variant match covers both.

cargo test -p lance --lib -- mem_wal → 561 passed, 1 ignored. cargo fmt --all clean. Clippy is clean on the changed file; -D warnings currently fails elsewhere in the tree on pre-existing single_range_in_vec_init under Rust 1.97.

Scope

rust/lance/src/dataset/mem_wal/write.rs only. No format change, nothing in lance-core, no public signature altered — only new fields on a struct and one new method.

🤖 Generated with Claude Code

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new surface needs to preserve truthful incident-state telemetry: uncommitted frozen memory must remain represented after a failed flush, and an ongoing first backpressure wait must be observable before it completes. A viable revision can separate authoritative live gauges from cumulative completed-wait counters instead of exposing bookkeeping whose failure semantics differ from the advertised metrics.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

frozen_bytes becomes zero after a failed flush even though the same frozen table remains resident and uncommitted. The flush handler unconditionally subtracts its size, then only removes or stamps the retained entry on success. This contradicts “heap bytes still owed to flush” and makes poison-time wal_unflushed_bytes under-report precisely when this snapshot is intended to remain useful.

Keep an authoritative uncommitted/resident byte gauge separate from the backpressure-drain bookkeeping (for example, sum retained entries whose flushed_at_ms is None under the existing read lock), and cover the failed-flush state.

Reproducer

I temporarily extended the existing deterministic test_frozen_retained_after_failed_flush after its failed drain:

let failed_stats = writer_a.memtable_stats().await.unwrap();
assert_eq!(failed_stats.frozen_count, 1);
assert_eq!(failed_stats.frozen_bytes, 0);

cargo test -p lance --lib test_frozen_retained_after_failed_flush passed. The retained table still held rows, but the snapshot observed count 1 and bytes 0; the advertised unflushed-memory contract requires those retained bytes to remain represented.

[lance-gatekeeper]

Fixed in 30102ef: frozen_bytes now sums retained tables whose flushed_at_ms is None, so failed or otherwise uncommitted tables remain represented while successfully flushed grace entries are excluded. The targeted failed-flush test passes on this head. This finding is fixed.

[lance-gatekeeper]

This accessor reports only completed waits: BackpressureStats::record runs only when the loop later observes a value below the threshold. During a first in-progress wait, the snapshot remains 0/0, so it cannot answer the documented “am I throttled” question or expose the stated live-throttling incident.

Track active waiter state or current elapsed wait separately (with cancellation-safe accounting), while keeping completed-wait totals explicitly distinct.

Reproducer

I temporarily ran this focused async test against the observed head:

let controller = Arc::new(BackpressureController::new(config));
let worker = Arc::clone(&controller);
let task = tokio::spawn(async move {
    worker.maybe_apply_backpressure(|| (1000, None)).await.unwrap();
});
tokio::time::sleep(Duration::from_millis(30)).await;
let snapshot = controller.stats().snapshot();
assert_eq!(snapshot.total_count, 0);
assert_eq!(snapshot.total_wait_ms, 0);
task.abort();

cargo test -p lance --lib gate_verify_in_progress_backpressure_is_not_counted passed, confirming that the actively throttled call was invisible to the exported counters.

[lance-gatekeeper]

Fixed in 30102ef: a drop guard now tracks active_count for the full wait and releases it on normal completion or cancellation, so a first in-progress wait is observable without conflating it with completed totals. The focused backpressure tests pass on this head. This finding is fixed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

2 fixed / 0 remain. The revised implementation preserves failed-flush memory in the observable backlog and exposes active backpressure with cancellation-safe accounting.

Please mark this PR with the breaking-change label.