bug: MemWAL LSM scan fails with page_lookup.lance Not found after pk-only flush (flush publishes SSTable before pk sidecar exists)

[u70b3]

Summary

On main, an LSM scan over a MemWAL shard can fail hard with:

OSError: Not found: .../base/_mem_wal/<shard_uuid>/<sstable>_gen_<n>/_pk_index/page_lookup.lance

Observed in CI (Python macOS 3.14 ARM): https://github.com/lance-format/lance/actions/runs/34075515757/job/101600711757 — python/python/tests/test_mem_wal.py::test_shard_writer_lsm_scanner_includes_own_sstables. File:line references below are as of 862de0b.

Root cause

1. The put path applies indexes asynchronously: insert_batches_only deliberately skips index updates (rust/lance/src/dataset/mem_wal/memtable.rs:476-505); the pk index is applied by the standalone IndexApplyHandler task (dispatched at freeze — write.rs:1758/1801 — and again at the end of put — write.rs:2904).
2. The flush handler waits for index catch-up only when secondary indexes are configured: write.rs:4085 gates the wait on !self.index_configs.is_empty() (introduced in 085354e / fix(hnsw): make a persisted HNSW cover its rows and reject invalid ones #8834 for HNSW coverage). A pk-only table has empty index_configs, so flush does not wait.
3. MemTableFlusher::flush order: data file → bloom filter → create_pk_index → manifest commit (flush.rs:286-311). create_pk_index returns early when the pk index is still empty (flush.rs:719-722) — no sidecar is written, but the manifest still publishes the SSTable.
4. The LSM scanner unconditionally opens the sidecar for every SSTable (scanner/block_list.rs:192 → open_pk_index); the BTree loader issues a HEAD on page_lookup.lance (object_store.rs:1615), and the single-partition fallback finds no part_* files and rethrows NotFound (lance-index/src/scalar/btree.rs:1761-1774).

Race window: the async index-apply task must be starved longer than [WAL flush + data file write] (a few ms). That starvation is realistic on oversubscribed runners (macOS CI: ~4 vCPU running 6 pytest-xdist workers) and vanishingly rare on idle many-core Linux — which is why only the macOS ARM job hits it.

A deterministic repro was verified locally (not committed): drive a memtable through insert_batch_only so the pk index is never applied, run the flush handler, then full-scan with LsmScanner — the manifest lists the SSTable while _pk_index/page_lookup.lance does not exist, and the scan fails with the byte-identical CI error.

Impact

• Any LSM scan crossing a pk-only flush boundary can fail with NotFound instead of returning rows.
• A subtler multi-batch variant exists: if the pk index is only partially applied at flush time, the sidecar is written with missing rows — cross-generation dedup then misses a mask and superseded/deleted rows can resurrect in scan results.

Proposed fix

Remove the !self.index_configs.is_empty() gate at write.rs:4085 so the flush handler waits for index catch-up whenever the memtable carries an IndexStore (every production memtable binds one, and insert_batches unconditionally advances indexed_count, index.rs:1135-1138). The comment should record that the pk sidecar depends on the same asynchronously maintained pk index.

Regression test: use the deterministic structure above (flush with an unapplied pk index) and assert the sidecar exists / the scan succeeds. This needs a small test hook to control the flush handler's wait point.