Problem
The Python write path classifies sources into "materialized" (wrapped in an in-memory MemTable) vs. "streaming" (one-shot), via _is_materialized / _coerce_reader in python/python/lance/types.py.
Re-readable but not-in-memory sources — LanceDataset, pa.dataset.Dataset, pa.dataset.Scanner — are converted with .to_reader() and fall through to the streaming path. On the Rust side they arrive as a one-shot SendableRecordBatchStream of unknown size, so with conflict_retries > 0 they get drained into a memory/disk spill (spilling_table_provider) on every merge_insert — even though the source could simply be re-scanned. The re-readability is thrown away at the boundary.
Impact is limited to wasted work (an extra buffer/double-store on retrying merges), not correctness, but it's avoidable.
Proposed improvements (coupled)
-
Re-scannable TableProvider. Add a provider that re-runs the source's scan on each .scan() — no spill, no full materialization — for sources that advertise re-readability (Scanner, LanceDataset, pa.dataset.Dataset). This eliminates both the spill and the double-store for those inputs.
-
singledispatch source registry. Refactor the source classification/coercion (_is_materialized + _coerce_reader) into a functools.singledispatch registry (as done in LanceDB), returning the source plus a strategy (materialize / re-scannable / one-shot). This collapses the two hand-maintained type lists into one extensible registry and lets first- and third-party types register their own handling — e.g. a re-scannable registration wouldn't spill.
These are really one refactor: (2) is the mechanism, (1) is its first payoff.
Context
Follow-up from #7368 review (thanks @hamersaw for flagging both the type-list maintainability and the auto-inference of spill-on-retry).
Problem
The Python write path classifies sources into "materialized" (wrapped in an in-memory
MemTable) vs. "streaming" (one-shot), via_is_materialized/_coerce_readerinpython/python/lance/types.py.Re-readable but not-in-memory sources —
LanceDataset,pa.dataset.Dataset,pa.dataset.Scanner— are converted with.to_reader()and fall through to the streaming path. On the Rust side they arrive as a one-shotSendableRecordBatchStreamof unknown size, so withconflict_retries > 0they get drained into a memory/disk spill (spilling_table_provider) on everymerge_insert— even though the source could simply be re-scanned. The re-readability is thrown away at the boundary.Impact is limited to wasted work (an extra buffer/double-store on retrying merges), not correctness, but it's avoidable.
Proposed improvements (coupled)
Re-scannable
TableProvider. Add a provider that re-runs the source's scan on each.scan()— no spill, no full materialization — for sources that advertise re-readability (Scanner,LanceDataset,pa.dataset.Dataset). This eliminates both the spill and the double-store for those inputs.singledispatchsource registry. Refactor the source classification/coercion (_is_materialized+_coerce_reader) into afunctools.singledispatchregistry (as done in LanceDB), returning the source plus a strategy (materialize / re-scannable / one-shot). This collapses the two hand-maintained type lists into one extensible registry and lets first- and third-party types register their own handling — e.g. a re-scannable registration wouldn't spill.These are really one refactor: (2) is the mechanism, (1) is its first payoff.
Context
Follow-up from #7368 review (thanks @hamersaw for flagging both the type-list maintainability and the auto-inference of spill-on-retry).