[python] Add TableRead.read(splits) returning Arrow batches#428
Merged
Conversation
QuakeWang
reviewed
Jul 1, 2026
Projection validation only happens in ReadBuilder::new_read(); the empty-splits early return skipped it, so an invalid projection returned [] for empty splits but raised for non-empty splits. Construct the read (and thus validate) before the empty check so behavior is consistent regardless of split count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Third PR of exposing PyPaimon's DataFrame read path to Rust (follows #415 scan/plan/splits and #419
with_filter). Refs #413.Adds the data-reading step both prior PRs deferred:
ReadBuilder.new_read().read(splits) → List[pyarrow.RecordBatch].Splits are picklable (from #415), so they can be planned in one process and read in another — groundwork for Ray distributed reads.
Notes
apply_read_config(projection/limit/filter) sonew_scan().plan()andnew_read().read()apply the same pushdown consistently. Pure refactor, no behavior change to scan.new_read()snapshots the builder's projection/limit/filter, so mutating the builder afterward does not affect an already-created read.read()collects the full Arrow stream into aList[RecordBatch].Splitelement raisesTypeError; empty splits return[]without entering the runtime. Off-GIL read viapy.detach+ runtimeblock_on, matching the existingplan()pattern.Out of scope (later PRs): pypaimon-side wiring.
Tests
bindings/python/tests/test_read.py— 7 new tests: expected rows, empty splits, non-SplitTypeError, pickled-split round-trip, projection applied, config-snapshot isolation, and awith_filterread smoke test. Full file 33/33 passing;cargo clippyclean.