Skip to content

[python] Add TableRead.read(splits) returning Arrow batches#428

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JunRuiLee:feat/py-read-builder-pr3
Jul 1, 2026
Merged

[python] Add TableRead.read(splits) returning Arrow batches#428
JingsongLi merged 3 commits into
apache:mainfrom
JunRuiLee:feat/py-read-builder-pr3

Conversation

@JunRuiLee

Copy link
Copy Markdown
Contributor

Purpose

Third PR of exposing PyPaimon's DataFrame read path to Rust (follows #415 scan/plan/splits and #419 with_filter). Refs #413.

Adds the data-reading step both prior PRs deferred: ReadBuilder.new_read().read(splits) → List[pyarrow.RecordBatch].

b = table.new_read_builder().with_projection(["id"])
splits = b.new_scan().plan().splits()
batches = b.new_read().read(splits)   # -> List[pyarrow.RecordBatch]

Splits are picklable (from #415), so they can be planned in one process and read in another — groundwork for Ray distributed reads.

Notes

  • Shared config: extracts apply_read_config (projection/limit/filter) so new_scan().plan() and new_read().read() apply the same pushdown consistently. Pure refactor, no behavior change to scan.
  • Config snapshot: new_read() snapshots the builder's projection/limit/filter, so mutating the builder afterward does not affect an already-created read.
  • Projection now applies: projection was a planning no-op in [python] Expose DataFrame-style scan/split planning API #415; it takes effect here on the read path.
  • Eager collection: read() collects the full Arrow stream into a List[RecordBatch].
  • Errors: a non-iterable argument or a non-Split element raises TypeError; empty splits return [] without entering the runtime. Off-GIL read via py.detach + runtime block_on, matching the existing plan() pattern.

Out of scope (later PRs): pypaimon-side wiring.

Tests

bindings/python/tests/test_read.py — 7 new tests: expected rows, empty splits, non-Split TypeError, pickled-split round-trip, projection applied, config-snapshot isolation, and a with_filter read smoke test. Full file 33/33 passing; cargo clippy clean.

Comment thread bindings/python/src/read.rs Outdated
Projection validation only happens in ReadBuilder::new_read(); the
empty-splits early return skipped it, so an invalid projection returned
[] for empty splits but raised for non-empty splits. Construct the read
(and thus validate) before the empty check so behavior is consistent
regardless of split count.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit f091bc2 into apache:main Jul 1, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants