Skip to content

feat(examples): end-to-end Spark bridge demo (6/6)#109

Draft
timsaucer wants to merge 6 commits into
apache:mainfrom
timsaucer:split/06-examples
Draft

feat(examples): end-to-end Spark bridge demo (6/6)#109
timsaucer wants to merge 6 commits into
apache:mainfrom
timsaucer:split/06-examples

Conversation

@timsaucer

@timsaucer timsaucer commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Stacked PR series (6 parts) — splitting the Spark DataSource V2 connector.
All six target main. They build on each other, so review and merge in order — until the earlier parts merge, this PR's diff includes their changes too.

  1. build: Cargo workspace + native-common extraction (1/6) #104 — Cargo workspace + native-common extraction
  2. feat(spark): datafusion-spark-bridge Rust SDK (2/6) #105 — datafusion-spark-bridge Rust SDK
  3. feat(spark): connector Java SPI module (3/6) #106 — Spark connector Java SPI
  4. feat(spark): DataSource V2 connector, Scala (4/6) #107 — Spark DataSource V2 connector (Scala)
  5. feat(spark): bridge scaffold generator (5/6) #108 — Bridge scaffold generator
    ➤ 6. feat(examples): end-to-end Spark bridge demo (6/6) #109 — End-to-end examples

Purpose

A worked example exercising the whole stack end to end: an examples/native cdylib (demo provider on the SDK, added as a workspace member), the ExampleBridge Java classes implementing the SPI, and a pyspark bridge_demo.py smoke test with READMEs. The reference integration.

🤖 Generated with Claude Code

timsaucer and others added 6 commits June 12, 2026 13:23
@timsaucer @claude
build: consolidate Rust crates into a Cargo workspace + extract nativ…
9f78869 
…e-common

Move the standalone `native` crate into a root Cargo workspace and extract
shared JNI plumbing (error->exception mapping, Tokio runtime singleton,
StreamingReader) into a new `datafusion-jni-common` crate under `native-common/`.
`native/src/errors.rs` moves to `native-common/src/errors.rs`; the nine native
modules now import error/runtime helpers from `datafusion_jni_common`.

Build glue follows: single root `Cargo.lock`, `.cargo/config.toml` redirects
output to `rust-target/`, Makefile/CI/poms updated to build `--workspace` and
target `-p datafusion-jni`. Core javadoc build commands updated to match.

Pure refactor; no behavior change. First of a 6-PR stack splitting the Spark
DataSource V2 connector work.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@timsaucer @claude
feat(spark): add datafusion-spark-bridge SDK for static bridges
28efd7c 
New `spark/bridge` workspace crate providing the `export_bridge!` macro that
generates the six JNI entry points a Spark connector bridge exposes
(providerSchemaIpc, createScan, partitionCount, executeStreamPartition,
executeStream, closeScan). Includes the options decoder, scan planning/execution
glue, and the Arrow type-widening layer (wraps any TableProvider for Spark type
compatibility).

Self-contained SDK with no Java/Scala coupling. Depends only on
datafusion-jni-common. Second of the 6-PR connector stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@timsaucer @claude
feat(spark): add Spark connector Java SPI module
fd9d42b 
Introduce the `spark` Maven module and the pure-Java contracts a bridge
implements: BridgeProviderFactory (no-arg factory + scanBackend()), ScanBackend
(delegates to the bridge's JNI methods), NativeLibraryLoader (cdylib
extraction/loading), OptionsCodec (cross-language options encoder), PartitionInfo
(one entry per Spark task), and ReportedPartitioning (optional shuffle-elision
declaration).

Compiles standalone with no Scala main yet. Includes the two SPI-only tests
(OptionsCodecTest, BridgeProviderFactoryDefaultsTest). Third of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@timsaucer @claude
feat(spark): Spark DataSource V2 connector (Scala)
5db93ae 
The connector implementation on top of the Java SPI and the bridge SDK:
DatafusionSource/Table/Scan/ScanBuilder DSv2 wiring, per-partition columnar
read path (FfiStream + Arrow->Spark batch conversion), V2 predicate pushdown
(SparkPredicateTranslator), shared-scan mode with a per-executor refcounted
cache (SharedScanCache, SharedScanPartitionReader, NativeSharedScanResources,
PinnedSessionConfig), and SupportsReportPartitioning for shuffle elision.

These pieces share the DatafusionScanMode sealed trait and the scan builder, so
they land together. Includes the connector test suite and the module README.
DataSourceRegister SPI file registers DatafusionSource. Fourth of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@timsaucer @claude
feat(spark): bridge scaffold generator
98a6381 
Add `spark/scaffold/new_bridge.py` plus the `bridge-template/` it stamps out: a
standalone Maven+Cargo bridge project wired to the datafusion-spark-bridge SDK —
a Rust cdylib with `export_bridge!` + a demo in-memory provider, the four Java
classes, the DataSourceRegister service file, a shaded-jar pom that bundles the
cdylib, and a pyspark smoke test. Stdlib-only generator.

Standalone tooling. Fifth of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@timsaucer @claude
feat(examples): end-to-end Spark bridge demo
f167e04 
Add a worked example exercising the full stack: examples/native cdylib (a demo
provider built on datafusion-spark-bridge, added as a workspace member), the
ExampleBridge Java classes implementing the SPI, a pyspark bridge_demo.py
end-to-end smoke test, and READMEs. Validates connector + bridge + SDK together.

Last of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 12, 2026
Merged
Draft
Draft
Draft
Draft
@timsaucer timsaucer marked this pull request as draft June 12, 2026 11:41
@timsaucer

timsaucer commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

Repo-layout docs follow-up (from #104 review): #104 trimmed docs/source/contributor-guide/development.md to the foundation crates. This PR adds examples/native, so re-add its doc slice:

  • add examples/native to the Cargo.toml workspace-members bullet;
  • extend the examples/ bullet to mention examples/native/ (the export_bridge! demo cdylib + ExampleBridgeProviderFactory + the pyspark script under examples/python/).

timsaucer added a commit to timsaucer/datafusion-java that referenced this pull request Jun 12, 2026
@timsaucer @claude
fix: scope foundation docs/excludes to what 1/6 ships
3ff4fda 
Address review feedback on the workspace-foundation PR:

- development.md: trim the repo-layout section to the crates this PR
  actually ships (native, native-common). It was forward-referencing
  spark/, spark/bridge, datafusion-spark-bridge, and examples/native --
  none of which exist until later PRs in the stack -- and called the
  member list "three" while listing four. Later PRs (apache#105/apache#106/apache#107/apache#109)
  carry notes to re-add their own slice when those dirs land.

- rat_exclude_files.txt: the Rust lockfile moved to the workspace root,
  so the stale native/Cargo.lock entry left the root Cargo.lock with no
  RAT exclude for the source-tarball check (check-rat-report.py). Point
  it at Cargo.lock.

- native-common: dedupe the panic-payload downcast -- StreamingReader::next
  now calls errors::panic_message instead of repeating the String/&str
  match inline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@timsaucer