feat(spark): datafusion-spark-bridge Rust SDK (2/6)

[timsaucer]

Stacked PR series (6 parts) — splitting the Spark DataSource V2 connector.
All six target main. They build on each other, so review and merge in order — until the earlier parts merge, this PR's diff includes their changes too.

1. build: Cargo workspace + native-common extraction (1/6) #104 — Cargo workspace + native-common extraction
   ➤ 2. feat(spark): datafusion-spark-bridge Rust SDK (2/6) #105 — datafusion-spark-bridge Rust SDK
2. feat(spark): connector Java SPI module (3/6) #106 — Spark connector Java SPI
3. feat(spark): DataSource V2 connector, Scala (4/6) #107 — Spark DataSource V2 connector (Scala)
4. feat(spark): bridge scaffold generator (5/6) #108 — Bridge scaffold generator
5. feat(examples): end-to-end Spark bridge demo (6/6) #109 — End-to-end examples

Purpose

Add the datafusion-spark-bridge Rust SDK — the foundation a domain connector ("bridge") builds on. Its export_bridge! macro generates the JNI entry points the JVM side calls (schema probe, scan create/execute/close); it also bundles the options decoder, scan glue, and the Arrow type-widening layer for Spark type compatibility.

Self-contained; depends only on native-common.

🤖 Generated with Claude Code

[timsaucer]

Repo-layout docs follow-up (from #104 review): PR #104 trimmed docs/source/contributor-guide/development.md to only what the foundation ships (native, native-common). When this PR lands spark/bridge, re-add its slice of that doc:

• add spark/bridge to the Cargo.toml workspace-members bullet;
• add a spark/bridge/ bullet describing the datafusion-spark-bridge Rust SDK (widening, scan machinery, export_bridge!).

[timsaucer]

Fold-in from #104 review — avro feature consistency

native-common/src/errors.rs gates the DataFusionError::AvroError(_) => IoException arm behind native-common's own avro feature (avro = ["datafusion/avro"]). That feature is independent of whether the unified datafusion in a given build graph actually has avro on.

In this PR, spark/bridge/Cargo.toml pulls:

datafusion = { workspace = true }                       # no avro
datafusion-jni-common = { path = "../../native-common" } # no avro feature

That's correct today — both are avro-off, so AvroError can't be produced and the missing arm doesn't matter. Each cdylib also statically links its own native-common copy, so native's avro-on graph doesn't leak in.

The footgun: the moment the bridge's datafusion gains avro (directly, or transitively via some dep) without also enabling datafusion-jni-common's avro feature, an Avro read error will fall through native-common's _ => DataFusionException catch-all instead of mapping to IoException — a silent misclassification, no compile error.

To fold in here (or before this lands):

• Keep the invariant explicit: a consumer must enable datafusion-jni-common/avro iff its datafusion has avro.
• Consider a compile-time guard in native-common (e.g. compile_error! when datafusion/avro is detected on but native-common's avro is off) so the two can't drift silently.

Ref: #104 (1/6) review.

[andygrove]

Please rebase this one when you can @timsaucer

[timsaucer]

Thanks @andygrove . I’m going to leave it in draft for a bit while I review Dewey’s counter proposal.