Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d1c4a9b
WIP using rerun specific table options
timsaucer Jun 10, 2026
baa9ed3
WIP on making table providers spark data sources with a working example
timsaucer Jun 10, 2026
e7b5848
feat(spark): add Spark DataSource V2 connector + pyspark FFI demo
timsaucer Jun 10, 2026
7ee7c9d
refactor(build): consolidate Rust crates into a Cargo workspace
timsaucer Jun 10, 2026
8db9d4a
feat(examples): pass user options through FFI table provider demo
timsaucer Jun 10, 2026
e8c70a9
update examples to build after last commit
timsaucer Jun 10, 2026
088474d
feat(spark): per-partition payload + preferred locations in FFI factory
timsaucer Jun 10, 2026
f7d3972
docs(examples): update SPARK_INTEGRATION for PartitionInfo + per-slic…
timsaucer Jun 10, 2026
daa3ba5
feat(spark): SupportsReportPartitioning via optional reportPartitioni…
timsaucer Jun 10, 2026
c926d3d
feat(spark): shared-scan mode with per-executor provider cache
timsaucer Jun 11, 2026
1cffd93
refactor(spark): fold scan planning/execution into connector cdylib
timsaucer Jun 11, 2026
1f73a6f
docs: rewrite examples README and move Spark guide into spark/
timsaucer Jun 11, 2026
e9f3f61
feat(spark): add datafusion-spark-bridge SDK for static bridges
timsaucer Jun 11, 2026
dc909ce
feat(spark): ScanBackend dispatch + one-method-minimum factory
timsaucer Jun 11, 2026
45c9613
feat(spark): reusable native loader + bridge packaging recipe
timsaucer Jun 11, 2026
cc35958
feat(spark): bridge scaffold generator
timsaucer Jun 11, 2026
9e47f0e
refactor(spark)!: remove the FFI provider path
timsaucer Jun 11, 2026
13bca92
docs: scrub dual-path language after FFI removal
timsaucer Jun 11, 2026
a1815a8
refactor(spark): rename optionsProtoBytes, fix stale createProvider docs
timsaucer Jun 11, 2026
4b99734
refactor(spark): rename LegacyMode to PerPartitionMode
timsaucer Jun 11, 2026
827068c
refactor(spark): move bridge scaffold from dev/ to spark/scaffold/
timsaucer Jun 12, 2026
0cc474a
add support for fixed sized list widening
timsaucer Jun 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Keep Cargo's workspace output out of `target/` so `mvn clean` (which deletes
# the root `target/`) does not nuke the Rust build cache.
[build]
target-dir = "rust-target"
4 changes: 2 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ jobs:
path: |
~/.cargo/registry
~/.cargo/git
native/target
key: ${{ runner.os }}-cargo-${{ hashFiles('native/Cargo.lock') }}
rust-target
key: ${{ runner.os }}-cargo-${{ hashFiles('Cargo.lock') }}
restore-keys: ${{ runner.os }}-cargo-

- name: Build native and run tests
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
run: ./mvnw -q spotless:check

- name: Check Rust formatting
run: cd native && cargo fmt --all -- --check
run: cargo fmt --all -- --check

clippy:
name: Clippy
Expand All @@ -81,9 +81,9 @@ jobs:
path: |
~/.cargo/registry
~/.cargo/git
native/target
key: ${{ runner.os }}-clippy-${{ hashFiles('native/Cargo.lock') }}
rust-target
key: ${{ runner.os }}-clippy-${{ hashFiles('Cargo.lock') }}
restore-keys: ${{ runner.os }}-clippy-

- name: Run clippy
run: cd native && cargo clippy --all-targets -- -D warnings
run: cargo clippy --workspace --all-targets -- -D warnings
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
target/
rust-target/
*.class
.idea/
.vscode/
Expand Down
Loading