feat: serialize ASOF join plans by Xuanwo · Pull Request #23832 · apache/datafusion

Xuanwo · 2026-07-23T06:12:25Z

Which issue does this PR close?

Part of ASOF join support / Specialize Range Joins #318.
Umbrella PR: feat: support ASOF joins #23738.
Depends on feat: add ASOF join logical semantics #23829 (and transitively feat: support floating-point ASOF equality keys #24375 and feat: add ASOF join physical operator #23828).

Rationale for this change

This is the serialization layer of the ASOF JOIN stack. It gives logical and
physical ASOF plans explicit protobuf representations without coupling wire
format review to SQL or DataFrame APIs.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-proto

What changes are included in this PR?

Add protobuf messages and enum values for logical and physical ASOF joins.
Encode and decode equality keys, ordered match expressions, match direction,
join constraint, and right output indices.
Implement physical serialization through ExecutionPlan::try_to_proto and
AsOfJoinExec::try_from_proto, following the current self-serializing
execution-plan pattern.
Regenerate prost and pbjson sources with the repository generator.
Add logical round trips for all four match directions and a physical
AsOfJoinExec round trip.

Are these changes tested?

Yes:

./datafusion/proto-models/regen.sh
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p datafusion-proto roundtrip_asof_join --all-features

Are there any user-facing changes?

Logical and physical ASOF join plans can be serialized through
datafusion-proto. The additions use new messages and append-only oneof/enum
tags, so existing wire tags are not reused. Generated public Rust enums gain new
variants, however, so downstream exhaustive matches must add arms; this is a
Rust source-compatibility break even though the wire additions are compatible.

This PR should be reviewed and merged after #23829. As its dependencies merge,
this branch will be restacked onto the latest main.

github-actions · 2026-07-23T06:35:57Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [  60.532s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [  58.049s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.650s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 121.618s] datafusion
    Building datafusion-expr v54.1.0 (current)
       Built [  28.977s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.075s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  29.234s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.077s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.239s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  61.139s] datafusion-expr
    Building datafusion-optimizer v54.1.0 (current)
       Built [  27.627s] (current)
     Parsing datafusion-optimizer v54.1.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-optimizer v54.1.0 (baseline)
       Built [  27.763s] (baseline)
     Parsing datafusion-optimizer v54.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.166s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  56.463s] datafusion-optimizer
    Building datafusion-physical-expr v54.1.0 (current)
       Built [  29.050s] (current)
     Parsing datafusion-physical-expr v54.1.0 (current)
      Parsed [   0.048s] (current)
    Building datafusion-physical-expr v54.1.0 (baseline)
       Built [  29.126s] (baseline)
     Parsing datafusion-physical-expr v54.1.0 (baseline)
      Parsed [   0.050s] (baseline)
    Checking datafusion-physical-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.359s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  59.494s] datafusion-physical-expr
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  38.206s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.148s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  38.650s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.147s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.641s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  79.079s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  53.915s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  53.879s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.121s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 109.248s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  25.460s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.128s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.995s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.130s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.805s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1457
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1457
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant ExprType:NormalizeFloatZero in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1653
  variant ExprType:NormalizeFloatZero in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1653

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  53.636s] datafusion-proto-models
    Building datafusion-sql v54.1.0 (current)
       Built [  43.213s] (current)
     Parsing datafusion-sql v54.1.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-sql v54.1.0 (baseline)
       Built [  42.881s] (baseline)
     Parsing datafusion-sql v54.1.0 (baseline)
      Parsed [   0.030s] (baseline)
    Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.247s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  87.690s] datafusion-sql
    Building datafusion-substrait v54.1.0 (current)
       Built [ 312.480s] (current)
     Parsing datafusion-substrait v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-substrait v54.1.0 (baseline)
       Built [ 320.562s] (baseline)
     Parsing datafusion-substrait v54.1.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-substrait v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.197s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 635.072s] datafusion-substrait

codecov-commenter · 2026-07-23T07:04:11Z

Codecov Report

❌ Patch coverage is 74.76766% with 1629 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.08%. Comparing base (a253a6a) to head (eed92bc).
⚠️ Report is 81 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/joins/asof_join.rs	79.54%	209 Missing and 72 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs	38.70%	100 Missing and 14 partials ⚠️
datafusion/core/src/physical_planner.rs	40.27%	85 Missing and 1 partial ⚠️
datafusion/physical-plan/src/execution_plan.rs	54.13%	58 Missing and 3 partials ⚠️
...sical-plan/src/aggregates/ordered_single_stream.rs	86.74%	49 Missing and 10 partials ⚠️
...n/src/joins/piecewise_merge_join/existence_join.rs	88.13%	11 Missing and 40 partials ⚠️
datafusion/datasource-parquet/src/file_format.rs	55.33%	37 Missing and 9 partials ⚠️
datafusion/expr/src/logical_plan/tree_node.rs	0.00%	43 Missing ⚠️
...afusion/core/src/datasource/file_format/options.rs	25.00%	42 Missing ⚠️
datafusion/physical-plan/src/aggregates/mod.rs	69.40%	27 Missing and 14 partials ⚠️
... and 71 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23832      +/-   ##
==========================================
+ Coverage   81.03%   81.08%   +0.04%     
==========================================
  Files        1107     1113       +6     
  Lines      385064   392544    +7480     
  Branches   385064   392544    +7480     
==========================================
+ Hits       312040   318293    +6253     
- Misses      54616    55509     +893     
- Partials    18408    18742     +334

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026

Xuanwo mentioned this pull request Jul 23, 2026

feat: support ASOF joins #23738

Open

github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026

github-actions Bot added the physical-expr Changes to the physical-expr crates label Aug 14, 2026

Xuanwo added 3 commits August 21, 2026 01:54

fix: support floating ASOF equality keys

8f501fd

feat: add ASOF join logical semantics

1a925ce

feat: serialize ASOF join plans

6ae0de3

Xuanwo force-pushed the xuanwo/asof-proto branch from 19940d8 to 6ae0de3 Compare August 20, 2026 17:56

Navigation Menu

Uh oh!

feat: serialize ASOF join plans - #23832

feat: serialize ASOF join plans#23832
Xuanwo wants to merge 3 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-proto

Xuanwo commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Xuanwo commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Xuanwo commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

codecov-commenter commented Jul 23, 2026 •

edited

Loading