feat: add Spark-compatible hypot function - #23774
Conversation
|
cc @Jefffrey @andygrove — this is my first contribution, so CI is currently awaiting a committer's approval. Would really appreciate a trigger when you have a moment. This adds a Spark-compatible Thanks! |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23774 +/- ##
==========================================
- Coverage 80.69% 80.69% -0.01%
==========================================
Files 1090 1091 +1
Lines 370357 370389 +32
Branches 370357 370389 +32
==========================================
+ Hits 298859 298878 +19
- Misses 53684 53689 +5
- Partials 17814 17822 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
## Which issue does this PR close? - Closes apache#23770 - Part of apache#15914 ## Rationale for this change Spark provides [`hypot(expr1, expr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or underflow. It was not yet implemented in `datafusion-spark` — only an auto-generated test stub existed at `spark/math/hypot.slt` with its query commented out. ## What changes are included in this PR? - Add `SparkHypot` (implementing `ScalarUDFImpl`) in `datafusion/spark/src/function/math/hypot.rs`, backed by Rust's `f64::hypot` — the same overflow-safe algorithm as Java/Spark's `Math.hypot`. - Register it in `datafusion/spark/src/function/math/mod.rs`. - Enable the `hypot.slt` sqllogictest. The signature is `exact(Float64, Float64) -> Float64`, following the `datafusion-spark` convention of only accepting types Spark supports. Computation uses the Arrow `binary` kernel so NULL in either argument propagates to a NULL result, matching Spark. ## Are these changes tested? Yes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers: - scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13), - double inputs, - NULL propagation when either argument is NULL, - the array path (including a NULL row), - overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive `sqrt(a^2 + b^2)` would overflow to `Infinity`. ## Are there any user-facing changes? Yes — adds the Spark-compatible `hypot` scalar function to `datafusion-spark`. No breaking changes to public APIs.
Which issue does this PR close?
datafusion-sparkSpark Compatible Functions #15914Rationale for this change
Spark provides
hypot(expr1, expr2), which returnssqrt(expr1^2 + expr2^2)computed without intermediate overflow or underflow. It was not yet implemented indatafusion-spark— only an auto-generated test stub existed atspark/math/hypot.sltwith its query commented out.What changes are included in this PR?
SparkHypot(implementingScalarUDFImpl) indatafusion/spark/src/function/math/hypot.rs, backed by Rust'sf64::hypot— the same overflow-safe algorithm as Java/Spark'sMath.hypot.datafusion/spark/src/function/math/mod.rs.hypot.sltsqllogictest.The signature is
exact(Float64, Float64) -> Float64, following thedatafusion-sparkconvention of only accepting types Spark supports. Computation uses the Arrowbinarykernel so NULL in either argument propagates to a NULL result, matching Spark.Are these changes tested?
Yes —
datafusion/sqllogictest/test_files/spark/math/hypot.sltcovers:hypot(3, 4)→ 5,hypot(5, 12)→ 13),hypot(3e200, 4e200)stays finite, whereas a naivesqrt(a^2 + b^2)would overflow toInfinity.Are there any user-facing changes?
Yes — adds the Spark-compatible
hypotscalar function todatafusion-spark. No breaking changes to public APIs.