Commit e095586
authored
docs: add suggest-native-expression skill for assessing native expression candidates (#5348)
* docs: add suggest-native-expression skill for assessing native candidates
Adds a skill that assesses whether a Spark expression currently running
through the JVM codegen dispatcher is worth implementing natively, scoring
compatibility confidence and native upside, recording the verdict in the
per-expression audit log, and filing one issue for the recommendation.
Also records the first assessment (unbase64) and documents the new audit
line kind in the expression-audits index.
* docs: record native candidate assessment for sequence
Second run of the suggest-native-expression skill. Recommends a native
kernel for integral element types only, with date and timestamp sequences
staying on the codegen dispatcher.
* docs: widen the suggest-native-expression candidate pool
The pool was built by grepping the auto-generated Implementation column of
the expression reference, which is lossy: it is derived from a function
registry lookup, so a dispatcher-routed serde whose SQL name does not
resolve renders as an em-dash and vanishes from the grep. `mask` is one
such case, and there are 67 classes extending CometCodegenDispatch against
58 rows labelled "Codegen dispatch". Build the pool from the serde
definitions instead and use the docs table only for cross-reference.
Also stop excluding the 42 `Hybrid` expressions. Their native path exists
but is off by default, so the dispatcher is what actually runs and the
user-visible win from making native the default is the same as writing a
new kernel. They are now a second sub-pool with their own proposed
approach.
Narrow the "recovers a whole-operator Spark fallback" upside signal to what
it was really meant to cover: inputs of an in-pool expression that the
dispatcher cannot compile. Expressions Comet does not support at all stay
out of pool, and the skill now says so explicitly instead of implying its
pool is the whole space.
* docs: derive Spark versions from pom.xml, move audit lines to a separate PR
Review feedback from @comphead:
- Stop hardcoding the Spark version list. The versions in the skill (3.4.3,
3.5.8, 4.0.1, 4.1.1, inherited from audit-comet-expression) were already
stale against pom.xml, which pins 3.4.3, 3.5.9, 4.0.4, 4.1.3 and has a
spark-4.2 profile. Derive the list from pom.xml at run time instead, so it
tracks support being added and dropped.
- Move the two audit-log lines to their own PR so the skill can be reviewed
separately from the assessments it produced.1 parent fbdc267 commit e095586
2 files changed
Lines changed: 550 additions & 1 deletion
0 commit comments