Requirements for scalar UDF preimage

[linhr]

ScalarUDFImpl::preimage allows defining "preimage" (range of input values) for function outputs, which can be used to rewrite binary comparison expressions.

The rewrite rules for equality (=) or inequality (!=) seem intuitive. But f(x) <= c would also be rewritten to x < u where u is the (exclusive) upper bound. (Here is the code.) I'm not sure if this holds in general. Does this assume that the function must be monotonically non-decreasing?

I was trying to see if the documentation mentions monotonicity for scalar UDFs but I couldn't find it. Let me know if I misunderstood anything!

[drin]

I tried to address this in a PR which has gone a bit stale but hopefully gets movement soon. Here is a relevant commit in the PR: 18648 - 3326794 - R39-R52

You're right that f(x) <= c --> x < u does not hold in general. The existing implementation supports: (1) direct preimage (c is a value produced by f) and (2) contiguous, monotonically increasing intervals for f(x).

For (1), there is a lot of discussion in the mentioned PR (#18648) about trying to expand the definition to support preimage intervals (c is in an interval produced by f, e.g. 8.3 can be in [8, 9) even if 8.3 is not directly produced by f). That expansion won't get accepted until it's broken into a different PR, but maybe addresses part of your question.

For (2), this portion of the commit 18648 - 3326794 - R149-R171 may expand a basic preimage test to provide a relevant example. In it, the intervals for f(x) are not adjacent (e.g. [100, 200), [300, 400)), so some boundary cases do not apply as if the intervals were adjacent (e.g. [100, 200), [200, 400)).

Finally, for (2), it is assumed that each interval is increasing and that assumption is built into the implementation in expr_simplifier. But, the result of expr_simplifier can be inverted for a monotonically decreasing function, which may not be clean and is probably not currently implemented/supported (though I'm not totally sure).

[linhr]

Thanks @drin!

My initial comment more came from the documentation perspective, since the monotonicity assumption is not obvious from the name preimage. So a developer may incorrectly implement preimage without knowing how it is actually used by the optimizer.

Maybe we could define certain monotonicity method for scalar UDFs and consider this property in the optimizer. Using expression simplifier to invert monotonically decreasing functions makes sense for numeric functions, though I'm not sure if it works well for functions returning e.g. strings (whose monotonicity could also be well-defined).

Glad to see the efforts in #18648!