Further refactoring of type coercion function code by Jefffrey · Pull Request #19603 · apache/datafusion

Jefffrey · 2026-01-02T08:43:05Z

Which issue does this PR close?

Follow up to Refactor duplicate code in type_coercion/functions.rs #19518
Initial work before tackling Improve error messages when coercion fails #19004

Rationale for this change

Found lots of code duplicated here, so unifying them.

What changes are included in this PR?

Reduce code duplication around handling scalar/aggregate/window UDFs in type coercion related code, via usage of new UDFCoercionExt trait which unifies some of their behaviours (introduced by Refactor duplicate code in type_coercion/functions.rs #19518)
Deprecate functions can_coerce_from() and generate_signature_error_msg() to work towards minimizing our public API surface
Fix some UDF signatures which nested UserDefined within OneOf
Fix bug where type coercion rewrites weren't being applied to arguments of non-aggregate window UDFs

Are these changes tested?

Existing tests.

Are there any user-facing changes?

Deprecated some functions.

Jefffrey · 2026-01-02T08:43:45Z

    }
 }

-/// Performs type coercion for scalar function arguments.


Reorganizing these so the only public function of this file is near the top (fields_with_udf())

Jefffrey · 2026-01-02T08:44:58Z

+            Expr::ScalarFunction(_)
+            | Expr::WindowFunction(_)
+            | Expr::AggregateFunction(_) => {
+                Ok(self.to_field(schema)?.1.data_type().clone())


Reusing existing logic from to_field() implementation

Jefffrey · 2026-01-02T08:45:27Z


+/// Verify that function is invoked with correct number and type of arguments as
+/// defined in `TypeSignature`.
+fn verify_function_arguments<F: UDFCoercionExt>(


With the new UDFCoercionExt trait we can now more easily share common code between scalar, aggregate and window UDFs

Jefffrey · 2026-01-02T08:46:52Z

    pub fn new() -> Self {
        Self {
-            signature: Signature::one_of(
-                vec![TypeSignature::Nullary, TypeSignature::UserDefined],


I don't really like the idea of having UserDefined nested within a OneOf signature, so amending some functions which are doing this. Ideally a function declaring UserDefined should handle all of its coercion logic itself, instead of mixing this custom logic with the other signature API.

Jefffrey · 2026-01-02T08:47:40Z

+                    expr::WindowFunctionDefinition::WindowUDF(udf) => {
+                        coerce_arguments_for_signature(args, self.schema, udf.as_ref())?


This arm for coercing non-aggregate window UDFs actually was missing; adding this back in lead to discovering some UDF window tests were incorrect (see next comment)

Jefffrey · 2026-01-02T08:48:48Z

            fn new(test_state: Arc<TestState>) -> Self {
                let signature =
-                    Signature::exact(vec![DataType::Float64], Volatility::Immutable);
+                    Signature::exact(vec![DataType::Int64], Volatility::Immutable);


This sample UDF declared it accepted f64's but internally it acted as if input was i64; this was never caught because apparently type coercion wasn't being run on non-aggregate window functions, and the input data was already i64. Nice little fix.

martin-g

LGTM

martin-g · 2026-01-02T12:39:25Z

        TypeSignature::Variadic(valid_types) => valid_types
            .iter()
-            .map(|valid_type| current_types.iter().map(|_| valid_type.clone()).collect())
+            .map(|valid_type| vec![valid_type.clone(); current_types.len()])


Jefffrey · 2026-01-11T03:42:14Z

-    ///
-    /// Otherwise, returns an error if there's a type mismatch between
-    /// the window function's signature and the provided arguments.
-    fn window_function_field(


This has been inlined to to_field()

Jefffrey · 2026-01-11T03:46:02Z

 /// (losslessly converted) into a value of `type_to`
 ///
 /// See the module level documentation for more detail on coercion.
+#[deprecated(since = "53.0.0", note = "Unused internal function")]


This was only ever used in a test; trying to minimize our API surface area for simplicity so deprecating this

Jefffrey · 2026-01-11T03:46:18Z

 ///     round(Float32)
 /// ```
 #[expect(clippy::needless_pass_by_value)]
+#[deprecated(since = "53.0.0", note = "Internal function")]


Similarly removing this from our public API since I see no need for it to be public

alamb

Looks like a nice cleanup to me -- thank you @Jefffrey and @martin-g

## Which issue does this PR close?  - Follow up to apache#19518 - Initial work before tackling apache#19004 ## Rationale for this change  Found lots of code duplicated here, so unifying them. ## What changes are included in this PR?  - Reduce code duplication around handling scalar/aggregate/window UDFs in type coercion related code, via usage of new `UDFCoercionExt` trait which unifies some of their behaviours (introduced by apache#19518) - Deprecate functions `can_coerce_from()` and `generate_signature_error_msg()` to work towards minimizing our public API surface - Fix some UDF signatures which nested `UserDefined` within `OneOf` - Fix bug where type coercion rewrites weren't being applied to arguments of non-aggregate window UDFs ## Are these changes tested?  Existing tests. ## Are there any user-facing changes?  Deprecated some functions.  --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…ls, matching pre-UDWF and documented early-validation behavior Prior to converting these to User Defined Window Functions (apache#13201, 2024-11-13), the built-in signatures were exact: first_value/last_value took exactly 1 argument, nth_value exactly 2 - there was never a zero-argument form. The UDWF conversion introduced `TypeSignature::Any(0)` (later renamed to `Nullary` in apache#19603) into the shared OneOf signature, which let a zero-argument call (e.g. `first_value()`) pass type checking. `NthValueEvaluator::evaluate` unconditionally indexes into `values[0]`, so this caused a process-killing panic (index out of bounds) instead of a query error. Two fixes, matching two separate safety nets that already exist in this codebase for exactly this class of bug: 1. Remove the zero-argument form from the signature. None of these functions have a meaningful 0-arg form - first_value/last_value need 1, nth_value needs 2, matching lead()/lag()'s analogous Any(1..N) signature, which never included it. This lets DataFusion's generic argument-count check (datafusion_expr::type_coercion::functions::data_types) reject `first_value()` cleanly during logical planning. 2. Validate the argument count in `partition_evaluator()` itself. Per the review discussion on apache#13201 (comment 2454209975) and the resulting `create_udwf_window_expr` code in datafusion/physical-plan/src/windows/mod.rs, UDWF signatures are deliberately permissive and argument validation is expected to happen inside the evaluator - `create_evaluator()` is called during physical planning specifically to surface such errors early rather than during execution. That validation was simply never implemented for the zero-argument case here. Together these give defense in depth: the signature fix rejects the common case even before physical planning, and the evaluator-level check protects any caller that reaches partition_evaluator() through a path other than SQL-level type coercion (e.g. the DataFrame API). Verified both `select first_value() over ()` and `select first_value() over (order by x)` - the latter used to return an unrelated Arrow schema-mismatch error by coincidence rather than because it was ever a valid call - now fail identically and cleanly during planning. Adds sqllogictest coverage for all three functions and unit tests for both the signature and the evaluator-level guard.

Further refactoring of type coercion function code

3d63dd0

github-actions Bot added logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate functions Changes to functions implementation spark labels Jan 2, 2026

Jefffrey commented Jan 2, 2026

View reviewed changes

Jefffrey marked this pull request as ready for review January 2, 2026 09:41

martin-g approved these changes Jan 2, 2026

View reviewed changes

Merge branch 'main' into further-refactoring-type-coercion

7b82b8a

Jefffrey commented Jan 11, 2026

View reviewed changes

fix deprecation versions to 53

c123c0b

Jefffrey commented Jan 11, 2026

View reviewed changes

Merge branch 'main' into further-refactoring-type-coercion

e7dbf5c

alamb approved these changes Jan 27, 2026

View reviewed changes

Merge branch 'main' into further-refactoring-type-coercion

4b63813

Jefffrey added this pull request to the merge queue Jan 28, 2026

Merged via the queue into apache:main with commit a77e5a5 Jan 28, 2026
32 checks passed

Jefffrey deleted the further-refactoring-type-coercion branch January 28, 2026 13:54

		expr::WindowFunctionDefinition::WindowUDF(udf) => {
		coerce_arguments_for_signature(args, self.schema, udf.as_ref())?

Uh oh!

Conversation

Jefffrey commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jefffrey Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jefffrey commented Jan 2, 2026 •

edited

Loading

Jefffrey Jan 2, 2026 •

edited

Loading