Summary
Pull the "SQL dialect builtin function name (+ arity) → canonical AggIntent"
mapping out of the hand-written match in frontend-sql::lower_agg_intent
and into its own small, independently-maintained (ideally generated) catalog
— inspired by tobilg/polyglot's
polyglot-sql-function-catalogs crate, which does exactly this for its
30+ SQL dialects: per-dialect data (function name, arity/overloads, casing),
deliberately not depending on the core parser crate (exposed instead via a
small CatalogSink trait the consumer implements), populated by extraction
tooling that introspects real databases rather than hand-maintained.
Where the pain shows up today
lower_agg_intent (crates/frontend-sql/src/sql/mod.rs) is a single
hand-written match from a DataFusion-resolved function name to an
AggIntent:
Ok(match name.as_str() {
"count" if agg_fn.distinct => AggIntent::Cardinality { .. },
"count" => AggIntent::Count { .. },
"sum" => AggIntent::Sum { .. },
...
"approx_percentile_cont" | "percentile_cont" => AggIntent::Quantile { .. },
"median" | "approx_median" => AggIntent::Quantile { q: 0.5, .. },
"approx_distinct" => AggIntent::Cardinality { .. },
_ => return Err(LoweringError::UnsupportedAggregate(name)),
})
This covers names DataFusion itself resolves. A dialect builtin DataFusion
doesn't know at all is a strictly bigger problem — the uniqExact case
(#221) needed, per its own doc comments:
- A stub
AggregateUDF registered into the SessionContext purely so
DataFusion's planner doesn't reject the unknown function name while
parsing (uniq_exact_udaf() — its accumulator is literally
unimplemented!(), dead code by construction).
- A custom
FunctionRewrite (UniqExactRewrite) rewriting every
uniqexact(x) call to DataFusion's own count(x) DISTINCT before
lower_agg_intent ever sees it, wired in by calling DataFusion's
ApplyFunctionRewrites analyzer rule directly (not through
Analyzer::execute_and_check, which runs an unwanted post-check) — a
dozen lines of doc comment just to justify how it's invoked.
That's a meaningful amount of DataFusion-internals engineering for one
ClickHouse builtin. with_dialect's own doc comment already names another
still-unhandled one (countIf), and there's no reason to expect the list
stops there — ClickHouse's builtin function surface is large.
Proposed direction
- Separate the data from the code that consumes it. A catalog module
(or crate, if it grows) mapping (dialect, function_name) → (arity, canonical semantic), structured close to polyglot-sql-function-catalogs's
shape: existence + arity/overloads + casing is enough — no need to model
full type signatures, matching that project's own deliberately shallow
scope. It should not depend on asap-frontend-sql; asap-frontend-sql
depends on it (or reads it through a small sink-style trait), keeping the
dependency arrow pointing the way the rest of the workspace already does
(asap-types "depends on nothing").
- Generalize the stub-UDAF-plus-
FunctionRewrite mechanism so adding a
new ClickHouse-only builtin is a catalog entry, not a new Rust type
implementing FunctionRewrite plus a new stub-UDAF function. uniqExact
is the shape every future one of these will take: register a stub so
DataFusion's planner accepts the call, rewrite it to something DataFusion
natively understands before lower_agg_intent runs.
- Where feasible, generate rather than hand-maintain, the way
tools/clickhouse/extract_functions.py /
tools/duckdb/extract_functions.py do in the reference project —
ClickHouse exposes system.functions; DataFusion's own function
registry is introspectable in-process (SessionContext's UDF/UDAF
lists). Needs a decision on where such extraction tooling would actually
run (a live ClickHouse instance in CI/dev is a real requirement, not a
given) — flagging as an open question rather than assuming it's free.
Explicitly out of scope
ElasticSQL having no vendored parser at all — a different, bigger
gap (no parser, not just missing functions); polyglot doesn't cover
Elasticsearch SQL either, so it offers no shortcut there. Separate issue
if/when that gets picked up.
- Modeling argument types / return types / coercion — matching
polyglot-sql-function-catalogs's own choice to leave these out; our
AggIntent construction doesn't need them either (it already rejects a
non-column argument explicitly, reducer_col).
Related
Summary
Pull the "SQL dialect builtin function name (+ arity) → canonical
AggIntent"mapping out of the hand-written match in
frontend-sql::lower_agg_intentand into its own small, independently-maintained (ideally generated) catalog
— inspired by
tobilg/polyglot'spolyglot-sql-function-catalogscrate, which does exactly this for its30+ SQL dialects: per-dialect data (function name, arity/overloads, casing),
deliberately not depending on the core parser crate (exposed instead via a
small
CatalogSinktrait the consumer implements), populated by extractiontooling that introspects real databases rather than hand-maintained.
Where the pain shows up today
lower_agg_intent(crates/frontend-sql/src/sql/mod.rs) is a singlehand-written match from a DataFusion-resolved function name to an
AggIntent:This covers names DataFusion itself resolves. A dialect builtin DataFusion
doesn't know at all is a strictly bigger problem — the
uniqExactcase(#221) needed, per its own doc comments:
AggregateUDFregistered into theSessionContextpurely soDataFusion's planner doesn't reject the unknown function name while
parsing (
uniq_exact_udaf()— itsaccumulatoris literallyunimplemented!(), dead code by construction).FunctionRewrite(UniqExactRewrite) rewriting everyuniqexact(x)call to DataFusion's owncount(x) DISTINCTbeforelower_agg_intentever sees it, wired in by calling DataFusion'sApplyFunctionRewritesanalyzer rule directly (not throughAnalyzer::execute_and_check, which runs an unwanted post-check) — adozen lines of doc comment just to justify how it's invoked.
That's a meaningful amount of DataFusion-internals engineering for one
ClickHouse builtin.
with_dialect's own doc comment already names anotherstill-unhandled one (
countIf), and there's no reason to expect the liststops there — ClickHouse's builtin function surface is large.
Proposed direction
(or crate, if it grows) mapping
(dialect, function_name) → (arity, canonical semantic), structured close topolyglot-sql-function-catalogs'sshape: existence + arity/overloads + casing is enough — no need to model
full type signatures, matching that project's own deliberately shallow
scope. It should not depend on
asap-frontend-sql;asap-frontend-sqldepends on it (or reads it through a small sink-style trait), keeping the
dependency arrow pointing the way the rest of the workspace already does
(
asap-types"depends on nothing").FunctionRewritemechanism so adding anew ClickHouse-only builtin is a catalog entry, not a new Rust type
implementing
FunctionRewriteplus a new stub-UDAF function.uniqExactis the shape every future one of these will take: register a stub so
DataFusion's planner accepts the call, rewrite it to something DataFusion
natively understands before
lower_agg_intentruns.tools/clickhouse/extract_functions.py/tools/duckdb/extract_functions.pydo in the reference project —ClickHouse exposes
system.functions; DataFusion's own functionregistry is introspectable in-process (
SessionContext's UDF/UDAFlists). Needs a decision on where such extraction tooling would actually
run (a live ClickHouse instance in CI/dev is a real requirement, not a
given) — flagging as an open question rather than assuming it's free.
Explicitly out of scope
ElasticSQLhaving no vendored parser at all — a different, biggergap (no parser, not just missing functions); polyglot doesn't cover
Elasticsearch SQL either, so it offers no shortcut there. Separate issue
if/when that gets picked up.
polyglot-sql-function-catalogs's own choice to leave these out; ourAggIntentconstruction doesn't need them either (it already rejects anon-column argument explicitly,
reducer_col).Related
crates/frontend-sql/src/sql/mod.rs—lower_agg_intent,uniq_exact_udaf,UniqExactRewriteuniqExactcase this generalizes fromtobilg/polyglot'spolyglot-sql-function-catalogscrate — the design this borrows from