feat(sql): lower ClickHouse argMax/argMin to AggIntent::Extension - #242
Merged
Merged
Conversation
argMax(arg, val)/argMin(arg, val) ("arg's value from the row where val
is maximal/minimal") are two-column, row-selecting aggregates -- unlike
every existing AggIntent reducer (Sum/Min/Max/Avg/...), which folds one
column to a value derived from itself, these return a *different*
column's value, selected by which row extremizes a second column. No
existing AggIntent shape fits.
Design decision (issue #232 asked for research before picking):
a repo-wide grep for any existing arg-max/arg-min concept (PromQL front
end, other SQL dialects, docs) found nothing -- only the 3 corpus
occurrences this closes. sql-function-catalog's own
KNOWN_UNMAPPED_NATIVE_FUNCTIONS already documents the same "two-column
reducer" and "value from a particular row" gaps for corr/covar/regr_*
and first_value/last_value/nth_value, for the identical reason:
AggIntent's input_col()/resolve_agg_intent/aggregate_output_schema
machinery is single-column by construction. Per AggIntent::Extension's
own bar ("core only grows for intents >= 2 deployment models actually
use"), with exactly one consumer found, this lowers to
AggIntent::Extension { ext_kind: "arg_max" | "arg_min", payload }
rather than earning first-class ArgMax/ArgMin core variants. This
required zero changes to crates/types: Extension already has complete
plumbing (resolve, output_column, requires/is_per_series, and
downstream PassThrough handling in asap-aware-mapping), which is itself
evidence the Extension escape hatch is doing its job.
Implementation:
- sql-function-catalog: new RewriteKind::PassThrough for a
CLICKHOUSE_BUILTINS entry with no native DataFusion shape to rewrite
to at all (argmax/argmin, arity 2) -- a new pattern alongside the
existing CountDistinct/CountIfToSum rewrite targets.
- frontend-sql: ClickHouseBuiltinRewrite treats PassThrough as a no-op,
so the call reaches lower_agg_intent under its own ClickHouse name.
New lower_arg_selector() validates both arguments as bare columns
(reducer_col's existing "no expression arguments" rule, issue #115,
generalized to 2 args) and builds the Extension node directly; core
never resolves Extension's payload, so both columns are kept as
ColumnRef names rather than positionally-bound ColumnIds.
- Documented, not fixed (matching #230's splitByChar-array-indexing
precedent): DerivedCols::rewrite_agg only special-cases an
aggregate's *first* argument, so val would be dropped from a Project
inserted for some unrelated reason in the same query. Does not affect
the corpus today -- all 3 argMax uses are bare columns.
Corpus tally (bgp_jan2024_workload): Lowered 145 -> 148, Plan 40 -> 37,
every other category unchanged -- all 3 argMax occurrences move
cleanly to Lowered with no companion gap.
Closes #232
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
argMax(arg, val)— "returnarg's value from the row wherevalis maximal" — appears 3 times in thebgp_jan2024_workloadcorpus and had noAggIntentrepresentation, unlike every existing reducer (Sum/Min/Max/Avg/...) which folds one column to a value derived from itself.argMax/argMinare two-column, row-selecting aggregates: they return a different column's value, selected by which row extremizes a second column. No existingAggIntentshape fits.Design decision:
AggIntent::Extension, not a new coreArgMax/ArgMinvariantIssue #232 asked for research before picking between (1) a first-class
AggIntent::ArgMax/ArgMincore variant, or (2)AggIntent::Extension— the crate's own escape hatch, whose doc comment says "core only grows for intents ≥2 deployment models actually use."Research performed: a repo-wide grep (
grep -rniE 'arg[_ ]?max|arg[_ ]?min'over.rs/.md/.toml, plus a full-repo scan forargmax/argmin) across the PromQL front end, other SQL dialects, and docs found zero other uses or discussion of an arg-max/arg-min concept anywhere in the codebase — only the 3 corpus occurrences this PR closes.More importantly,
crates/sql-function-catalog/src/lib.rs's ownKNOWN_UNMAPPED_NATIVE_FUNCTIONStable already documents this exact gap shape twice, independently:first_value/last_value/nth_value: "noAggIntentvariant models 'the value from a particular row' as a reduction"corr/covar/regr_*: "everyAggIntentvalue reducer takes one input column (reducer_col...), so these have no home yet"argMax/argMinare exactly the union of both gaps (a two-column reducer and a "value from a particular row" selector). Structurally,AggIntent'sinput_col()/resolve_agg_intent/aggregate_output_schema/collect_referenced_columnsmachinery is single-column by construction throughoutcrates/types/src/pre_asap/; giving a new variant a real second column would mean widening that machinery for a shape exactly one deployment model (this ClickHouse dialect) needs today — precisely what the "≥2 deployment models" bar exists to gate against.Given that, this lowers
argMax/argMintoAggIntent::Extension { ext_kind: "arg_max" | "arg_min", payload }. This required zero changes tocrates/types—Extensionalready has complete plumbing (resolve_agg_intent,output_column,requires/is_per_series, and downstreamPassThroughhandling inasap-aware-mapping'sboundary.rs/bind.rs/cost_model.rs), which is itself evidence theExtensionescape hatch is doing its intended job here.Implementation
crates/sql-function-catalog: newRewriteKind::PassThroughvariant for aCLICKHOUSE_BUILTINSentry with no native DataFusion aggregate shape to rewrite to at all — a new pattern alongside the existingCountDistinct/CountIfToSumrewrite targets (which do rewrite to an already-handled native shape). New entries:argmax/argmin, arityExact(2).crates/frontend-sql/src/sql/mod.rs:ClickHouseBuiltinRewrite::rewritetreatsPassThroughas a no-op (Transformed::no) — the call survives tolower_agg_intentunder its own ClickHouse name, still routed through the existing stub-AggregateUDFregistration loop inbuild_context(so DataFusion's planner accepts the name at all).lower_arg_selector()recognizes"argmax"/"argmin", validates both arguments as bare columns (generalizingreducer_col's existing "no expression arguments" rule from issue L3: AggIntent::Quantile / Cardinality / TopK drop their input column — distinct aggregates compare equal #115 to 2 args), and builds theExtensionnode directly. Core never resolvesExtension'spayload, so both columns are kept asColumnRefs (JSON) rather than being run throughresolve_agg_intent's positionalColumnIdbinding — consistent withExtension's documented opaque-to-core contract.splitByChar's array-indexing gap undone:DerivedCols::rewrite_aggonly materializes/passes through an aggregate's first argument when deriving columns beneath anAggregatenode. If some other aggregate in the same query forces aProjectto be inserted,argMax's second argument (val) could be silently dropped from it. This does not affect the corpus today — all 3argMaxoccurrences have both arguments as bare columns and no suchProjectgets inserted for them. Documented inlower_arg_selector's doc comment.Scope boundary
Purely structural, per the precedent issue #230 already set: getting
argMax/argMinto lower to a structurally correctAggIntentnode. NoSummaryKind/sketch-selection or real runtime binding forExtension'sarg_max/arg_minpayload inasap-aware-mapping— it defaults toImplementation::PassThrough, same as every other unrecognizedExtensionkind today.Verification
crates/frontend-sql/tests/sql_lowering.rs: 4 new unit tests —argMax/argMineach lower to their ownExtension { ext_kind, .. }, the payload preserves both column names correctly, and a non-column argument is rejected (not silently dropped).bgp_jan2024_workloadcorpus tally (corpus_lowering_matches_the_pinned_aggregate_tally):LoweredPlanAll 3 corpus
argMaxoccurrences move cleanly fromPlan("unknown function: argmax") toLowered, with no companion gap.cargo build --workspace --all-targets,cargo test --workspace,cargo fmt --all -- --check,cargo clippy --workspace --all-targets --all-features -- -D warnings— all clean.Closes #232