Skip to content

feat: make AggIntent inputs arity-agnostic, and lower multi-column COUNT(DISTINCT) - #426

Open
Selvomega wants to merge 1 commit into
mainfrom
feat/supporting-composite-count-distinct-and-vec-input-for-aggintent
Open

Selvomega wants to merge 1 commit into
mainfrom
feat/supporting-composite-count-distinct-and-vec-input-for-aggintent

Conversation

@Selvomega

@Selvomega Selvomega commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Two changes, in dependency order. The second is only expressible because of the
first.

Closes #425
Closes #424

Why

The IR asserted that an aggregate reads one column. AggIntent exposed
input_col() -> Option<C>, and every consumer that needed an input column went
through it. That is fine for SUM(x), but it is a claim about the whole
vocabulary, and the vocabulary had already outgrown it: #421 added
PearsonCorr { left, right } and had to make it return None from
input_col() defensively, so that no single-column consumer could pick up
half of the pair. The accessor had become a trap that each new multi-column
intent must remember to opt out of — and the cost of forgetting is a silently
wrong number, not a compile error.

A concrete query had nowhere to land. COUNT(DISTINCT a, b) counts distinct
tuples. SQL lowering used to drop every argument after the first, reporting
single-column cardinality where the query asked for tuple cardinality. #419
stopped the silent miscount by rejecting the call outright
(UnsupportedAggregate("multi-column COUNT(DISTINCT)")).

These are the same problem at two levels. The rejection in #419 was not a
missing feature so much as a vocabulary that could not express the intent.

What

IR — inputs are arity-agnostic.

  • AggIntent::input_col() is removed. input_cols() -> Vec<C> is the only
    column accessor.
  • PearsonCorr's opt-out is gone; it is now ordinary rather than a special
    case that has to neutralize an accessor.

Query — multi-column COUNT(DISTINCT).

  • AggIntent::Cardinality carries cols: Vec<C> instead of col: Option<C>.
    One entry is COUNT(DISTINCT col); several count distinct tuples; empty keeps
    the PromQL "the sample value" convention (count_values,
    distinct_over_time).
  • COUNT(DISTINCT a, b, ...) lowers, plans, and realizes end to end.
  • Serialized Cardinality changes field colcols. Breaking for a
    payload that spells the field out; a payload that omits it still reads as the
    implicit input.

Before this PR

IR contract. Two accessors, with a documented trap:

/// The explicit input of a single-column reducer. `PearsonCorr`,
/// argument-less aggregates, and implicit PromQL sample inputs return `None`.
/// Use `input_cols` for dependency tracking; this accessor is for consumers
/// that have already selected a single-column implementation.
pub fn input_col(&self) -> Option<C>
pub fn input_cols(&self) -> Vec<C>

A new multi-column intent had to remember to fall through input_col()'s _
arm. Nothing enforced it, and getting it wrong yields a wrong number.

Query behavior.

$ uv run python pyscripts/asap-planner_test/run_dag_export.py -w workload/tpch_deequ.yaml
skipping "U-P2b" — lowering failed: unsupported aggregate: multi-column COUNT(DISTINCT)
pre.json: 49/50 exported, 1 failed to lower; 224 node(s) over 176 distinct, 1 shared

U-P2b is the check that (l_orderkey, l_linenumber) is a key:

SELECT count(DISTINCT l_orderkey, l_linenumber) * 1.0 / count(*) AS pk_distinctness
FROM lineitem

Corpus ratchets on main:

Corpus Lowered Rejected
tpch_deequ 49 / 50 [("P2b", "multi-column COUNT(DISTINCT)")]
synthetic_packet_trace 61 / 70 9 tuple counts

After this PR

IR contract. One accessor, no opt-out to remember:

/// Every value-column dependency, in argument order. The only accessor:
/// an intent's arity is its own business, so no consumer can ask for "the"
/// input column of an aggregate that reads two [...] and silently receive
/// one leg of it.
pub fn input_cols(&self) -> Vec<C>

Adding a two-column covar or a three-column regr_* is now a variant plus a
lowering arm. No consumer needs to learn about it, and no variant needs to
defend itself against an accessor.

Query behavior.

$ uv run python pyscripts/asap-planner_test/run_dag_export.py -w workload/tpch_deequ.yaml
pre.json:  50/50 exported, 0 failed to lower; 228 node(s) over 179 distinct, 1 shared
post.json: 50/50 exported, 0 failed to lower; 228 node(s) over 179 distinct, 1 shared

U-P2b lowers to one measure carrying both columns, in argument order:

Aggregate {
  measures: [Cardinality { cols: [0, 1], accuracy: Exact }],
  ...
}
Corpus Lowered Rejected
tpch_deequ 50 / 50 none
synthetic_packet_trace 70 / 70 none

A data-quality workload can express a composite-key uniqueness check and have
it planned like any other aggregate — sketched under an ε target, exact
pass-through at AccuracyTarget::Exact.

Verification

cargo test --workspace: 1157 passed, 0 failed (62 test binaries).
cargo clippy --workspace --all-targets: 0 warnings. cargo fmt --all --check:
clean. All four also ran as pre-commit hooks.

Tests added:

Test Behavior verified
input_cols_tracks_only_reducers (types) The single accessor's contract: a unary reducer reports one column, an empty list is the implicit PromQL/row-count input
distinct_tuple_cardinality_contract (types) A tuple count exposes every leg via input_cols, is mergeable, is not exact, and reports the same output shape as the one-column form
resolve_distinct_tuple_columns (types) Each leg resolves independently with qualifiers; one unknown leg fails rather than silently shortening the tuple
sketch_realizes_over_the_intents_input_columns (mapping) Case table over arity 1 / 1 / 0 / 2 — the 2-column case must produce Tuple, not a single leg
composite_distinct_counts_tuples (sql) COUNT(DISTINCT a,b)cols == [0,1]; COUNT(DISTINCT a)cols == [0]
composite_distinct_rejects_expression_arguments (sql) COUNT(DISTINCT a, b+1) is rejected, not reduced over a probe column
multi_arg_count_distinct_flow_counts_the_whole_tuple (sql) The 5-tuple flow key reaches the intent as [0,1,2,3,4]

The existing pearson_corr_contract keeps passing with its input_col()
assertion dropped — the property it asserted is now structural.
agg_intent_to_summary_kind_coverage_matrix pins the new decisions (ε → HLL,
Exact → pass-through) at both arities. Both corpus ratchets moved in this
commit, as the sidra workload and this test corpus are coupled.

End-to-end evidence is the run_dag_export.py output above. Performance
measurement: not applicable — no hot path changed, and planner time is
unchanged at ~310 ms for this workload. Screenshots: not applicable.

Limitations and follow-ups

  • The unary reducers were not widened. Sum, Avg, Quantile and the rest
    still carry col: Option<C>, which is right for their semantics. What this PR
    removes is the consumer-side assumption, not per-variant arity. A future
    multi-column reducer adds its own field shape.
  • UnivMon over tuples is withheld, not proven unusable. Its Cardinality
    readout is L0 of the item-frequency vector, so a tuple item is plausibly
    correct — but its sizing and cost model were written for a single value
    stream, and I did not verify them at tuple arity. Withholding is the
    conservative choice; admitting it is a separate change with its own evidence.
  • Serialization break. A stored AggIntent that spells out "col": <id> no
    longer deserializes. Following feat: support corr with a general bivariate aggregate type #421's precedent (kind: pearson_corr, dropped
    op), no migration is provided.
  • Single-column qualifier change. COUNT(DISTINCT t.a) now lowers to a
    qualified ColumnRef. Called out above; flag it if this PR should not carry
    it.

…UNT(DISTINCT)

Two changes, in dependency order. The second is only expressible because of
the first.

`AggIntent` asserted that an aggregate reads one column: `input_col() ->
Option<ColumnId>` was how every consumer reached an input. #421 added
`PearsonCorr { left, right }` and had to make it return `None` from that
accessor *defensively*, so no single-column consumer could pick up half of the
pair. The accessor had become a trap each new multi-column intent must
remember to opt out of, and the cost of forgetting is a silently wrong number
rather than a compile error.

That is also why `COUNT(DISTINCT a, b)` had nowhere to land. It counts
distinct *tuples*; SQL lowering dropped every argument after the first,
reporting single-column cardinality where the query asked for tuple
cardinality. #419 stopped the miscount by rejecting the call outright, which
left TPC-H/Deequ U-P2b and nine synthetic-packet-trace flow counts permanently
unlowerable.

Before this PR:

    SELECT count(DISTINCT l_orderkey, l_linenumber) * 1.0 / count(*)
    FROM lineitem
    -> lowering failed: unsupported aggregate: multi-column COUNT(DISTINCT)

After this PR:

    Aggregate { measures: [Cardinality { cols: [0, 1], .. }], .. }

What changed:

- `input_col()` is removed. `input_cols() -> Vec<C>` is the only column
  accessor, so no consumer can ask for "the" input column of an aggregate that
  reads two. An intent may now declare whatever arity its semantics need
  without a defensive opt-out, and `PearsonCorr`'s opt-out is gone. The
  genuinely unary reducers keep `col: Option<C>` — this relaxes what consumers
  assume, it does not widen every variant.
- `AggIntent::Cardinality` takes `cols: Vec<C>`. One entry is
  `COUNT(DISTINCT col)`, several count distinct tuples, empty keeps the PromQL
  "the sample value" convention. It is the first variable-arity intent, and was
  not expressible while the accessor split existed.
- Realization is the single-column one with a wider item: a tuple becomes a
  `SummaryInputExpr::Tuple`, which HLL/Theta/KMV hash as one value. UnivMon is
  withheld from a tuple — it estimates frequency moments over a single value
  stream — and the value-frequency rule refuses a multi-column intent directly,
  so that invariant does not rest on the candidate table alone.
- SQL lowering resolves every `COUNT(DISTINCT ...)` argument as a grouping key,
  so a qualifier survives a join; an expression argument is rejected rather
  than reduced over a probe column.

Serialization of `Cardinality` changes from `col` to `cols`; a payload that
omits the field still reads as the implicit input.

Both corpus ratchets move in this commit: tpch_deequ 49 -> 50 lowered with no
rejections, synthetic_packet_trace 61 -> 70.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment on lines +125 to +144
`Cardinality { cols, accuracy }` carries a list, not one column. One entry is
SQL `COUNT(DISTINCT col)`; several count distinct *tuples*
(`COUNT(DISTINCT a, b)`), which is not the distinct count of any one of them.
Empty is the PromQL convention "the sample value" (`count_values`,
`distinct_over_time`).

`input_cols()` is the only column accessor on `AggIntent` — an intent's arity is
its own business, so no consumer can ask for "the" input column of an aggregate
that reads two and silently receive one leg of it. That mattered concretely:
before `Cardinality` took a list, SQL lowering dropped every argument after the
first, reporting single-column cardinality as tuple cardinality.

Realization is the single-column one with a wider item: a tuple becomes a
`SummaryInputExpr::Tuple`, which the distinct-count sketches (HLL, Theta, KMV)
hash as one value. UnivMon is withheld from a tuple — it estimates frequency
moments over a single value stream. At `AccuracyTarget::Exact` the node stays a
logical pass-through at any arity. Each SQL argument must be a bare column,
qualifier preserved so a tuple over a join resolves to the correct side; an
expression argument is rejected rather than reduced over a probe column.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont understand any of this.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lololololololol Will double check. I indeed did not check doc part

@milindsrivastava1997

Copy link
Copy Markdown
Collaborator

@Selvomega Apart from the stuff added to design doc, rest LGTM.
@zzylol pls review.

@Selvomega

Copy link
Copy Markdown
Collaborator Author

@Selvomega Apart from the stuff added to design doc, rest LGTM. @zzylol pls review.

lolol I will double check that later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] AggIntent should in-general support multiple columns as input [Feature] Composite Count Distinct should be identified by the IR

2 participants