Skip to content

fix(group-by): emit an explicit terms size so GROUP BY returns every group (#205) - #206

Merged
fupelaqu merged 2 commits into
mainfrom
fix/205-group-by-default-terms-size
Aug 8, 2026
Merged

fupelaqu merged 2 commits into
mainfrom
fix/205-group-by-default-terms-size

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Fixes #205.

What

SELECT category, COUNT(*) FROM t GROUP BY category on an index with 100 distinct categories
returned 10 rows — Elasticsearch's default terms bucket size — with no error, no warning,
and no truncation flag. In SQL, GROUP BY with no LIMIT means every group.

Fix (2 defects)

  1. Bucket.update (sql/…/query/GroupBy.scala) — when the statement has no LIMIT, the
    bucket size is now Bucket.DefaultSize = 65536 (Elasticsearch's own search.max_buckets
    ceiling) instead of None. The generated terms aggregation always carries an explicit
    size, so the default matches SQL semantics; a cardinality beyond 65536 fails loudly
    server-side (too_many_buckets_exception) rather than truncating silently. An explicit
    LIMIT n still maps to size = n (the existing top-N push-down), unchanged.

  2. Terms-emission discard chain (bridge/…/ElasticAggregation.scala, both the template and the
    hand-maintained es6 copy)
    — latent value-discard bug surfaced by (1): size, HAVING-derived
    include, and exclude were each applied to the original termsAgg binding, so each
    assignment silently discarded the previous one (an exclude dropped the size just applied; an
    exclude also dropped an include). The steps now chain. This is why exclude-bearing GROUP BY
    tests initially kept passing with no size — the bug was masking the fix.

Tests

  • SQLQuerySpec (template + es6): 18 expected JSONs updated per bridge — every GROUP BY terms
    now asserts "size": 65536 (or the LIMIT-derived size). 118/118 green on the template,
    es6, es7, es8 and es9 bridges; sql module 395/395 green.
  • New GroupByCompletenessSpec (testkit template + concrete subclasses for es6 rest, es6
    jest, es7 rest, es8 java, es9 java): 3-shard index, 37 categories (cat_i holds exactly
    i docs, so group count and per-group counts and total doc coverage are exact oracles):
    • no LIMIT → 37 groups, every per-group count exact, sum = all 703 docs (the GROUP BY without LIMIT silently returns only the top 10 groups (silent wrong answer) #205 guard);
    • LIMIT 5 → 5 groups (push-down preserved);
    • ORDER BY category DESC LIMIT 3 → exact top keys/counts. Ordered on the key, not the
      metric: a metric-ordered terms top-N is shard-approximate by design (doc_count_error) and
      the first run of this spec actually caught it returning cat_34 instead of cat_35.
    • Green on real ES 6.8, 7.17, 8.18, 9.0 (Docker).

Not in this PR (follow-ups)

  • Window PARTITION BY has the same defect (partition terms gets no size; >10 partitions
    silently lose window values). Deliberately not folded in — it interacts with the top_hits
    caps and the inline OVER LIMIT push-down. Filed separately.
  • sum_other_doc_count-based truncation warning (issue option 3) — defense-in-depth for the

    65536-cardinality edge, needs per-client response plumbing.

  • Downstream repos (extensions/jdbc/arrow) that pin generated GROUP BY JSON in their own tests
    will need the same "size": 65536 expectation update when they bump core.

🤖 Generated with Claude Code

fupelaqu and others added 2 commits August 7, 2026 16:05
…group (#205)

GROUP BY with no LIMIT emitted no size on the terms aggregation, so
Elasticsearch silently returned its default 10 buckets - a 100-category
index reported 10 plausible-looking groups with no truncation flag.

Bucket.update now defaults the bucket size to Bucket.DefaultSize (65536,
Elasticsearch's own search.max_buckets ceiling) when the statement has no
LIMIT, so the default matches SQL semantics and cardinalities beyond the
ceiling fail loudly instead of truncating silently. An explicit LIMIT n
still maps to size = n (unchanged top-N push-down).

Also fixes the latent value-discard chain in the terms emission (bridge +
hand-maintained es6 bridge): size, HAVING-derived include and exclude were
each applied to the original termsAgg binding, so each assignment silently
discarded the previous one - an exclude dropped the size just applied.

New GroupByCompletenessSpec (testkit + es6 rest/jest, es7 rest, es8/es9
java): 3-shard index, 37 categories with exact per-group oracles; asserts
group count, per-group counts and total doc coverage with no LIMIT, LIMIT
push-down, and exact key-ordered top-N. Green on real ES 6.8/7.17/8.18/9.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fupelaqu
fupelaqu merged commit d65b8f2 into main Aug 8, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GROUP BY without LIMIT silently returns only the top 10 groups (silent wrong answer)

1 participant