Skip to content

fix(window): size the partition terms so PARTITION BY covers every partition (#207) - #208

Merged
fupelaqu merged 2 commits into
mainfrom
fix/207-window-partition-terms-size
Aug 8, 2026
Merged

fupelaqu merged 2 commits into
mainfrom
fix/207-window-partition-terms-size

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #207.

What

Same silent-wrong-answer class as #205, on the window path: the terms aggregation wrapping
every OVER (PARTITION BY ...) window had no explicit size, so Elasticsearch computed only its
default 10 partition buckets. With more than 10 distinct partition values, rows in partitions
11+ silently got no window value (ROW_NUMBER/RANK/DENSE_RANK: those rows are missing from
the ranking enrichment entirely; FIRST_VALUE/LAST_VALUE/STDDEV/PERCENTILE_CONT OVER:
enrichment null).

Fix

One line in the version-agnostic sql module — WindowFunction.buckets
(sql/…/function/aggregate/package.scala) now creates partition buckets with
Some(Bucket.DefaultSize) (the 65536 = search.max_buckets ceiling introduced by #205) instead
of None. Partition buckets never pass Bucket.update (no statement LIMIT applies to the
partition count — the inline OVER LIMIT bounds top_hits rows per partition, which is
unchanged), so the size must be set at construction.

The top_hits interplay that kept this out of PR #206 is bounded per partition, not per query:
partition count is capped by actual data cardinality (and search.max_buckets), and each
partition still carries at most its existing top_hits size (100 default / inline OVER LIMIT).
The old behavior wasn't a smaller response — it was a wrong one.

Tests

  • SQLQuerySpec (template + es6): the 5 window expected JSONs per bridge now assert
    "size": 65536 on the partition terms (4 ranking + the top-hits aggregation test). All five
    bridges 118/118 (es9 under sbt17 — its elastic4s is Java-17 bytecode), sql 395/395,
    core 724/724, macros green, cross-compiled 2.12 + 2.13.
  • New WindowPartitionCompletenessSpec (testkit template + concrete subclasses for es6
    rest, es6 jest, es7 rest, es8 java, es9 java): 3-shard index, 15 partitions (cat_i holds
    3 docs with amounts i*100+1..3, exact per-partition oracles):
    • ROW_NUMBER OVER (PARTITION BY category ORDER BY amount DESC) → all 15 partitions ranked,
      ranks 1..3 each, rank-1 row is the per-partition max;
    • FIRST_VALUE(amount) OVER (...) → every row of every partition enriched with the exact
      partition max.
    • Green on real ES 6.8, 7.17, 8.18, 9.0 (Docker).

Existing window fixtures (emp: 5 departments) could never catch this — same cardinality-≤10
blind spot documented in #205.

Found while validating (filed as #209)

The spec's first live run exposed a third member of the truncation family, pre-existing and
out of scope here: a plain SELECT with no LIMIT on the non-scroll search path emits no
top-level size, so Elasticsearch returns its default 10 base hits (45-row index → 10
arbitrary rows; the window enrichment itself was complete). Every row-count-asserting
WindowFunctionSpec query carries an explicit LIMIT, which is why it never surfaced. The new
spec uses LIMIT 100 with a comment pointing at that behavior, keeping the two truncations
separate.

🤖 Generated with Claude Code

fupelaqu and others added 2 commits August 8, 2026 08:47
…rtition (#207)

Window partition buckets were created with size = None and never pass
Bucket.update, so the partition terms aggregation fell back to
Elasticsearch's default 10 buckets - with more than 10 distinct partition
values, rows in partitions 11+ silently got no window value (ranking
windows lost those rows from the enrichment entirely).

WindowFunction.buckets now creates partition buckets with
Bucket.DefaultSize (65536, the search.max_buckets ceiling introduced by
the #205 fix). The statement LIMIT keeps bounding base rows only - it
never shrinks the partition count; the inline OVER LIMIT keeps bounding
top_hits rows per partition.

New WindowPartitionCompletenessSpec (testkit + es6 rest/jest, es7 rest,
es8/es9 java): 3-shard index, 15 partitions with exact per-partition
oracles; asserts ROW_NUMBER ranks and FIRST_VALUE enrichment across all
15 partitions. Green on real ES 6.8/7.17/8.18/9.0. Its first run also
surfaced a pre-existing, separate truncation (plain SELECT without LIMIT
returns ES's default 10 base hits) - documented in the spec, tracked
separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fupelaqu
fupelaqu merged commit 1ddfc4c into main Aug 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Window PARTITION BY silently truncates to 10 partitions (same class as #205)

1 participant