Skip to content

fix(search): route un-LIMITed row queries through scroll so SELECT returns every row (#209) - #210

Merged
fupelaqu merged 2 commits into
mainfrom
fix/209-route-unlimited-select-to-scroll
Aug 8, 2026
Merged

fupelaqu merged 2 commits into
mainfrom
fix/209-route-unlimited-select-to-scroll

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Fixes #209 — implements the decided option 2: route un-LIMITed SELECTs to the scroll path internally.

What

A row query with no LIMIT through the one-shot search path emitted no top-level size, so
Elasticsearch returned its default 10 hits silently (45-row index → 10 arbitrary rows,
HTTP 200, no flag). The gateway executor already routed plain projections to scroll — which is
why the REPL rarely showed it — but the programmatic surface (search/searchAs/searchAsync)
always truncated, and so did the row shapes the gateway predicate misses (windowed and
script-field-only projections, whose SingleSearch.fields is empty).

How

  • SingleSearch (sql) gains the canonical response-shape discriminators:
    • windowRowQuery — the window-dispatch condition previously duplicated inline in search
      and searchAsync;
    • returnsRows — row-shaped (plain / script-field / window-enriched) vs aggregation-shaped
      (GROUP BY / metric-only).
  • SearchApi (core) — search and searchAsync route limit.isEmpty && returnsRows
    queries through the public scroll (complete paging via PIT/search_after per PIT + search_after injects _shard_doc as primary sort, defeating doc-id skip on ES 8+ #197; window
    enrichment and script fields already handled there), collecting rows into the same
    ElasticResponse. Aggregation-shaped queries are never routed — their result is the
    aggregation itself, bounded by explicit terms sizes since GROUP BY without LIMIT silently returns only the top 10 groups (silent wrong answer) #205/Window PARTITION BY silently truncates to 10 partitions (same class as #205) #207. The materializer is a
    JVM-shared daemonic ActorSystem (cannot keep the JVM alive; clients don't own it). The
    interception is a runtime this match { case s: ScrollApi => … } so no trait signature
    changes reach downstream repos.
  • CoreDqlExtension (core) — the licensed maxQueryResults cap (ADR D4 rule 2) widens
    from fields.nonEmpty to returnsRows: windowed/script-only no-LIMIT queries previously hit
    the "ES-bounded" searchAsync branch; now that they return every row they are capped at the
    quota with the truncation flag, exactly like plain projections. Without this, the routing
    would have opened a quota bypass.

Tests

  • ReturnsRowsSpec (sql): pins the discriminator on 8 statement shapes (plain, SELECT *,
    script-only, windowed, metric-only, GROUP BY, GROUP BY+window, LIMIT-independence).
  • New SelectCompletenessSpec (testkit + es6 rest, es6 jest, es7 rest, es8 java, es9 java):
    3-shard, 45-doc index (15 categories × 3, exact oracles):
  • sql 403, core 724, bridge 118 green; cross-compiled 2.12 + 2.13.

Found while validating (separate issue to propose)

script_fields values come back wrapped in Elasticsearch's per-field array on every path
(n -> List("CAT_02")): searchAs conversion of a script field to a scalar has always failed,
and no existing test ever content-asserted a script-field value. Pre-existing, independent of
the routing; the spec asserts the script-only case via the raw search API and the defect will
be filed separately.

Notes

  • MultiSearch (UNION ALL) without LIMIT is out of scope (multi-index; scroll does not support
    it) — unchanged behavior.
  • Un-LIMITed row queries that SELECT score() now ride scroll like the gateway already did for
    plain projections (no _score relevance on the PIT/_doc path) — consistent with the
    established gateway semantics.

🤖 Generated with Claude Code

fupelaqu and others added 2 commits August 8, 2026 09:52
…turns every row (#209)

A row query with no LIMIT executed through the one-shot search path
emitted no top-level size, so Elasticsearch returned its default 10 hits
silently - a 45-row index reported 10 arbitrary rows with HTTP 200. The
gateway executor already routed plain projections to scroll, but the
programmatic surface (search / searchAs / searchAsync) and the row
queries the gateway predicate misses (windowed / script-field-only,
whose `fields` is empty) all truncated.

SingleSearch gains the canonical response-shape discriminators:
windowRowQuery (the dispatch condition previously duplicated inline in
search and searchAsync) and returnsRows (row-shaped vs
aggregation-shaped). search/searchAsync now route no-LIMIT row-shaped
queries through the scroll path - which pages completely (PIT /
search_after, #197) and already handles window enrichment and script
fields - collecting into the same ElasticResponse. Aggregation-shaped
queries are never routed (their result is the aggregation, bounded by
explicit terms sizes since #205/#207). The materializer is a JVM-shared
daemonic ActorSystem, so it can never keep the JVM alive.

CoreDqlExtension's licensed result cap widens with the routing: rule (2)
now keys on returnsRows instead of fields.nonEmpty, so windowed and
script-only no-LIMIT queries - which previously hit the "bounded"
searchAsync branch and now return every row - are capped at
maxQueryResults with the truncation flag like plain projections.

Tests: ReturnsRowsSpec pins the shape discriminator on 8 statement
shapes. New SelectCompletenessSpec (testkit + es6 rest/jest, es7 rest,
es8/es9 java): 3-shard 45-doc index; plain, SELECT *, script-field-only
and window-enriched projections without LIMIT return all 45 rows; an
explicit LIMIT keeps its one-shot bound. Green on real ES
6.8/7.17/8.18/9.0. The script-field-only case is asserted via the raw
search API: script_fields values come back wrapped in Elasticsearch's
per-field array on every path - a pre-existing conversion defect
surfaced by this spec, tracked separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fupelaqu
fupelaqu merged commit b730cad into main Aug 8, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SELECT without LIMIT returns only 10 rows on the non-scroll search path

1 participant