Skip to content

feat(cost): lower raw query DAGs into evidence-backed physical plans - #327

Merged
zzylol merged 12 commits into
feat/cost-statistics-scopefrom
feat/cost-query-dag-stack
Sep 3, 2026
Merged

zzylol merged 12 commits into
feat/cost-statistics-scopefrom
feat/cost-query-dag-stack

Conversation

@zzylol

@zzylol zzylol commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

The analytical estimator can compose a physical DAG only after a selected raw query has been lowered completely. Shape-specific handling of Scan -> Aggregate or one Count/Top-K example misses nested operators, can lose edge cardinality, and cannot distinguish repeated logical occurrences from deliberately shared physical work.

The lowering must also preserve the work that changes a physical algorithm's cost. Two filters with different predicate work, a one-key and a three-key join, or a global and partitioned sort must not receive the same CPU/memory estimate merely because they share an operator name.

What

  • Recursively lowers the supported raw QueryExpr operator DAG into the existing PhysicalOperator vocabulary.
  • Carries provider-owned physical identity, output-buffer evidence, source coverage, and typed OperatorStatistics for every node.
  • Records query-owned physical work such as predicate/expression operations, grouping and accumulator counts, ordering keys, join keys, and deduplication keys on PhysicalOperator.
  • Adds per-partition evidence for in-memory comparison sort and SQL analytic-window operators.
  • Renames InMemoryOrderedWindow to InMemoryAnalyticWindow to distinguish SQL analytic execution from streaming tumbling/sliding/exponential-histogram windows.
  • Treats non-identity PromQL offset/@ as unavailable until temporal context can be propagated into descendant Scan evidence.
  • Fixes evidence invariants discovered while integrating lowering: non-empty Scan cannot claim zero source reads, Concat must have an input, and grouped/ungrouped aggregates handle empty input correctly.
  • Keeps unsupported logical shapes and missing or inconsistent evidence fail closed.

This PR does not lower SummaryExpr. Post-ASAP summary physical lowering and complete-alternative ranking remain separate stack layers.

How

resolved raw QueryExpr
        |
        | recursive lowering + physical algorithm configuration
        v
PhysicalOperator DAG ----> PhysicalNodeEvidenceProvider
        |                         |
        |                         +-- physical_id
        |                         +-- OperatorStatistics
        |                         +-- output_buffer_bytes
        v
validated PhysicalDag ----> analytical resource estimator

Every logical occurrence is independent by default. The evidence provider owns physical identity; two occurrences are charged once only when it deliberately returns the same non-empty physical_id with identical operator, children, source coverage, statistics, and buffer evidence. The lowerer does not equate Rc address identity with physical sharing.

Parent input rows/bytes must match the corresponding child output. Source storage bytes remain separate from decoded logical edge bytes. Evidence is snapshotted into the returned PhysicalDag, so costing does not reread a live provider.

For ordered operators, PartitionStatistics supplies the rows and logical bytes of every independently processed partition. The checked sum must equal the input edge. Global ordering requires exactly one partition; partitioned ordering computes:

CPU         = sum(n_i * ceil(log2(n_i)) * comparison work)
peak memory = max(partition_i logical bytes)

The physical integration design now includes a definition, query example, required evidence, and CPU/memory/I/O behavior for every physical node.

Supported raw mappings

QueryExpr shape Physical realization
Scan Scan, plus an explicit Filter for pushed predicates
Filter predicate-work-aware Filter
Project expression-work-aware Project
supported reducing aggregate without HAVING/without HashAggregate
Dedup HashDeduplicate
proven equi-join HashJoin with explicit build side and equality-key count
Concat or UNION ALL Concat
global or partitioned Sort evidence-backed InMemoryComparisonSort
global Sort followed by positive Limit bounded heap TopK
ordered RowNumber/Rank/DenseRank SQL window InMemoryAnalyticWindow
unordered Limit Limit
identity TimeShift PassThrough

Unsupported aggregate algorithms, HAVING, without grouping with runtime-unknown keys, cross/non-equi joins, distinct set operations, non-identity temporal shifts, PromQL range/subquery/vector operators, and post-ASAP summary operations remain unavailable until their physical implementations and evidence are explicit.

Before this PR

Only a few hand-written raw query shapes could reach analytical costing. Operator-local expression/key work and partition distribution were erased. A non-empty Scan could report zero disk reads, and ungrouped or empty-input aggregation evidence could not be represented correctly.

After this PR

For example:

SELECT service, COUNT(*) AS frequency
FROM metrics
WHERE status = 500 AND latency > 100
GROUP BY service
ORDER BY frequency DESC
LIMIT 10;

lowers recursively to:

Scan
  -> Filter { predicate_operations_per_row: 3 }
  -> HashAggregate { grouping_key_count: 1, accumulator_count: 1 }
  -> TopK { limit: 10, offset: 0, ordering_key_count: 1 }

Each edge has its own rows/bytes, only Scan charges physical source reads, the aggregate retains every group, and Top-K retains only its bounded heap. Any missing physical fact makes the complete query unavailable.

Validation

  • cargo test -p asap-aware-mapping — 270 tests passed
  • cargo clippy -p asap-aware-mapping --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

Depends on #326. #328 adds PromQL-specific physical operators and validation.

@zzylol
zzylol force-pushed the feat/cost-statistics-scope branch from 0176c51 to 50b2a80 Compare September 2, 2026 19:28
@zzylol
zzylol force-pushed the feat/cost-query-dag-stack branch from f1a650a to 3d0702a Compare September 2, 2026 19:28
@zzylol
zzylol force-pushed the feat/cost-statistics-scope branch from 50b2a80 to fa19c73 Compare September 3, 2026 02:54
@zzylol
zzylol force-pushed the feat/cost-query-dag-stack branch from 3d0702a to fa09dcf Compare September 3, 2026 02:54
@zzylol
zzylol marked this pull request as ready for review September 3, 2026 03:17
@zzylol
zzylol force-pushed the feat/cost-statistics-scope branch from fa19c73 to e1411fd Compare September 3, 2026 16:05
@zzylol
zzylol force-pushed the feat/cost-query-dag-stack branch from fa09dcf to 2c86288 Compare September 3, 2026 16:07
@zzylol
zzylol force-pushed the feat/cost-statistics-scope branch from e1411fd to 0b03b6e Compare September 3, 2026 16:29
@zzylol
zzylol force-pushed the feat/cost-query-dag-stack branch from 2c86288 to 0a91fbc Compare September 3, 2026 16:29
@zzylol
zzylol force-pushed the feat/cost-query-dag-stack branch from 0a91fbc to 1307e34 Compare September 3, 2026 18:38
@zzylol
zzylol force-pushed the feat/cost-query-dag-stack branch from 1307e34 to 228ea1e Compare September 3, 2026 18:47
@zzylol zzylol changed the title feat(cost): lower query DAGs into physical plans feat(cost): lower raw query DAGs into evidence-backed physical plans Sep 3, 2026
@zzylol
zzylol merged commit a1c19df into feat/cost-statistics-scope Sep 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant