Skip to content

refactor(cost): define scoped physical statistics contract - #326

Merged
zzylol merged 8 commits into
feat/analytical-resource-cost-323from
feat/cost-statistics-scope
Sep 3, 2026
Merged

zzylol merged 8 commits into
feat/analytical-resource-cost-323from
feat/cost-statistics-scope

Conversation

@zzylol

@zzylol zzylol commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

Analytical costing needs two independent guarantees:

  1. raw and post-ASAP plans describe the same data and query workload; and
  2. every physical algorithm receives the workload-dependent evidence required by its resource formula.

Operator names alone cannot provide that evidence. The cost of the same hash aggregate changes with input cardinality, decoded input bytes, distinct-group count, key width, and accumulator width. Likewise, logical bytes transferred between operators are not the same as physical bytes read from storage. Missing or inconsistent evidence must therefore make the entire candidate unavailable instead of becoming an optimistic zero or falling back to structural node counting.

This PR defines the scoped physical-statistics contract between workload/catalog evidence, physical lowering, and analytical resource estimation:

DataWorkload + QueryWorkload + catalog/runtime evidence
                              |
                              v
                   analytical_statistics.rs
            ComparisonScope + OperatorStatistics
                              |
                              v
                      analytical_cost.rs
                    ResourceEstimate

What changed

Comparison scope

ComparisonScope records the complete boundary that every alternative must share:

  • data-arrival mode;
  • planning time and finite comparison horizon;
  • query recurrence;
  • event-time selection;
  • physical sources;
  • provider-owned source_snapshot_id values, such as catalog versions, object generations, or snapshot timestamps;
  • canonical source predicates.

validate_comparison_scopes requires exact equality before estimates can be ranked. The resource estimator does not guess predicate subsumption or source coverage.

Sources of truth and physical-layer boundary

QueryExpr   ------+
                  +-- physical lowering --> PhysicalOperator DAG
SummaryExpr ------+                              |
                                                  v
                                         OperatorStatistics
                                                  |
                                                  v
                                                 cost
  • QueryExpr is authoritative for the original query semantics.
  • SummaryExpr is authoritative for logical summary semantics and the selected summary family.
  • PhysicalOperator is authoritative for the physical algorithms being costed.
  • OperatorStatistics corresponds one-to-one with PhysicalOperator and supplies workload/catalog evidence for those algorithms.

Neither logical IR is the statistics schema: one logical node may lower to several physical nodes or to different physical algorithms with different evidence requirements. This PR covers every PhysicalOperator currently defined by the analytical cost layer. It does not claim that every post-ASAP logical operation is already lowered. SummaryAgg, SummaryJoin, SummaryMerge, SummarySubtract, SummaryDelete, and SummaryEstimate require explicit physical realization before they can be costed. Until an operation has a physical operator, statistics contract, validation rules, and resource formula, its complete candidate is unavailable.

The new physical-plan integration design documents this pipeline, lowering obligations, post-ASAP coverage, physical identity, and fail-closed behavior. The analytical resource-cost design links to it while retaining the estimator-specific model.

Typed operator evidence

The former flat collection of optional fields is replaced by an internally tagged OperatorStatistics enum:

Scan { edges, source_read_bytes }
Filter { edges }
Project { edges }
HashAggregate {
    edges,
    group_count,
    key_bytes,
    accumulator_bytes_per_group,
}
InMemoryComparisonSort { edges }
TopK { edges }
HashJoin { edges }
HashDeduplicate { edges, distinct_key_count, key_bytes }
Concat { inputs, output }
InMemoryOrderedWindow { edges }
Limit { edges }
PassThrough { edges }

Unary variants carry UnaryEdgeStatistics; HashJoin carries ordered BinaryEdgeStatistics; and Concat is explicitly variadic. Serialized evidence rejects unknown fields. This makes invalid combinations unrepresentable: for example, a filter cannot carry group cardinality, a Top-K cannot carry join configuration, and only a scan can carry physical source-read bytes.

EdgeStatistics { rows, bytes } remains operator-independent because it describes the decoded logical data crossing a DAG edge. It is checked between each child's output and the corresponding parent input. Scan.source_read_bytes separately describes physical storage I/O, allowing a compressed scan to emit more logical bytes without charging the source scan again at its parent.

Plan configuration stays on physical operators

Configuration selected by lowering is not catalog/workload evidence and remains on PhysicalOperator:

Physical operator Plan-owned configuration
TopK limit and offset; heap capacity is limit + offset
Limit limit and offset
HashJoin explicit left/right build side

This removes generic statistics fields such as k, limit_rows_consumed, and an optional join build side. Consumption is derived from physical configuration and edge cardinality. Algorithm assumptions are also explicit in names such as InMemoryComparisonSort, HashDeduplicate, and InMemoryOrderedWindow; a different algorithm must add its own physical variant, evidence requirements, validation, and formula.

Explicit input and child arity

Statistics-input arity and physical-DAG child arity are different concepts and are validated separately:

Operator shape Statistics inputs DAG children
Scan 1 external source edge 0
Unary operator 1 1
HashJoin 2 ordered inputs 2 ordered children
Concat one per input one per child

The implementation exhaustively matches every PhysicalOperator; there is no wildcard default that silently assigns arity to a future operator. Adding a new operator therefore requires an explicit statistics variant, both arity definitions, semantic validation, and a resource formula.

Evidence provider and fail-closed validation

OperatorStatisticsProvider supplies one complete typed record for every reachable physical node and owns evidence provenance and freshness. One immutable snapshot is resolved per estimate.

The whole candidate becomes unavailable when any node is missing or when validation finds stale/inconsistent evidence, an operator/statistics variant mismatch, invalid arity, conflicting parent/child edges, missing source coverage, or a violated operator-specific invariant.

Estimation flow

  1. Build a ComparisonScope from canonical DataWorkload, QueryWorkloadEntry, the finite horizon, and source snapshot identities.
  2. Resolve every reachable physical node through one OperatorStatisticsProvider snapshot.
  3. Validate sources, variants, input and child arities, edge continuity, and operator-specific invariants.
  4. Derive execution multiplicity from query recurrence over the horizon.
  5. Pass only the validated DAG, scope, and evidence snapshot to analytical_cost.rs.
  6. Rank raw and post-ASAP resource vectors only after exact scope equality succeeds.

Result

Before this change, callers could combine unrelated optional facts, omit required physical configuration, charge source bytes at non-scan nodes, or apply a generic formula to an unsupported algorithm. After this change, evidence is shaped by the selected physical operator, required facts are structurally present, logical edge bytes and storage reads are distinct, all reachable edges are checked, and incomplete candidates fail closed.

Validation

  • cargo test -p asap-aware-mapping — 258 tests passed
  • cargo clippy -p asap-aware-mapping --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

The full local workspace build reached final linking before the runner's linker terminated with a bus error; the affected workspace targets do not consume this crate-private API. GitHub CI is authoritative for the full matrix.

Depends on #332. #327 consumes this contract while recursively lowering query DAGs.

@zzylol
zzylol force-pushed the feat/analytical-resource-cost-323 branch from af100ed to bbf72f7 Compare September 2, 2026 19:28
@zzylol
zzylol force-pushed the feat/cost-statistics-scope branch 2 times, most recently from 50b2a80 to fa19c73 Compare September 3, 2026 02:54
@zzylol
zzylol marked this pull request as ready for review September 3, 2026 03:09
@zzylol
zzylol requested a review from Selvomega September 3, 2026 15:28
@zzylol
zzylol force-pushed the feat/analytical-resource-cost-323 branch from 87e8e32 to 8fdfdbf Compare September 3, 2026 16:02
@zzylol
zzylol force-pushed the feat/cost-statistics-scope branch from fa19c73 to e1411fd Compare September 3, 2026 16:05
@zzylol zzylol changed the title feat(cost): validate analytical comparison evidence refactor(cost): type physical operator evidence and validate comparison scope Sep 3, 2026
@zzylol zzylol changed the title refactor(cost): type physical operator evidence and validate comparison scope refactor(cost): define scoped physical statistics contract Sep 3, 2026
@zzylol
zzylol merged commit 695f16d into feat/analytical-resource-cost-323 Sep 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant