refactor(cost): define scoped physical statistics contract - #326
Merged
zzylol merged 8 commits intoSep 3, 2026
Merged
Conversation
zzylol
force-pushed
the
feat/analytical-resource-cost-323
branch
from
September 2, 2026 19:28
af100ed to
bbf72f7
Compare
zzylol
force-pushed
the
feat/cost-statistics-scope
branch
2 times, most recently
from
September 3, 2026 02:54
50b2a80 to
fa19c73
Compare
zzylol
marked this pull request as ready for review
September 3, 2026 03:09
zzylol
force-pushed
the
feat/analytical-resource-cost-323
branch
from
September 3, 2026 16:02
87e8e32 to
8fdfdbf
Compare
zzylol
force-pushed
the
feat/cost-statistics-scope
branch
from
September 3, 2026 16:05
fa19c73 to
e1411fd
Compare
zzylol
force-pushed
the
feat/cost-statistics-scope
branch
from
September 3, 2026 16:29
e1411fd to
0b03b6e
Compare
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Analytical costing needs two independent guarantees:
Operator names alone cannot provide that evidence. The cost of the same hash aggregate changes with input cardinality, decoded input bytes, distinct-group count, key width, and accumulator width. Likewise, logical bytes transferred between operators are not the same as physical bytes read from storage. Missing or inconsistent evidence must therefore make the entire candidate unavailable instead of becoming an optimistic zero or falling back to structural node counting.
This PR defines the scoped physical-statistics contract between workload/catalog evidence, physical lowering, and analytical resource estimation:
What changed
Comparison scope
ComparisonScoperecords the complete boundary that every alternative must share:source_snapshot_idvalues, such as catalog versions, object generations, or snapshot timestamps;validate_comparison_scopesrequires exact equality before estimates can be ranked. The resource estimator does not guess predicate subsumption or source coverage.Sources of truth and physical-layer boundary
QueryExpris authoritative for the original query semantics.SummaryExpris authoritative for logical summary semantics and the selected summary family.PhysicalOperatoris authoritative for the physical algorithms being costed.OperatorStatisticscorresponds one-to-one withPhysicalOperatorand supplies workload/catalog evidence for those algorithms.Neither logical IR is the statistics schema: one logical node may lower to several physical nodes or to different physical algorithms with different evidence requirements. This PR covers every
PhysicalOperatorcurrently defined by the analytical cost layer. It does not claim that every post-ASAP logical operation is already lowered.SummaryAgg,SummaryJoin,SummaryMerge,SummarySubtract,SummaryDelete, andSummaryEstimaterequire explicit physical realization before they can be costed. Until an operation has a physical operator, statistics contract, validation rules, and resource formula, its complete candidate is unavailable.The new physical-plan integration design documents this pipeline, lowering obligations, post-ASAP coverage, physical identity, and fail-closed behavior. The analytical resource-cost design links to it while retaining the estimator-specific model.
Typed operator evidence
The former flat collection of optional fields is replaced by an internally tagged
OperatorStatisticsenum:Unary variants carry
UnaryEdgeStatistics;HashJoincarries orderedBinaryEdgeStatistics; andConcatis explicitly variadic. Serialized evidence rejects unknown fields. This makes invalid combinations unrepresentable: for example, a filter cannot carry group cardinality, a Top-K cannot carry join configuration, and only a scan can carry physical source-read bytes.EdgeStatistics { rows, bytes }remains operator-independent because it describes the decoded logical data crossing a DAG edge. It is checked between each child's output and the corresponding parent input.Scan.source_read_bytesseparately describes physical storage I/O, allowing a compressed scan to emit more logical bytes without charging the source scan again at its parent.Plan configuration stays on physical operators
Configuration selected by lowering is not catalog/workload evidence and remains on
PhysicalOperator:TopKlimitandoffset; heap capacity islimit + offsetLimitlimitandoffsetHashJoinThis removes generic statistics fields such as
k,limit_rows_consumed, and an optional join build side. Consumption is derived from physical configuration and edge cardinality. Algorithm assumptions are also explicit in names such asInMemoryComparisonSort,HashDeduplicate, andInMemoryOrderedWindow; a different algorithm must add its own physical variant, evidence requirements, validation, and formula.Explicit input and child arity
Statistics-input arity and physical-DAG child arity are different concepts and are validated separately:
ScanHashJoinConcatThe implementation exhaustively matches every
PhysicalOperator; there is no wildcard default that silently assigns arity to a future operator. Adding a new operator therefore requires an explicit statistics variant, both arity definitions, semantic validation, and a resource formula.Evidence provider and fail-closed validation
OperatorStatisticsProvidersupplies one complete typed record for every reachable physical node and owns evidence provenance and freshness. One immutable snapshot is resolved per estimate.The whole candidate becomes unavailable when any node is missing or when validation finds stale/inconsistent evidence, an operator/statistics variant mismatch, invalid arity, conflicting parent/child edges, missing source coverage, or a violated operator-specific invariant.
Estimation flow
ComparisonScopefrom canonicalDataWorkload,QueryWorkloadEntry, the finite horizon, and source snapshot identities.OperatorStatisticsProvidersnapshot.analytical_cost.rs.Result
Before this change, callers could combine unrelated optional facts, omit required physical configuration, charge source bytes at non-scan nodes, or apply a generic formula to an unsupported algorithm. After this change, evidence is shaped by the selected physical operator, required facts are structurally present, logical edge bytes and storage reads are distinct, all reachable edges are checked, and incomplete candidates fail closed.
Validation
cargo test -p asap-aware-mapping— 258 tests passedcargo clippy -p asap-aware-mapping --all-targets -- -D warningscargo fmt --all -- --checkgit diff --checkThe full local workspace build reached final linking before the runner's linker terminated with a bus error; the affected workspace targets do not consume this crate-private API. GitHub CI is authoritative for the full matrix.
Depends on #332. #327 consumes this contract while recursively lowering query DAGs.