docs(asap-aware-mapping): flesh out ReplacementStrategy/search design, open issue #33 sub-issues - #258
Conversation
…, link issue #33 sub-issues docs/asap_aware_mapping.md already stubbed TargetSubDAG/ReplacementSubDAG/ ReplacementStrategy/CostModel and a Volcano/Cascades-style search pseudocode as "not yet implemented" (since #206/#211). This turns that stub into an actionable design and ties each piece to a tracked sub-issue of #33: - #251: generalize today's single-pick decisions (boundary::implementation_for, cse::share_common_subtrees) into real ReplacementStrategy impls that report every valid candidate instead of collapsing to one. - #252: the Cascades/Volcano-style search engine itself — MEMO-based candidate plan space (not a flat plan list), deduped via pre_asap::cse's existing structural-hash/InternTable machinery, sorted by CostModel. Explicitly reconciled against docs/cse-cost-model-decision.md (#237), which scoped a narrower binary share/don't-share decision away from full search infrastructure — #252 is the multi-axis case that decision itself flagged as the reason a real engine would eventually be needed. - #253: semantic-equivalent rewriting (avg -> sum/count) as a ReplacementStrategy, no longer needing its own bespoke before/after-CSE heuristic once a real search exists to let both forms compete on cost directly. - #254: group-by-lattice roll-up reuse (AHA vs. independent per-subpopulation treatment) as a ReplacementStrategy, gated on agg_is_mergeable and Schema::unique_keys the same way CSE's own legality gate is. - #256: GroupingStrategy::{PerSubpopulationInstance, SharedMultiSubpopulation} (Hydra) as a new axis orthogonal to SketchKind/SamplingKind/..., named for sharing across a query's own subpopulations (not multi-tenant deployment isolation). - #257: rebuild applicability.rs (#247) as a view over the search's candidate space instead of a parallel tree-walking system. Also resolves the doc's stale "(TODO: is this implemented?)" on parameter sizing — boundary::implementation_for's bind_summary_with already sizes every candidate via CostModel::size_params. No code change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
milindsrivastava1997
left a comment
milindsrivastava1997
left a comment
There was a problem hiding this comment.
@zzylol pls see review. Also see my most recent commit where I made some changes for readability.
…259 comments - MEMO-sharing bullet: spell out why flat plan lists blow up and how Rc-sharing avoids it, without leaning on jargon. - Drop the docs/cse-cost-model-decision.md (#237) history/scope-change framing; keep just the forward-looking design rationale. - Applicability-reporting paragraph: explain MEMO groups concretely instead of the abstract 'candidate-plan space' framing.
Updated the documentation for ASAP-aware mapping to improve clarity and structure. Enhanced sections on key concepts, goals, and design principles, and corrected formatting issues.
Added glossary terms and clarified definitions related to ASAP-aware mapping, including distinctions between alternatives and candidate plans.
Companion to docs/reorganize-into-folders (main) and docs/replacement-strategy-search-design (#258), which move the design docs into docs/design_docs/ and docs/user-guide.md into docs/user-guide/. All three branches converge on the same final docs/ layout. Updated the link to the design doc to point at its eventual docs/design_docs/asap_aware_mapping.md location -- note this link is only valid once this branch merges after (or alongside) the reorg; on this branch alone, asap_aware_mapping.md is still at its old top-level docs/ path, since moving it is #258's change, not this one's.
|
|
||
| A **Candidate Plan** is a complete post-ASAP plan formed by choosing a compatible set of alternatives across the plan. | ||
|
|
||
| For example, one candidate plan may choose: |
There was a problem hiding this comment.
The following block is confusing. It does not look like a real plan... Is it the "strategies activated" for the plan?
| ## Pseudocode for Replacement Plan Searching (not yet implemented) | ||
| ## Overview | ||
|
|
||
| ASAP-aware mapping decides **whether and how a query intent can be answered using summaries instead of scanning raw data**. |
There was a problem hiding this comment.
Then how is data scan considered? Are you implicitly assuming pre-computation in your articulation?
|
|
||
| The **Cost Model** estimates the trade-offs of candidate plans. | ||
|
|
||
| Depending on the planning stage, costs may include: |
There was a problem hiding this comment.
I cannot understand the following taxonomy... Lacking some context
|
|
||
| The central design principle is that ASAP-aware mapping should **preserve alternatives long enough to reason about their interactions**. | ||
|
|
||
| Suppose a plan contains several independent-looking decision points: |
There was a problem hiding this comment.
For now we can assume independency. I doubt this won't be true in the future. My general feeling is that currently all planning logics we all have are all ad-hoc logics each of us come up with.
Eventually we need to bake in more existing ideas...
| while new_plans != [] | ||
| The planner should represent common structure once and attach alternatives only at the decision points where plans differ. | ||
|
|
||
| Conceptually, each decision point forms an **alternative group** containing its local choices: |
There was a problem hiding this comment.
Where do "alternative groups" come from? Is each group derived by branches of one optimization rule?
| ## Summary mapping | ||
| The planner should evaluate alternatives at the plan level because several dimensions interact: | ||
|
|
||
| ```text |
There was a problem hiding this comment.
Again I feel this taxonomy is a little bit too ad-hoc. But I admit it already has enough potential to cover my current algo.
hmmmm May be already enough for an MVP
|
|
||
| --- | ||
|
|
||
| # Cross-Query Optimization |
There was a problem hiding this comment.
The relation between cross-query and summary is clear in terms of sharing summaries across queries.
However sharing can also be achieved by sharing data stream (say, two KLL on different sub-population on the same source) or by sharing operators (say, a Projection generating the union of columns from two individual projections). I just do not see the relationship between the summary story and this kind of sharing.
Companion to docs/reorganize-into-folders (main) and docs/replacement-strategy-search-design (#258), which move the design docs into docs/design_docs/ and docs/user-guide.md into docs/user-guide/. All three branches converge on the same final docs/ layout. Updated the link to the design doc to point at its eventual docs/design_docs/asap_aware_mapping.md location -- note this link is only valid once this branch merges after (or alongside) the reorg; on this branch alone, asap_aware_mapping.md is still at its old top-level docs/ path, since moving it is #258's change, not this one's.
|
|
||
| ### 2. Explore interactions between alternatives | ||
|
|
||
| Optimization choices cannot always be made independently. |
There was a problem hiding this comment.
"Optimization choices cannot always be made independently. " What does this mean?
| - whether the result can be shared with another query, | ||
| - whether multiple group-by levels can be computed through roll-up, | ||
| - whether subpopulations share one summary or use separate summaries, | ||
| - whether a semantic rewrite exposes additional sharing, |
There was a problem hiding this comment.
I actually thought your first MVP can delay of the semantic rewriting and sharing with another sub-query.
| rewrite avg as sum/count | ||
| ``` | ||
|
|
||
| Another candidate plan may make different choices at any of these decision points. |
There was a problem hiding this comment.
It is probably better to define what exactly the output of post-ASAP plan format is.
4 participants
Part of #33.
What
docs/asap_aware_mapping.mdhas stubbedTargetSubDAG/ReplacementSubDAG/ReplacementStrategy/CostModeland a Volcano/Cascades-style search pseudocode as "not yet implemented" since #206/#211. This PR turns that stub into an actionable design and opens the sub-issues that will actually implement it, per our discussion on issue #33 going beyond the original comment's four asks into a full framework: enumerate every applicable optimization and every applicable summary/degree-of-freedom for a workload, via a real search over candidate plans.Sub-issues opened (all linked as GitHub sub-issues of #33)
boundary::implementation_for,cse::share_common_subtrees) into realReplacementStrategyimpls that report every valid candidate instead of collapsing to onepre_asap::cse's existing structural-hash/InternTablemachinery, sorted byCostModelavg→sum/count) as aReplacementStrategyReplacementStrategyGroupingStrategy::{PerSubpopulationInstance, SharedMultiSubpopulation}(Hydra vs. one-sketch-per-subpopulation), named for sharing across a query's own subpopulations rather than multi-tenant deployment isolationapplicability.rs(feat(asap-aware-mapping): optimization-applicability rule framework #247) as a view over the search's candidate space instead of a parallel tree-walking systemOn
docs/cse-cost-model-decision.md(#237)That decision deliberately chose a direct cost comparison over full Volcano/Cascades infrastructure for one binary, single-candidate-pair decision (share a CSE'd subtree or don't), reasoning "this repo has no plan-enumeration/DP-search engine anywhere" and didn't need one for that narrow question. #252 is where that stops being true, for the reason #237 itself named: once multiple interacting axes exist at once, a per-decision-point heuristic can't see interactions across sites the way a real candidate-plan search can.
CostModel::cse_share_decisionisn't thrown away — it becomes the cost function backing the CSE strategy's two candidates inside the new engine. This PR's doc changes call that out explicitly so the scope change reads as intentional, not as ignoring #237.Also
Resolves the doc's stale "(TODO: is this implemented?)" on parameter sizing —
boundary::implementation_for'sbind_summary_withalready sizes every candidate viaCostModel::size_params.No code change — this PR only updates the design doc and opens tracking issues; #251/#252/#253/#254/#256/#257 are where the actual implementation lands.
Checks
🤖 Generated with Claude Code