docs: split design.md into per-layer docs (L1-L5) + a slim index - #157
Merged
Merged
Conversation
Base automatically changed from
docs/l4node-execution-model
to
feat/summary-executor-interface
July 22, 2026 21:14
zzylol
force-pushed
the
docs/split-l1-l2-l3-l5
branch
from
July 22, 2026 21:15
38badf0 to
7d44247
Compare
zzylol
force-pushed
the
docs/split-l1-l2-l3-l5
branch
from
July 22, 2026 21:19
7d44247 to
2cd2564
Compare
Contributor
Author
|
@milindsrivastava1997, please just review the design.md for overall convergence for now. Thanks! Before we flesh out the API/interface details per layer. |
milindsrivastava1997
left a comment
Collaborator
There was a problem hiding this comment.
A running example (however toy) will help.
zzylol
force-pushed
the
docs/split-l1-l2-l3-l5
branch
from
July 28, 2026 21:00
dbf01d2 to
ea1ffcc
Compare
Rebased onto main (now past #162/#164/#165/#166) after PR #157's original 37-commit history hit real conflicts with the intervening Reduction/L4 grouping-model work -- squashed to one commit here since the PR merges as a squash anyway, so the intermediate history wasn't worth preserving through a 37-way conflict resolution. `docs/design.md` was a 1500+ line document mixing layer design, migration planning, and crate-by-crate speculation, most of it already stale relative to the shipped code. This does the same move `l4node-execution- model.md` already made for L4 across the rest of the pipeline: each layer gets its own doc, `design.md` becomes a thin index into them, and every doc is cross-checked against current code rather than copied from old drafts. - New per-layer docs: `l1-query-language.md`, `l2-logical-plan.md`, `l3-intent-algebra.md`, `l5-physical-plan.md` (l4's already existed, renamed to `l4-summary-bound-ir.md` to match the shipped `SummaryExpr`/`SummaryKind`/`L4Node` naming instead of the older "sketch" framing). `l1-query-language.md` also absorbs the old standalone `promql-lowering.md` as a section. - `design.md` cut to a slim index: one paragraph per L1-L5 linking to its doc, plus a trimmed Glossary. - Deleted `docs/migration-plan.md` and `docs/intent-algebra- reconciliation.md` -- both were migration-era planning artifacts, the latter already self-annotated as a superseded historical snapshot. - `crates/sketch/src/exec.rs`'s doc comments repointed at the renamed `l4-summary-bound-ir.md`. Known gap carried over from the original PR (not introduced by this rebase): README.md still links to `docs/migration-plan.md` and to `docs/design.md`'s old §3/§6/§8/§9/§11/§12 section numbers, none of which exist after this change -- worth a follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 task
zzylol
added a commit
that referenced
this pull request
Jul 28, 2026
…167) * docs: fix README links left dangling by the design.md restructuring PR #157 cut design.md from 1500+ lines to a ~200-line index and deleted migration-plan.md, but never touched README.md, which still pointed at migration-plan.md and design.md's old §3/§6/§8/§9/§11/§12 section numbers -- none of which exist anymore. Repoints at the new per-layer docs (l1-query-language.md through l5-physical-plan.md) and this README's own "Consumption modes" section where that's where the content actually lives now; drops the migration-plan.md reference entirely since the file's gone and nothing replaced it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: align README with current crate layout and summary terminology Beyond the dangling doc links, the README had drifted from the actual codebase in several ways: - "sketch" framing (query->sketch pipeline, sketch algebra, sketch IR) predates the summary-bound-IR renaming docs/l4-summary-bound-ir.md already uses -- aligned the descriptive language to "summary" while leaving real identifiers (asap-sketch, SummaryKind, SummaryExpr) as-is. - L4's binding pass and the summary-vs-exact accuracy decision (issue #98) were described as "stub"/"planned" -- #98 has been closed for a while; bind.rs/boundary.rs/cost_model.rs are real, landed code. - asap-plan's Cargo.toml depends on asap-sketch (needs the L4 types for binding), not "asap-ir alone" as the isolation-wins bullet claimed. - asap-sketch gained exec.rs (the serving-time SummaryExecutor trait + execute() walker, ~700 lines) since the crate table and directory tree were last written -- both still described it as "type definitions only". - asap-e2e now also depends on asap-sketch/asap-plan (l4_binding.rs tests lowering through to L4), not just asap-frontend-promql. - Refreshed the ~LOC column against current `wc -l` per crate -- asap-sketch and asap-plan had roughly 4x and 7x more code than listed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
Jul 29, 2026
Fixes #168. PR #157 rewrote design.md + the per-layer docs to be pure principle/ design-level, stripped of code-level detail on purpose. This adds the promised follow-up: a concrete Rust API surface per layer, verified against current `main` rather than carried over from an old draft. - L1: no shared trait exists -- each front end exposes its own free functions (lower_promql*, lower_sql*), converging on the same L3 QueryExpr return type. Kept thin rather than padded, since that's the honest current shape. - L2: SchemaCatalog (the pluggable schema-source trait) + Binder + convert_root. - L3: QueryExpr's shape (a representative slice, not the full ~24-variant enum), Reduction + GroupKeys in full (small, central to this doc's own design principles), a representative AggIntent slice, and Schema. - L4: SummaryExpr/L4Node, the binding entry points + Implementation, the CostModel extension point (all default-having methods shown), and SummaryExecutor in full -- the real, load-bearing serving-time interface just resolved via #163/#164/#165 this session. - L5: nothing is built yet, so this is explicitly speculative -- recovered the PhysicalPlanner/TopologyDescriptor/StageAllocator sketch an earlier draft of this doc had (before PR #157's final rewrite trimmed it), updated to current summary-bound-IR naming (L4Node, SummaryKind) and kept the same "target to design against, not an API to depend on" framing so it can't be mistaken for shipped code. Every signature was checked against the actual current source in this repo (crates/frontend-promql, crates/frontend-sql, crates/l2, crates/ir, crates/sketch, crates/plan) rather than assumed from memory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Aug 5, 2026
* docs: add an Interface section to each L1-L5 doc Fixes #168. PR #157 rewrote design.md + the per-layer docs to be pure principle/ design-level, stripped of code-level detail on purpose. This adds the promised follow-up: a concrete Rust API surface per layer, verified against current `main` rather than carried over from an old draft. - L1: no shared trait exists -- each front end exposes its own free functions (lower_promql*, lower_sql*), converging on the same L3 QueryExpr return type. Kept thin rather than padded, since that's the honest current shape. - L2: SchemaCatalog (the pluggable schema-source trait) + Binder + convert_root. - L3: QueryExpr's shape (a representative slice, not the full ~24-variant enum), Reduction + GroupKeys in full (small, central to this doc's own design principles), a representative AggIntent slice, and Schema. - L4: SummaryExpr/L4Node, the binding entry points + Implementation, the CostModel extension point (all default-having methods shown), and SummaryExecutor in full -- the real, load-bearing serving-time interface just resolved via #163/#164/#165 this session. - L5: nothing is built yet, so this is explicitly speculative -- recovered the PhysicalPlanner/TopologyDescriptor/StageAllocator sketch an earlier draft of this doc had (before PR #157's final rewrite trimmed it), updated to current summary-bound-IR naming (L4Node, SummaryKind) and kept the same "target to design against, not an API to depend on" framing so it can't be mistaken for shipped code. Every signature was checked against the actual current source in this repo (crates/frontend-promql, crates/frontend-sql, crates/l2, crates/ir, crates/sketch, crates/plan) rather than assumed from memory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: address PR #169 review comments on the L3/L4/L5 interface sections - L3: clarify reduction/GroupKeys/Reduction aren't tree nodes (they're plain data reached only through Aggregate.reduction), and expand the AggIntent listing from a representative slice to the full vocabulary. - L4: explain what "Implementation" means (the per-intent binding decision's own result type); note the sketch/summary field-naming tension (sketch/sketch_input predate this doc's own "summary" terminology) and the Sketch/ExactAccumulator merge question as deliberate follow-ups, not silently resolved here -- see PR discussion. - L5: fold "what does edge mean", "StageId values are deployment-owned", and "interface only in ASAPController" into one clarification at the top of the Interface section, plus reword StageId's own comment so its example values don't read as a fixed vocabulary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): update Interface section now that sketch/summary rename + Implementation merge landed Both were called out as pending follow-ups in this PR's review; #170 implemented them. Updates the code snippets and prose here to match the real, current shape instead of describing the old one as still-pending. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l3): explain Group and CountValues in the AggIntent listing Review comment on #169: these two were listed with no explanation while every other category got at least a grouping comment. Both need one because they don't fit the "reduce to one number per group" shape every other reducer follows here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l5): actually explain what a StageEdge/edge is, not just who owns StageId values The earlier fix for this review comment conflated two different questions — "what does an edge mean" and "who owns the concrete StageId values" — and only answered the second. Adds the missing explanation: an edge is a declared connection in the topology's connectivity graph, separate from which stages merely exist. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: resolve 6 more PR #169 review comments — full listings, examples, constraints - l3: expand QueryExpr from an abbreviated slice to the full, real variant list (25 variants, matching crates/ir/src/intent_algebra/query_expr.rs). - l3: add a source-expression example per AggIntent variant, sourced from the accurate PromQL/SQL mappings already in agg_intent.rs's own doc comments. - l3: explain why the *OverTime range reducers can't be composed from Sort/Limit or any other generic node (no node exposes a TimeRange window's raw sample sequence as queryable rows). - l3: add a worked Schema example (sum by (job) (...)), showing the input/output shape difference an aggregate produces. - l2: add a concrete usage example for SchemaCatalog/Binder (PromQL's usage-derived default vs. a SQL catalog-backed case). - l4: list the per-SummaryExpr-node validity constraints (subtractable/ deletable/mergeable catalog flags, state-vs-value child requirements), correcting along the way which nodes actually carry a `summary` field of their own vs. which infer their kind from a child's output type. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): turn the per-SummaryExpr-node constraints into a table + a diagram Prose bullet list was hard to scan against 7 variants. Table gives a one-glance summary; the mermaid diagram visualizes the rule that's hardest to hold in your head from prose alone (SummaryMerge only accepts state-producing children, never a value-producing one). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): a taxonomy and recipe for composing summaries (bespoke) (#174) * docs(l4): design proposals for #171 and #172 composition gaps Stacked on #169's Interface section. Adds two concrete proposals: - #171 (exact <-> summary composition, either nesting order): a new SummaryFold node for an outer exact fold over an inner realized summary, computed generically by the shared execute walk with no new SummaryExecutor method; and a relaxed SummaryAgg input-column rule letting an outer summary read directly off an exact accumulator's own state (zero added error), while explicitly refusing the approximate- child case since that's #172's gap, not this one's. - #172 (nested approximate-over-approximate error propagation): a SummaryKind::implied_accuracy inverse-sizing function, a new CostModel::size_params child_accuracy parameter (opt-in, default ignores it, matching this trait's existing default-body convention), and a paired accepts_nested_approx gate so an outer approximate summary over an approximate child's estimate refuses by default instead of silently double-approximating. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): reframe #171/#172 proposal as a shared accuracy algebra, fold in #173 Replaces the earlier implementation-narrative version (which described specific match arms/functions in boundary.rs/bind.rs) with a design-level treatment: a formal (epsilon, delta) accuracy guarantee every SummaryKind already satisfies, a composition theorem (triangle inequality + union bound) for building a summary over another summary's output, and clean interfaces (Accuracy, ColumnRef::FromSummary, CostModel::size_params/ accepts_nested_approx, FoldOp/SummaryFold) derived from it. Shows #171 direction 2 (exact child) and #172 (approximate child) are the same composition formula at Accuracy::EXACT vs. Accuracy > 0, rather than two separate rules. Folds in #173: generalizes FromSummary/SummaryFold to N children for Hydra-style sketch-of-sketches, and adds a Bounded Accuracy variant (deterministic interval radius) for QTree-style range trees alongside the Probabilistic (epsilon, delta) variant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): separate the design from which issue it solves; norm-aware composition Two changes, both from review: - Restructure into "the design" (guarantee, composition theorem, three interfaces — no issue numbers threaded through the reasoning) followed by a single "Which issue this solves" section mapping each piece back to #171/#172/#173, instead of interleaving issue callouts throughout. - The composition theorem assumed a single scalar Lipschitz constant L under an implicit pointwise/L1 error model. That's wrong for this catalog's own CountSketch/CountSketchWithHeap, whose guarantee is stated against the L2 norm of the residual frequency vector, not L1 -- a materially different, not-directly-comparable quantity on skewed data. Accuracy and compose now carry an explicit ErrorNorm; compose returns None on a norm mismatch instead of silently assuming L=1. Folded Accuracy::Bounded (deterministic radius, for #173's QTree case) into the same enum/compose function rather than a separate type. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): replace unified accuracy algebra with three grounded composition patterns Surveyed how real systems actually compose summaries (DGIM SICOMP'02, UnivMon SIGCOMM'16, Hydra VLDB'22, PromSketch VLDB'25) rather than inventing one generic accuracy algebra. Finding: there is no single law -- three structurally distinct patterns exist, only two have a governing theorem, and the pattern #171/#172 actually describe has none at all. - Pattern A (same statistic across partitions -- EH/DGIM windowing): real, DGIM Theorem 6/7 grounds it precisely, but composes the same statistic across a partition of the same data, not different statistics along a query's data-flow -- out of scope for #171/#172/#173, flagged as a separate future gap (sliding-window range-vector queries). - Pattern B (hash-collision routing -- Hydra's sketch of sketches): Theorem 2's asymmetric bound (multiplicative + additive-vs-global-total) doesn't fit the shared Accuracy shape -- Hydra and QTree each need their own SummaryKind fed directly from raw data, not a generalization of FromSummary/SummaryFold. This replaces the previous (incorrect) N-ary FromSummary framing for #173. - Pattern D (heterogeneous sequential nesting -- what #171/#172 actually ask about): no borrowed theorem exists for this shape in any of the four systems surveyed -- all four avoid it structurally. Split into D1 (inner exact, composes free -- #171 direction 2), D2 (outer exact fold over inner summary -- #171 direction 1, SummaryFold), and D3 (outer sketch over an inner statistic with no exact sub-linear form -- #172's actual remaining scope, narrower than "any nested approximation"). D3's compose()/accepts_nested_approx gate is now labeled honestly as this design's own first-principles proposal, not a literature result -- defaulting closed matches what every system surveyed actually does (none of them re-sketch another sketch's readout). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): recipe (not formula) for compound summary bounds, state not readout Reframes the design around the actual structural move every surveyed system (DGIM, UnivMon, Hydra, PromSketch) makes: never compose past a readout -- build the outer structure directly over the inner's state. Replaces the "three patterns" framing with a four-type taxonomy (same- kind merge; inner-exact; heterogeneous-with-derivable-bound; heterogeneous- undecidable) and, for the interesting type-3 case, a four-step recipe for deriving a compound bound instead of a formula: (1) treat inner's state as the outer's input schema, (2) check the outer's own construction algorithm can actually run over that state, (3) if so, re-derive -- not reuse -- the outer's own concentration argument against the composed randomness, (4) accept the resulting bound is pair-specific, not universal (DGIM's (1+e)^2*Cf^2/k+Cf-1+e and Hydra's Gi(1+/-e)+eps*GS are worked examples of the same recipe, not the same formula). Drops the generic Accuracy::compose()/Sensitivity mechanism from the previous draft -- it implied a bound could be computed from kind- independent accuracy values alone, which the DGIM/Hydra worked examples show is false. CostModel::size_params_composed now takes the child's concrete (kind, params) directly and has no default body: the deployment supplies its own pair-specific derivation, or the composition is refused (accepts_composed_state defaults false). FromSummary(Rc<L4Node>) is now constrained to state-producing nodes only (SummaryAgg/SummaryMerge), reusing the same rule this doc's SummaryMerge table already enforces, rather than allowing reference to a readout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): restore readout-based composition as a distinct, legitimate mechanism Correction: the previous commit banned composing over a readout entirely, requiring FromSummary to reference only state-producing nodes. That over-corrected -- readout-based composition (mechanism 3a) is a real, general mechanism in its own right: any inner reporting a public (eps, delta) can be composed via a Lipschitz-style Sensitivity argument (triangle inequality + union bound), without needing access to the inner's raw state at all. It's strictly looser than state-based composition (3b, the DGIM/Hydra recipe) when 3b is available, but it's the only option when the inner's state genuinely isn't accessible (e.g. across a service boundary) or when 3b's construction-compatibility check fails. ColumnRef now has two variants -- FromReadout (value-producing node) and FromState (state-producing node) -- routing to two separate CostModel gates: accepts_readout_composition/sensitivity_for/compose_over_readout for 3a, accepts_state_composition/size_params_from_state for 3b. A deployment facing a novel (outer, inner) pair has a real choice between the two, not one closed gate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): drop the "Which issue this solves" mapping section Requested in review. The design (four compound types, mechanisms 3a/3b) stands on its own; the issue-by-issue mapping was redundant restatement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): tighten prose, cut redundant explanation No content change -- same taxonomy, same two mechanisms, same recipe, same interfaces. Trims repeated justification and meta-commentary about prior review rounds, shortens sentences, removes duplicate restatement of the DGIM/Hydra examples across steps. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): make the 3b recipe steps readable -- one idea per step, examples in a table The four recipe steps each interleaved DGIM and Hydra inline, forcing readers to track two examples at once inside every sentence. Splits the steps (now plain, one idea each) from the worked examples (DGIM/EH, Hydra, and a KLL-over-HLL counter-example), laid out as a comparison table keyed to steps 2 and 3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(l4): rename DGIM/EH to Exponential Histogram [Datar et al., SICOMP'02] Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/design.mdwas a 1500+ line document mixing layer design, migrationplanning, and crate-by-crate speculation, most of it already stale
relative to the shipped code. This PR does the same move
l4node-execution-model.mdalready made for L4 across the rest of the pipeline: each layer gets its
own doc,
design.mdbecomes a thin index into them, and every doc iscross-checked against current code rather than copied from old drafts.
l1-query-language.md,l2-logical-plan.md,l3-intent-algebra.md,l5-physical-plan.md(l4's already existed).l1-query-language.mdalso absorbs the old standalonepromql-lowering.mdas a section.real staleness — SQL now parses via DataFusion's own front end with 3
dialects (
DataFusionSQL/ClickhouseSQL/ElasticSQL), not a baresqlparser wrap; both front ends emit the same
asap-l2::relational::QueryExprtoday, not one enum per language as an earlier draft assumed;
Schema'sreal field is
columns(notfields) and gained aclosedflag thedocs never mentioned;
unique_keyspropagation table had several wrongrows (
Projectwas listed as pass-through when it resets keys,Joinas inheriting the left side's when it always resets);
unique_keys'consumer was mis-attributed to a
CostModel::workload_costmethod thatwas never built.
ColumnId-bindingdesign that had ended up in
l1-query-language.mdis really L2/L3material (name resolution lives in the Binder, which is
asap-l2; whypositions and
unique_keysexist isasap-ir's IR design) — moved intol2-logical-plan.mdandl3-intent-algebra.mdrespectively.l1nowstays focused on parsing.
design.mdcut to ~115 lines: one paragraph per L1-L5 linking to itsdoc, plus a trimmed Glossary. Everything else (Principles, repo layout,
crate-by-crate details, the end-to-end worked example, wire protocols,
Cargo surface, open questions, future work, success criteria) is gone —
either redundant with the per-layer docs or migration/roadmap content
with no doc left to hold it now that
migration-plan.mdis deleted too.docs/migration-plan.mdanddocs/intent-algebra-reconciliation.md— both were migration-era planning artifacts, the latter already
self-annotated as a superseded historical snapshot.
where it describes the L4 IR/physical-choice layer, matching the shipped
SummaryExpr/SummaryKind/L4Nodetype names instead of the older"sketch" framing.
Rebased directly onto
mainnow that #155/#156 are both merged.Test plan
N/A — docs only.
🤖 Generated with Claude Code