diff --git a/README.md b/README.md index e66a5d0d..6ed0603c 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,16 @@ Step 2 **maps** query workload semantics to ASAP primitives. Adding a new ASAP p 3. **Mapping intents to ASAP primitives** — decide whether and how each intent can be answered by a summary, and select/size the corresponding summary family. +## Building + +```sh +cargo build +cargo test --workspace +``` + +No external setup required. See [`docs/user-guide.md`](docs/user-guide.md) for how to run a +query through the pipeline. + ## Glossary Let us define a few terms. @@ -139,13 +149,12 @@ topk( ## Unified intent ```text -TopK( - k = 10, - key = FieldRef("service"), - measure = Count, - input = Filter( - predicate = region = "us-east", - input = Scan("metrics") +Aggregate( + reduction = Reduce(by = [service]), + measures = [TopK { k = 10 }], + child = Filter( + pred = region = "us-east", + child = Scan("metrics") ) ) ``` @@ -169,13 +178,12 @@ LIMIT 5; ## parse + canonicalize ```text -TopK( - k = 5, - key = FieldRef("service"), - measure = Count, - input = Filter( - predicate = region = "us-east", - input = Scan("metrics") +Aggregate( + reduction = Reduce(by = [service]), + measures = [TopK { k = 5 }], + child = Filter( + pred = region = "us-east", + child = Scan("metrics") ) ) ``` @@ -185,7 +193,7 @@ Equivalent PromQL converges to the same structure. ## ASAP-aware mapping ```text -TopK(Count, service, 5) +Aggregate(reduction = Reduce(by = [service]), measures = [TopK { k = 5 }], ...) ↓ SpaceSaving(k=5) ``` @@ -214,7 +222,7 @@ semantic information that matters downstream. - Remove legacy data structures and types (tracked in [#179](https://github.com/ProjectASAP/ASAPPlanner/issues/179), [#205](https://github.com/ProjectASAP/ASAPPlanner/issues/205) - Implement the ASAP-aware mapping [logic and interfaces](docs/asap_aware_mapping.md) - Connect output of ASAPPlanner to asap-fusion -- Connect output of ASAPPlanner to ASAPPlanner and ASAPQuery (see open question #1 below) +- Connect output of ASAPPlanner to ASAPCollector and ASAPQuery (see open question #1 below) # Next steps @@ -227,13 +235,11 @@ semantic information that matters downstream. # Open questions 1. **Integration with downstream artifacts:** How to connect the output of ASAPPlanner to ASAPQuery? ASAPPlanner produces a post-ASAP plan that has semantics of batch query execution over data at rest. Somehow this needs to be converted into two plans (1) streaming dataflow graph that computes summaries on raw data, and (2) batch query execution plan that uses summaries to answer queries. -2. **Time semantics:** Should `TimeWindow` be an explicit node, or should time restriction be - represented as a specialized predicate? -3. **Grouping semantics:** Should grouping remain embedded in `Aggregate`, or should grouping +2. **Grouping semantics:** Should grouping remain embedded in `Aggregate`, or should grouping become a reusable relational dimension node? -4. **Expression semantics:** Which arithmetic or derived expressions need dedicated semantic +3. **Expression semantics:** Which arithmetic or derived expressions need dedicated semantic nodes because they materially affect summary selection? -5. **Approximation contracts:** Should accuracy/error requirements be fields on the intent, +4. **Approximation contracts:** Should accuracy/error requirements be fields on the intent, the workload, or the measure itself? -6. **Summary composability:** How should nested intents describe summaries that can be merged, +5. **Summary composability:** How should nested intents describe summaries that can be merged, transformed, or reused across queries? diff --git a/docs/parse_and_canonicalize.md b/docs/parse_and_canonicalize.md index 2208cd8d..b111f973 100644 --- a/docs/parse_and_canonicalize.md +++ b/docs/parse_and_canonicalize.md @@ -8,12 +8,12 @@ Examples: ```text PromQL topk(10, count by (service) (...)) - -> TopK(k=10, key=FieldRef("service"), measure=Count, ...) + -> Aggregate(reduction=Reduce(by=[service]), measures=[TopK{k=10}], ...) ``` ```text SQL ORDER BY COUNT(*) DESC LIMIT 10 - -> TopK(k=10, key=..., measure=Count, ...) + -> Aggregate(reduction=Reduce(by=[...]), measures=[TopK{k=10}], ...) ``` ## Canonicalize @@ -32,10 +32,11 @@ Aggregate -> Sort(desc) -> Limit(k) can canonicalize into: ```text -TopK(k, key, measure, Aggregate(...)) +Aggregate(reduction=Reduce(by=[key]), measures=[TopK{k}], ...) ``` -when the ordering expression and limit form the semantics of a top-k request. +when the ordering expression and limit form the semantics of a top-k request — the +heavy-hitter intent becomes an `AggIntent::TopK` measure on `Aggregate`, not a separate node. > Canonicalization should be driven by **semantic equivalence and summary relevance**, not by trying to reproduce every relational operator in a universal AST. diff --git a/docs/pre-asap-ir.md b/docs/pre-asap-ir.md index c8a45a73..60fb4a7b 100644 --- a/docs/pre-asap-ir.md +++ b/docs/pre-asap-ir.md @@ -361,7 +361,7 @@ SELECT DISTINCT srcip, dstip FROM packets ### Join -Logical join; the physical strategy (hash/merge/broadcast) is picked at L4. SQL `JOIN`. +Logical join; the physical strategy (hash/merge/broadcast) is picked in the post-ASAP IR. SQL `JOIN`. ```sql SELECT u.prefix FROM bgp_updates u JOIN bgp_rib_state r ON u.prefix = r.prefix diff --git a/docs/user-guide.md b/docs/user-guide.md index 2fd2230a..1cda23fc 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -13,13 +13,13 @@ promql> quantile(0.99, rate(http_requests_total[5m])) sql> SELECT service, COUNT(*) FROM metrics GROUP BY service ``` -**Pre-ASAP IR** (L3 — the sketch-agnostic intent algebra: `QueryExpr`/`AggIntent`): +**Pre-ASAP IR** (ASAP-agnostic `QueryExpr`/`AggIntent`): ```sh cargo run -p asap-devtools --bin show_pre_asap_ir -- queries.txt ``` -**Post-ASAP IR** (L4 — the sketch-bound IR: `SummaryExpr`/`L4Node`, one layer downstream, with +**Post-ASAP IR** (ASAP-aware IR with `SummaryExpr`/`L4Node`, one layer downstream, with concrete `SummaryKind`/`SummaryParams` committed per aggregate): ```sh @@ -61,7 +61,7 @@ asap-frontend-promql = { git = "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/ProjectASAP/ASAPPlanner", pac asap-aware-mapping = { git = "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/ProjectASAP/ASAPPlanner", package = "asap-aware-mapping" } ``` -### Step 1 — get the pre-ASAP IR (L3) +### Step 1 — get the pre-ASAP IR Lower a query string with a front end. Front ends never depend on each other or on the binder — pull only the one you need. @@ -70,7 +70,7 @@ pull only the one you need. use asap_frontend_promql::lower_promql; use asap_types::types::AccuracyTarget; -let l3 = lower_promql( +let pre_asap = lower_promql( "quantile(0.99, rate(http_requests_total[5m]))", AccuracyTarget::Epsilon(0.01), )?; // QueryExpr @@ -82,7 +82,7 @@ describing your tables; see `crates/devtools/src/bin/show_pre_asap_ir.rs` for a `AccuracyTarget` travels with the query, not the crate — pass `Exact` for no approximation allowed, `Epsilon(e)` / `EpsilonDelta{epsilon, delta}` otherwise. -### Step 2 — get the post-ASAP IR (L4) +### Step 2 — get the post-ASAP IR Feed the `QueryExpr` to `asap-aware-mapping`. This crate depends only on `asap-types`, never on a front end, so it's agnostic to which language produced the tree. @@ -90,7 +90,7 @@ front end, so it's agnostic to which language produced the tree. ```rust use asap_aware_mapping::implement_tree; -let l4 = implement_tree(&l3)?; // Rc — the SummaryExpr DAG +let post_asap = implement_tree(&pre_asap)?; // Rc — the SummaryExpr DAG ``` Two entry points, both re-exported from the crate root: @@ -98,7 +98,7 @@ Two entry points, both re-exported from the crate root: - `implementation_for(&AggIntent) -> Implementation` — the single-node decision (sketch, exact accumulator, or pass-through) for one aggregation. - `implement_tree(&QueryExpr) -> Result, ImplementError>` — walks a whole tree, calling - the per-node decision at every `Aggregate` and emitting the full L4 DAG. + the per-node decision at every `Aggregate` and emitting the full post-ASAP DAG. Each has a `_with(..., &dyn CostModel)` variant. A `CostModel` is the extension point for a deployment that wants its own candidate ranking or parameter sizing instead of this crate's @@ -108,7 +108,7 @@ overridable hooks (`rank_candidates`, `size_params`, `realize_extension`). ### Reading the result -Match on `l4.expr` (a `SummaryExpr`): +Match on `post_asap.expr` (a `SummaryExpr`): - `Logical(Box)` — this subtree wasn't rewritten; execute it exactly. - `SummaryAgg { summary, params, .. }` — an exact accumulator (`summary.is_exact()`) or an diff --git a/tools/dag-viewer/README.md b/tools/dag-viewer/README.md index d2007528..fb38a820 100644 --- a/tools/dag-viewer/README.md +++ b/tools/dag-viewer/README.md @@ -1,12 +1,12 @@ # ASAP query DAG viewer -Interactive viewer for the L3 query IR (`QueryExpr`), for manual IR +Interactive viewer for the pre-ASAP query IR (`QueryExpr`), for manual IR review/debugging, eyeballing common shapes across a corpus, and spotting shared sub-DAGs across queries. ## Generate a graph ```sh -cargo run -p asap-lower --bin dag_export -- \ +cargo run -p asap-devtools --bin dag_export -- \ --sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \ --sql "SELECT service, AVG(latency) FROM metrics GROUP BY service" --name q2 \ --promql "topk(5, rate(http_requests_total[5m]))" --name q3 \ @@ -58,7 +58,7 @@ proposed scope for a v2. The highlight is computed by hashing each node's `(kind, detail, children)` bottom-up and matching hashes across queries — see the doc comment on -`crates/ir/src/dag_export.rs`. It is **not** driven by +`crates/types/src/dag_export.rs`. It is **not** driven by `asap_plan::cse::dedupe_subtrees`: that pass isn't wired into any end-to-end multi-root planning path today (no caller outside its own unit tests), so there's nothing yet that would emit a real `CseWorkloadPlan` to visualize. diff --git a/tools/dag-viewer/RUNNING.md b/tools/dag-viewer/RUNNING.md index 35b30234..37559a9e 100644 --- a/tools/dag-viewer/RUNNING.md +++ b/tools/dag-viewer/RUNNING.md @@ -18,7 +18,7 @@ the script and edit the `--sql`/`--promql` lines to try your own instead. For more control, call the underlying binary directly: ```sh -cargo run -p asap-lower --bin dag_export -- \ +cargo run -p asap-devtools --bin dag_export -- \ --sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \ --promql "topk(5, rate(http_requests_total[5m]))" --name q2 \ > tools/dag-viewer/dag.json diff --git a/tools/dag-viewer/generate-sample.sh b/tools/dag-viewer/generate-sample.sh index 1444f20f..693bb0db 100755 --- a/tools/dag-viewer/generate-sample.sh +++ b/tools/dag-viewer/generate-sample.sh @@ -5,7 +5,7 @@ set -euo pipefail cd "$(dirname "${BASH_SOURCE[0]}")/../.." -cargo run -p asap-lower --bin dag_export -- \ +cargo run -p asap-devtools --bin dag_export -- \ --sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \ --sql "SELECT service, AVG(latency) FROM metrics GROUP BY service" --name q2 \ --promql "topk(5, rate(http_requests_total[5m]))" --name q3 \