Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 28 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,16 @@ Step 2 **maps** query workload semantics to ASAP primitives. Adding a new ASAP p
3. **Mapping intents to ASAP primitives** — decide whether and how each intent can be answered by a summary,
and select/size the corresponding summary family.

## Building

```sh
cargo build
cargo test --workspace
```

No external setup required. See [`docs/user-guide.md`](docs/user-guide.md) for how to run a
query through the pipeline.

## Glossary

Let us define a few terms.
Expand Down Expand Up @@ -139,13 +149,12 @@ topk(
## Unified intent

```text
TopK(
k = 10,
key = FieldRef("service"),
measure = Count,
input = Filter(
predicate = region = "us-east",
input = Scan("metrics")
Aggregate(
reduction = Reduce(by = [service]),
measures = [TopK { k = 10 }],
child = Filter(
pred = region = "us-east",
child = Scan("metrics")
)
)
```
Expand All @@ -169,13 +178,12 @@ LIMIT 5;
## parse + canonicalize

```text
TopK(
k = 5,
key = FieldRef("service"),
measure = Count,
input = Filter(
predicate = region = "us-east",
input = Scan("metrics")
Aggregate(
reduction = Reduce(by = [service]),
measures = [TopK { k = 5 }],
child = Filter(
pred = region = "us-east",
child = Scan("metrics")
)
)
```
Expand All @@ -185,7 +193,7 @@ Equivalent PromQL converges to the same structure.
## ASAP-aware mapping

```text
TopK(Count, service, 5)
Aggregate(reduction = Reduce(by = [service]), measures = [TopK { k = 5 }], ...)
SpaceSaving(k=5)
```
Expand Down Expand Up @@ -214,7 +222,7 @@ semantic information that matters downstream.
- Remove legacy data structures and types (tracked in [#179](https://github.com/ProjectASAP/ASAPPlanner/issues/179), [#205](https://github.com/ProjectASAP/ASAPPlanner/issues/205)
- Implement the ASAP-aware mapping [logic and interfaces](docs/asap_aware_mapping.md)
- Connect output of ASAPPlanner to asap-fusion
- Connect output of ASAPPlanner to ASAPPlanner and ASAPQuery (see open question #1 below)
- Connect output of ASAPPlanner to ASAPCollector and ASAPQuery (see open question #1 below)

# Next steps

Expand All @@ -227,13 +235,11 @@ semantic information that matters downstream.
# Open questions

1. **Integration with downstream artifacts:** How to connect the output of ASAPPlanner to ASAPQuery? ASAPPlanner produces a post-ASAP plan that has semantics of batch query execution over data at rest. Somehow this needs to be converted into two plans (1) streaming dataflow graph that computes summaries on raw data, and (2) batch query execution plan that uses summaries to answer queries.
2. **Time semantics:** Should `TimeWindow` be an explicit node, or should time restriction be
represented as a specialized predicate?
3. **Grouping semantics:** Should grouping remain embedded in `Aggregate`, or should grouping
2. **Grouping semantics:** Should grouping remain embedded in `Aggregate`, or should grouping
become a reusable relational dimension node?
4. **Expression semantics:** Which arithmetic or derived expressions need dedicated semantic
3. **Expression semantics:** Which arithmetic or derived expressions need dedicated semantic
nodes because they materially affect summary selection?
5. **Approximation contracts:** Should accuracy/error requirements be fields on the intent,
4. **Approximation contracts:** Should accuracy/error requirements be fields on the intent,
the workload, or the measure itself?
6. **Summary composability:** How should nested intents describe summaries that can be merged,
5. **Summary composability:** How should nested intents describe summaries that can be merged,
transformed, or reused across queries?
9 changes: 5 additions & 4 deletions docs/parse_and_canonicalize.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ Examples:

```text
PromQL topk(10, count by (service) (...))
-> TopK(k=10, key=FieldRef("service"), measure=Count, ...)
-> Aggregate(reduction=Reduce(by=[service]), measures=[TopK{k=10}], ...)
```

```text
SQL ORDER BY COUNT(*) DESC LIMIT 10
-> TopK(k=10, key=..., measure=Count, ...)
-> Aggregate(reduction=Reduce(by=[...]), measures=[TopK{k=10}], ...)
```

## Canonicalize
Expand All @@ -32,10 +32,11 @@ Aggregate -> Sort(desc) -> Limit(k)
can canonicalize into:

```text
TopK(k, key, measure, Aggregate(...))
Aggregate(reduction=Reduce(by=[key]), measures=[TopK{k}], ...)
```

when the ordering expression and limit form the semantics of a top-k request.
when the ordering expression and limit form the semantics of a top-k request — the
heavy-hitter intent becomes an `AggIntent::TopK` measure on `Aggregate`, not a separate node.

> Canonicalization should be driven by **semantic equivalence and summary relevance**, not by
trying to reproduce every relational operator in a universal AST.
2 changes: 1 addition & 1 deletion docs/pre-asap-ir.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,7 +361,7 @@ SELECT DISTINCT srcip, dstip FROM packets

### Join

Logical join; the physical strategy (hash/merge/broadcast) is picked at L4. SQL `JOIN`.
Logical join; the physical strategy (hash/merge/broadcast) is picked in the post-ASAP IR. SQL `JOIN`.

```sql
SELECT u.prefix FROM bgp_updates u JOIN bgp_rib_state r ON u.prefix = r.prefix
Expand Down
16 changes: 8 additions & 8 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ promql> quantile(0.99, rate(http_requests_total[5m]))
sql> SELECT service, COUNT(*) FROM metrics GROUP BY service
```

**Pre-ASAP IR** (L3 — the sketch-agnostic intent algebra: `QueryExpr`/`AggIntent`):
**Pre-ASAP IR** (ASAP-agnostic `QueryExpr`/`AggIntent`):

```sh
cargo run -p asap-devtools --bin show_pre_asap_ir -- queries.txt
```

**Post-ASAP IR** (L4 — the sketch-bound IR: `SummaryExpr`/`L4Node`, one layer downstream, with
**Post-ASAP IR** (ASAP-aware IR with `SummaryExpr`/`L4Node`, one layer downstream, with
concrete `SummaryKind`/`SummaryParams` committed per aggregate):

```sh
Expand Down Expand Up @@ -61,7 +61,7 @@ asap-frontend-promql = { git = "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/ProjectASAP/ASAPPlanner", pac
asap-aware-mapping = { git = "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/ProjectASAP/ASAPPlanner", package = "asap-aware-mapping" }
```

### Step 1 — get the pre-ASAP IR (L3)
### Step 1 — get the pre-ASAP IR

Lower a query string with a front end. Front ends never depend on each other or on the binder —
pull only the one you need.
Expand All @@ -70,7 +70,7 @@ pull only the one you need.
use asap_frontend_promql::lower_promql;
use asap_types::types::AccuracyTarget;

let l3 = lower_promql(
let pre_asap = lower_promql(
"quantile(0.99, rate(http_requests_total[5m]))",
AccuracyTarget::Epsilon(0.01),
)?; // QueryExpr
Expand All @@ -82,23 +82,23 @@ describing your tables; see `crates/devtools/src/bin/show_pre_asap_ir.rs` for a
`AccuracyTarget` travels with the query, not the crate — pass `Exact` for no approximation
allowed, `Epsilon(e)` / `EpsilonDelta{epsilon, delta}` otherwise.

### Step 2 — get the post-ASAP IR (L4)
### Step 2 — get the post-ASAP IR

Feed the `QueryExpr` to `asap-aware-mapping`. This crate depends only on `asap-types`, never on a
front end, so it's agnostic to which language produced the tree.

```rust
use asap_aware_mapping::implement_tree;

let l4 = implement_tree(&l3)?; // Rc<L4Node> — the SummaryExpr DAG
let post_asap = implement_tree(&pre_asap)?; // Rc<L4Node> — the SummaryExpr DAG
```

Two entry points, both re-exported from the crate root:

- `implementation_for(&AggIntent) -> Implementation` — the single-node decision (sketch, exact
accumulator, or pass-through) for one aggregation.
- `implement_tree(&QueryExpr) -> Result<Rc<L4Node>, ImplementError>` — walks a whole tree, calling
the per-node decision at every `Aggregate` and emitting the full L4 DAG.
the per-node decision at every `Aggregate` and emitting the full post-ASAP DAG.

Each has a `_with(..., &dyn CostModel)` variant. A `CostModel` is the extension point for a
deployment that wants its own candidate ranking or parameter sizing instead of this crate's
Expand All @@ -108,7 +108,7 @@ overridable hooks (`rank_candidates`, `size_params`, `realize_extension`).

### Reading the result

Match on `l4.expr` (a `SummaryExpr`):
Match on `post_asap.expr` (a `SummaryExpr`):

- `Logical(Box<QueryExpr>)` — this subtree wasn't rewritten; execute it exactly.
- `SummaryAgg { summary, params, .. }` — an exact accumulator (`summary.is_exact()`) or an
Expand Down
6 changes: 3 additions & 3 deletions tools/dag-viewer/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# ASAP query DAG viewer

Interactive viewer for the L3 query IR (`QueryExpr`), for manual IR
Interactive viewer for the pre-ASAP query IR (`QueryExpr`), for manual IR
review/debugging, eyeballing common shapes across a corpus, and spotting shared sub-DAGs across queries.

## Generate a graph

```sh
cargo run -p asap-lower --bin dag_export -- \
cargo run -p asap-devtools --bin dag_export -- \
--sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \
--sql "SELECT service, AVG(latency) FROM metrics GROUP BY service" --name q2 \
--promql "topk(5, rate(http_requests_total[5m]))" --name q3 \
Expand Down Expand Up @@ -58,7 +58,7 @@ proposed scope for a v2.

The highlight is computed by hashing each node's `(kind, detail, children)`
bottom-up and matching hashes across queries — see the doc comment on
`crates/ir/src/dag_export.rs`. It is **not** driven by
`crates/types/src/dag_export.rs`. It is **not** driven by
`asap_plan::cse::dedupe_subtrees`: that pass isn't wired into any end-to-end
multi-root planning path today (no caller outside its own unit tests), so
there's nothing yet that would emit a real `CseWorkloadPlan` to visualize.
Expand Down
2 changes: 1 addition & 1 deletion tools/dag-viewer/RUNNING.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ the script and edit the `--sql`/`--promql` lines to try your own instead.
For more control, call the underlying binary directly:

```sh
cargo run -p asap-lower --bin dag_export -- \
cargo run -p asap-devtools --bin dag_export -- \
--sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \
--promql "topk(5, rate(http_requests_total[5m]))" --name q2 \
> tools/dag-viewer/dag.json
Expand Down
2 changes: 1 addition & 1 deletion tools/dag-viewer/generate-sample.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/../.."

cargo run -p asap-lower --bin dag_export -- \
cargo run -p asap-devtools --bin dag_export -- \
--sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \
--sql "SELECT service, AVG(latency) FROM metrics GROUP BY service" --name q2 \
--promql "topk(5, rate(http_requests_total[5m]))" --name q3 \
Expand Down
Loading