From 34ed182767e825af9fac376891367f8965fd5783 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Wed, 6 May 2026 16:35:50 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20spec=20MVP=20v6=20=E2=80=94=20controlle?= =?UTF-8?q?r-driven=20multi-stage=20demo=20for=20issue=20#46?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue #46's MVP scope was substantially expanded by the user. This spec captures the new requirements: - 3 query classes: window-per-series, label-aggregation-at-instant, combined window+label - Controller drives sketch + stage placement based on input query workload (one-shot static plan, no dynamic replanning in MVP) - Multi-stage topology: 10 SDK / 2 agents / 1 gateway / 1 backend with 10K total cardinality split per production-pyramid pattern - 6 criteria: bandwidth, query latency, combined resource, accuracy, cold-fallback, freshness - Freshness measured by encoding emit timestamp in synthetic counter value (assumes single-host clock sync) Non-goals explicit: dynamic replan, OpAMP hot reconfig, production- scale cardinality, multi-host federation, full PromQL on archive. 6 phases (A=verify controller plumbing, B=config emitter, C=multi-stage overlay + gateway aggregation, D=workload + freshness wiring, E=driver + report v6, F=run + verify + comment). Total ~1-2 weeks contingent on Phase A. Builds on v4 (merged) and v5 (in flight). Co-Authored-By: Claude Opus 4.7 (1M context) --- ...p-v6-controller-driven-multi-stage-demo.md | 354 ++++++++++++++++++ 1 file changed, 354 insertions(+) create mode 100644 docs/spec-mvp-v6-controller-driven-multi-stage-demo.md diff --git a/docs/spec-mvp-v6-controller-driven-multi-stage-demo.md b/docs/spec-mvp-v6-controller-driven-multi-stage-demo.md new file mode 100644 index 00000000..f1510224 --- /dev/null +++ b/docs/spec-mvp-v6-controller-driven-multi-stage-demo.md @@ -0,0 +1,354 @@ +# MVP v6 — controller-driven multi-stage end-to-end demo (issue #46) + +## Status + +Spec, 2026-05-06. Not yet implemented. Builds on v4 (PR #290 merged) and v5 +(PR in flight as `mvp/v5-postings-compactor` + `mvp/v5-postings-aware-engine`). + +## Goal + +Demonstrate **end-to-end** that ASAP's controller picks sketches and stage +placements from an input PromQL query workload, the runtimes start with that +plan, and the resulting pipeline serves the three canonical query classes +within the six MVP criteria. + +## Non-goals + +- **Dynamic replanning during the demo run.** The controller computes ONE + plan based on the input workload at start; runtimes start with that plan + baked in. Workload-driven replan-while-running is a separate work item. +- **OpAMP-driven hot reconfiguration.** Today the agents/gateways must be + stopped + restarted to pick up a new sketch config. OpAMP push is wired but + not yet exercised end-to-end. Out of scope for this MVP. +- **Production-scale cardinality** (1M+ series). Single-host bench at 10K + total backend cardinality is the target, distributed across the topology + per the production-pyramid pattern below. +- **Cold/JSONL fallback.** v5's `data_source: gorilla_archive` confirmation + satisfies criterion ⑤. The cold-fallback JSONL path stays exactly as it is + today and is exercised but not modified. + +## Topology + +10K total backend cardinality, spread across the topology following typical +production ratios: + +``` +10 SDK instances (fake-exporter) + ~100-500 series each (= 1K-5K total at the SDK layer) + │ + ▼ +2 agent collectors (sketchcol) + ~5K series each at the agent layer + │ + ▼ +1 gateway collector (sketchcol-gateway) + ~10K series aggregated at the gateway layer + │ + ▼ +1 backend (precompute_engine) + ingests sketch envelopes + serves PromQL +``` + +Distribution rationale (from real-world numbers): +- SDK per-process: hundreds to low thousands. ASAP MVP uses 100-500 per + fake-exporter instance. +- Agent collector: aggregates from many pods. ASAP MVP gives each agent ~5K. +- Gateway collector: fan-in across agents. ASAP MVP runs ONE gateway at 10K + to keep the demo container count manageable; production would have 2+ for + HA. +- Backend: millions in production. ASAP MVP uses 1, sized for 10K. + +For the actual fake-exporter knob: each emits e.g. `card_per_agent=500` series +across (`{node, pod, rack, zone}`-style label combos). Ten of them via N=10 +is the existing v4 mechanism. + +## Three canonical query classes + +The MVP must demonstrate **all three** for the controller-plan story to +matter. Each class has different sketch + stage-placement implications: + +1. **Window aggregation per series.** + Examples: `quantile_over_time(0.99, http_requests_total_latency_ms[1m])`, + `rate(http_requests_total[5m])`, `sum_over_time(http_freshness_probe[10s])`. + Per-series time-axis reduction. +2. **Label aggregation across multiple series at one timestamp.** + Examples: `sum by (zone) (http_requests_total)`, + `topk(10, http_requests_total{status=~"5.."})`, + `count(http_requests_total{service="api"})`. + Label-axis reduction at instant. +3. **Combined window + label aggregation.** + Examples: `sum by (zone) (rate(http_requests_total[5m]))`, + `topk(5, quantile_over_time(0.99, latency[1m]))`. + Inner window per-series, outer label aggregation across the per-series + results. + +## Controller responsibility + +Input: a list of PromQL queries (a "workload manifest"). For the MVP this is +a static file — `deploy/configs/mvp-v6-workload.yaml` — containing the three +query classes above, mapped to the metrics they reference. + +Pipeline (already implemented in `controller/src/`): + +``` +L1 query_language parse PromQL strings → AST +L2 logical_plan AST → logical operators +L3 intent_algebra AggIntent + QueryExpr DAG + Schema unique_keys +L4 sketch_algebra SketchExpr + Bind* rules: pick sketch family per metric +L5 stage_split StageAllocator + ThreeStageEmitter: assign each + SketchExpr to a stage (edge / gateway / backend) + with per-stage cost +``` + +**Output for the MVP**: a static plan describing, per metric: +- Sketch family (DDSketch / KLL / HLL / CountSketch / Count-Min Sketch / + raw / Gorilla-archive) +- Stage of placement (SDK / agent / gateway / backend) +- Window `W`, label projection `L`, encoding triple + +Plan is materialised as **per-runtime config files** (mirroring how v4 +works): `sketchcol-agent-*.yaml` for agents, `sketchcol-gateway-*.yaml` for +the gateway, `backend-storage-routing.yaml` for the backend's +`EngineRouter`. + +For the MVP, we don't need the controller to *push* plans at runtime — we +need it to *generate* the config files that the runtimes start with. + +### Open question on controller plumbing + +Today the controller's L5 produces a plan struct in memory. **Whether it can +serialise to runtime config YAMLs is unverified.** Three possibilities: + +(a) **It already does** — Phase E shipped a "config emitter" that writes the + YAMLs. Verify before implementation. + +(b) **It almost does** — the StageAllocator's output is the right shape but + needs a thin emitter (~1 day of work) to write YAML. + +(c) **It doesn't** — needs a new module. ~3-5 days. + +The first implementation step in v6 is to figure out which of (a)/(b)/(c) is +the case, by reading `controller/src/stage_split/` and the existing +`workloads.yaml` codepath. + +## What runs where in the MVP plan + +Concrete example — for the workload manifest above, a plausible plan output +is: + +| Metric | Query class | Sketch family | Stage placement | Why | +|---|---|---|---|---| +| `http_requests_total_latency_ms` | window quantile | DDSketch | edge (per agent) | quantile is sketchable; emit per-agent so backend gets pre-aggregated state | +| `http_requests_total{status=~"5.."}` | label-aggregation count | raw | gateway (sum-merge) | count is exact, gateway aggregates to reduce series count | +| `http_requests_total` | combined: `sum by (zone) (rate(...[5m]))` | raw + agent rollup | edge (rate) → gateway (sum by zone) | inner rate at edge, outer label-sum at gateway; backend just stores the result | +| `http_freshness_probe` | freshness measurement | raw | backend (Gorilla-archive) | exact retention; gateway tees to S3 via `gorillas3processor` | + +The actual values depend on the controller's cost model output for the +workload, but the schema-of-output is what runtime config emission needs to +target. + +## Six MVP criteria — measurement plan + +### ① Bandwidth (X) + +Per-stage on-the-wire bytes/s, measured at: +- SDK → agent (per-agent input) +- agent → gateway +- gateway → backend +- (separately) gateway → S3 for the Gorilla-archive metric + +Compared against B0 (raw end-to-end via Prometheus remote_write) on the +same workload, same cardinality split. Reported per-edge of the topology so +the user can see which stage saved how much. + +Tooling: extend `measure_stages.py` (v4) to label per-edge bytes +(`edge=sdk_to_agent`, `edge=agent_to_gateway`, etc.). + +### ② Aggregation query latency (Y) + +p50 / p99 of each of the three query classes, compared against B0 +Prometheus on the same workload. Measured by `promql_replay.py` (existing). + +Reported per query class so we can see which class wins more. + +### ③ Combined resource (Z) + +Per-stage CPU + RSS + disk, summed across all containers per stage: +- `agent` (×2 containers) +- `gateway` (×1) +- `backend-ingest` + `backend-query` + `backend-storage` (the precompute_engine + MinIO/S3 process) + +vs. the same partition for B0 (where `backend-ingest/storage/query` is +Prometheus + its TSDB on disk). + +Tooling: `measure_stages.py` (v4 already has stage labelling). Update to +match the new topology (2 agents instead of 1). + +### ④ Accuracy + +Per-row relative error from the existing `accuracy_reduce.py`: +- Quantile rel-err inside DDSketch's ε envelope +- Sum-over-time exact (Count-Min within εδ) +- Topk recall vs cold-tier truth + +Reported per query class. v4 fixed the NaN problem by adding query-side +warm-up; that fix carries forward. + +### ⑤ Cold-store fallback for ad-hoc queries + +Replay client fires an ad-hoc query that the warm tier cannot answer: +- `count(http_requests_total{service="payments"})` on a metric the controller + did NOT plan for warm-tier coverage of `service=payments` +- The query routes via `BackendStorageRouting` to `GorillaQueryEngine` +- Response includes `data_source: gorilla_archive` (verified in v4) +- Latency reported + +### ⑥ Freshness + +For each of the three serving paths (B0 raw / warm-tier sketch / +Gorilla-archive), measure: + +``` +Δ = (timestamp when query first returns the sample) + − (timestamp when the sample was generated by fake-exporter) +``` + +Mechanic (assuming clock sync within the host, which is fine for +single-host bench): + +1. Fake-exporter emits a synthetic counter `http_freshness_probe_total{kind=...}` whose + *value* is the Unix ms epoch at emission time (so the timestamp is encoded + in the value itself). +2. Replay client polls `last_over_time(http_freshness_probe_total[10s])` + every 100ms. +3. On first non-empty response: Δ = `poll_response_ts_ms − observed_value`. +4. Repeat for at least 60 samples per path; report p50, p99, count. + +Three probes (one per path), each routed to its respective tier: +- `http_freshness_probe_raw` — B0 Prometheus +- `http_freshness_probe_warm` — sketch warm tier (any sketch family — pick + whichever the controller plans) +- `http_freshness_probe_archive` — Gorilla-archive (controller plan flags + this metric `StorageBackend::GorillaS3`) + +The v4 freshness was UNKNOWN because the probe never reached Prometheus on +B0/B1/B5 (gateway didn't forward it) and ASAP's sketch flush window hadn't +fired in 60s. The v6 fixes: +- Probe routing tested explicitly for all three paths during pre-flight. +- Sketch window for the freshness probe metric pinned at ≤ 1s. +- Soak duration extended only as needed for the slowest path (probably the + Gorilla-archive path, since chunks aren't queryable until the per-window + flush lands in S3 — typically ≥ 60s). + +## Demo workflow + +``` +0. user authors mvp-v6-workload.yaml (3 queries, fixed) +1. controller reads workload + cost model +2. controller emits per-runtime config files +3. docker-compose up with the emitted configs +4. fake-exporters emit +5. 60s warm-up +6. 30s query-side warm-up (poll for non-zero on a representative metric) +7. 60s measurement: replay client + freshness probes + measure_stages.py + s3_cost_tracker +8. Cold-fallback ad-hoc query + verify data_source: gorilla_archive +9. mvp_report.py emits MVP_REPORT_v6.md +10. comment posted on issue #46 with v6 results +``` + +## Phased implementation + +### Phase A — verify controller plumbing (~1 day) + +- Read `controller/src/stage_split/` and figure out which of (a)/(b)/(c) is + the case for runtime config emission. +- If (c) — flag as a blocker; spec a separate emitter PR. + +### Phase B — controller config emitter (~1-3 days, depends on Phase A) + +- Implement / verify the controller's "plan → per-runtime YAML" path. +- Test: given a fixed workload manifest, the controller produces stable, + diff-able config YAMLs that the runtimes already accept. + +### Phase C — multi-agent + gateway-aggregation overlay (~2-3 days) + +- New compose overlay `mvp-v6-multi-stage.yml` with 10 fake-exporters, + 2 agents, 1 gateway, 1 backend, 1 MinIO, 1 Prometheus (B0 baseline only). +- Gateway collector with sketch processors enabled (today's gateway is + pass-through; needs the same processor config the agent uses, scoped to + per-metric). +- Cardinality split: 100-500 series per fake-exporter, 5K per agent, 10K + at gateway/backend. + +### Phase D — workload + freshness wiring (~1 day) + +- `mvp-v6-workload.yaml` defining the three query classes +- Three freshness probe metrics with proper routing (per Phase C overlay) +- Replay client extended to fire all three query classes per measurement + window + +### Phase E — driver + report (~1 day) + +- `run_mvp_demo_v6.sh` driving Phases C+D end-to-end +- `mvp_report.py` v6 with sections per criterion + per query class +- Stage-separated bandwidth tooling (`measure_stages.py` extended for + per-edge labelling) + +### Phase F — run + verify + comment (~half day) + +- Run end-to-end on the host +- Verify the six criteria each have measured numbers +- Post comment on issue #46 with `MVP_REPORT_v6.md` verbatim + +Total: **~1-2 weeks** of focused work, depending on Phase A's verdict on the +controller emitter. + +## Dependencies on in-flight work + +- **v5-author** (postings + concat-compactor + cost tracker + label-predicate + ad-hoc queries) — substrate. Must be merged before v6 demo run, since v6's + ⑤ cold-fallback and ⑥ archive freshness need partial S3 reads + cost + measurement to be well-instrumented. +- The cold-store JSONL deprecation work — orthogonal to v6; can ship later. + +## What v6 does NOT verify (be honest about it) + +- ❌ Dynamic plan transitions while the demo runs (one-shot static plan) +- ❌ OpAMP hot reconfig (config baked into runtime startup) +- ❌ 1M+ cardinality (10K bench) +- ❌ Multi-host federation (single-host) +- ❌ PromQL completeness on the archive tier (curated subset only) + +These are flagged in the §Non-goals and `MVP_REPORT_v6.md`'s caveats section. +The paper's §Eval discusses them under "future work" or "out of scope at +demo scale". + +## Success criteria + +- All six MVP criteria report measured numbers (not UNKNOWN, not fabricated) +- Controller's stage-split visibly drives the demo (the per-runtime configs + are observable as the controller's output, not hand-authored) +- Three query classes all answered, per-class latency reported +- v5 substrate (postings + compactor + cost tracker) actually exercised +- Comment on issue #46 with `MVP_REPORT_v6.md` verbatim +- Both v5 PRs and v6 PRs merged + +## Risks and mitigations + +| Risk | Mitigation | +|---|---| +| Controller's L5 doesn't have a config emitter (Phase A is (c)) | Build a thin emitter; track as a separate PR; Phase B's spec doc precedes implementation | +| Gateway sketch processors don't load cleanly (today's gateway is pass-through) | Mirror the agent's `sketchcol-agent-b6-asap-single-sketch.yaml` config to a `sketchcol-gateway-mvp-v6.yaml`; if the binary doesn't load processor config, this becomes the blocker — escalate before Phase C | +| Sketch window flush timing causes ⑥ freshness UNKNOWN again | Pin window to ≤ 1s for the freshness probe metric; extend soak as needed for archive path | +| Z resource doesn't show savings at 10K cardinality (v4 saw negative) | Honest report; explain in caveats that the savings are at higher cardinality where sketch state amortises better; cross-reference 60-cell sweep numbers | + +## References + +- Issue #46 (updated): https://github.com/ProjectASAP/ASAPCollector/issues/46 +- v4 PR (merged): https://github.com/ProjectASAP/ASAPCollector/pull/290 +- v4 issue comment: https://github.com/ProjectASAP/ASAPCollector/issues/46#issuecomment-4391549654 +- v5 PRs (in flight): `mvp/v5-postings-compactor`, + `mvp/v5-postings-aware-engine` +- Comparison doc: `docs/comparison-asap-vs-databricks-pantheon-hydra.md` +- Controller align design: `docs/control-plane-design.md` +- Gorilla-S3 design: `docs/design-gorilla-s3-cold-engine.md`