ProjectASAP · zzylol · Apr 30, 2026 · Apr 30, 2026
diff --git a/benchmarks/.gitignore b/benchmarks/.gitignore
@@ -0,0 +1,11 @@
+# Python cache
+scripts/__pycache__/
+*.pyc
+
+# Generated benchmark output
+reports/*.json
+reports/*.csv
+reports/*.md
+
+# Cold-store data populated by seed_cold_store.py — keep the dir, drop the data
+cold-store/raw/
diff --git a/benchmarks/README.md b/benchmarks/README.md
@@ -0,0 +1,128 @@
+# Three-Way Query Benchmark Harness
+
+This directory holds the docker-compose-based query benchmark harness for
+the central paper claim:
+
+> **ASAPQuery-backend (sketch + ad-hoc cold-store fallback) vs. Prometheus
+> vs. VictoriaMetrics** on the same input stream.
+
+The harness is intentionally thin. It reuses `asap-quickstart`'s
+docker-compose for the ASAP stack (kafka, arroyo, planner, summary-ingest,
+queryengine, prometheus, 7 pattern-based fake exporters), layers
+VictoriaMetrics + a cold-store volume on top via an override, and adds
+runner scripts that record per-query latency / throughput across all
+three backends.
+
+## Workloads
+
+| Workload | Source | What it tests |
+|----------|--------|---------------|
+| **W1** | `queries/promql_suite.json` (14 queries) | ASAP's sketch path: sum, avg, max, min, quantiles at p50/p90/p95/p99, group-bys |
+| **W2** | `queries/adhoc_suite.json` (7 queries) | ASAP's fallback path: regex matchers, label_replace, rate/increase, exact histograms — hits `ColdFallback` (LocalFsColdStore) first, then forwards to Prometheus for query shapes the cold tier doesn't support |
+| **W3** | both suites, run directly against Prom & VM | baseline ground truth + industry comparison |
+
+## Files
+
+```
+benchmarks/
+├── docker-compose.yml          # OVERRIDE for asap-quickstart: adds VM + cold-store volume
+├── config/
+│   └── vmagent-scrape.yml      # VM scrape config (same exporters as Prom)
+├── cold-store/                 # Bind-mounted into queryengine; seeded by seed_cold_store.py
+├── queries/
+│   ├── promql_suite.json       # W1 — pre-existing
+│   └── adhoc_suite.json        # W2 — NEW
+├── scripts/
+│   ├── run_asap_workloads.py   # NEW: W1+W2 against ASAP, median + P99
+│   ├── run_prom.py             # NEW: W1+W2 against Prometheus baseline
+│   ├── run_vm.py               # NEW: W1+W2 against VictoriaMetrics
+│   ├── run_concurrency_sweep.py  # NEW: throughput-vs-concurrency CSV
+│   ├── compare_three_way.py    # NEW: renders three_way_eval.md
+│   ├── seed_cold_store.py      # NEW: pre-populates cold-store for W2
+│   ├── wait_for_stack.sh       # pre-existing — waits for ASAP/Prom/Arroyo healthy
+│   ├── ingest_wait.sh          # pre-existing — waits for arroyo pipeline RUNNING
+│   ├── run_asap.py             # pre-existing — kept for the existing CI gate
+│   ├── run_baseline.py         # pre-existing — kept for the existing CI gate
+│   └── compare.py              # pre-existing — kept for the existing CI gate
+├── reports/                    # JSON / CSV / MD output lands here
+└── run_full_eval.sh            # NEW: orchestrator
+```
+
+## Quick start
+
+```bash
+# from repo root
+./benchmarks/run_full_eval.sh
+```
+
+That will:
+
+1. `docker compose up -d` the merged asap-quickstart + benchmarks stack.
+2. Wait for Prometheus, Arroyo, QueryEngine, VictoriaMetrics to be healthy.
+3. Wait for the Arroyo pipeline `asap-demo` to reach RUNNING and let
+   sketches accumulate (`ingest_wait.sh`).
+4. Seed `benchmarks/cold-store/` with raw JSONL fixtures for W2.
+5. Run W1+W2 against each backend (3 iterations per query, captures
+   median + P99).
+6. Run the concurrency sweep at C ∈ {1, 4, 16, 64} for 60s each.
+7. Render `benchmarks/reports/three_way_eval.md`.
+
+Pass `--down` to tear the stack down at the end, or `--skip-sweep` to
+skip the 4×60s concurrency sweep.
+
+## Manual / partial runs
+
+```bash
+# Only one runner
+python3 benchmarks/scripts/run_asap_workloads.py --asap-url http://localhost:8088
+python3 benchmarks/scripts/run_prom.py           --prometheus-url http://localhost:9090
+python3 benchmarks/scripts/run_vm.py             --vm-url http://localhost:8428
+
+# Smaller sweep
+python3 benchmarks/scripts/run_concurrency_sweep.py --duration 10 --concurrency 1,8
+
+# Re-render the report after edits
+python3 benchmarks/scripts/compare_three_way.py
+```
+
+## Output artefacts
+
+| File | Producer | Shape |
+|------|----------|-------|
+| `reports/asap_promql.json`  | `run_asap_workloads.py` | per-query latencies, median, P99, result data |
+| `reports/asap_adhoc.json`   | `run_asap_workloads.py` | same shape, W2 |
+| `reports/prom_promql.json`  | `run_prom.py`           | same shape, baseline |
+| `reports/prom_adhoc.json`   | `run_prom.py`           | same shape, W2 |
+| `reports/vm_promql.json`    | `run_vm.py`             | same shape, VM |
+| `reports/vm_adhoc.json`     | `run_vm.py`             | same shape, W2 |
+| `reports/concurrency_sweep.csv` | `run_concurrency_sweep.py` | `backend,concurrency,total_queries,throughput_qps,p50_ms,p99_ms` |
+| `reports/three_way_eval.md` | `compare_three_way.py`  | Tables 1–3 + capability matrix |
+
+## What it deliberately does NOT do (per scope)
+
+- The user explicitly de-scoped controller polish, sketch reallocation, and
+  backfill correctness. We rely on `asap-quickstart`'s pre-baked
+  `controller-config.yaml` to hand `streaming_config.yaml` to
+  `asap-summary-ingest` — whatever sketches that produces is fine; only
+  the 14 W1 queries need to come back non-empty.
+- The W2 cold-store fixtures are deterministic synthetic data, not a
+  faithful replay of the live exporter stream. We only need the
+  fallback path to return non-empty so latencies are real.
+- Smoke tests assume Docker is installed locally. CI uses
+  `accuracy_performance.yml` which already runs the existing
+  `compare.py` gate; this harness is additive.
+
+## Wiring assumptions / TODO
+
+- `queryengine` reads `ASAP_COLD_STORE_ROOT` from env. Today the binary
+  doesn't expose a `--cold-store-root` flag in `main.rs` — the wiring
+  exists in `AdapterConfig::prometheus_promql_with_cold` and the
+  `LocalFsColdStore` struct, but `main.rs` itself constructs
+  `AdapterConfig::prometheus_promql(...)` (no cold tier). Until that
+  flag lands, ASAP W2 results will reflect only the Prometheus
+  forwarding leg, not the cold-store leg. The bind-mount + seed script
+  + adhoc_suite.json are correct as-is and will start exercising the
+  cold path the moment the flag is added.
+- The concurrency sweep uses `ThreadPoolExecutor` and the `requests`
+  library — fine for ≤64 concurrency on localhost; if we ever push to
+  higher fan-out, switch to `aiohttp`.
diff --git a/benchmarks/cold-store/.gitkeep b/benchmarks/cold-store/.gitkeep
diff --git a/benchmarks/config/vmagent-scrape.yml b/benchmarks/config/vmagent-scrape.yml
@@ -0,0 +1,22 @@
+# VictoriaMetrics scrape config — points at the same fake exporters as
+# the asap-quickstart Prometheus scrape config, so VM ingests an
+# identical input stream. 1s scrape matches asap-quickstart prometheus.yml.
+
+global:
+  scrape_interval: 1s
+
+scrape_configs:
+  - job_name: 'pattern-exporters'
+    metric_relabel_configs:
+      - source_labels: [__name__]
+        regex: sensor_reading
+        action: keep
+    static_configs:
+      - targets:
+          - 'fake-exporter-constant:50000'
+          - 'fake-exporter-linear-up:50001'
+          - 'fake-exporter-linear-down:50002'
+          - 'fake-exporter-sine:50003'
+          - 'fake-exporter-sine-noise:50004'
+          - 'fake-exporter-step:50005'
+          - 'fake-exporter-exp-up:50006'
diff --git a/benchmarks/docker-compose.yml b/benchmarks/docker-compose.yml
@@ -1,20 +1,101 @@
-# CI image override: replaces quickstart's pinned release images with images
-# built from the current branch. Intended for use as a Compose override:
+# Three-way query benchmark harness — OVERRIDE for asap-quickstart.
 #
-#   ASAP_IMAGE_TAG=sha-<short-sha> docker compose \
+# This file is intended to be merged with asap-quickstart/docker-compose.yml:
+#
+#   docker compose \
 #     --project-directory . \
 #     -f asap-quickstart/docker-compose.yml \
 #     -f benchmarks/docker-compose.yml \
 #     up -d
 #
-# ASAP_IMAGE_TAG is set automatically by the 'build' job in accuracy_performance.yml.
+# What this override adds on top of the asap-quickstart base stack
+# (kafka + arroyo + planner + summary-ingest + queryengine + prometheus +
+# 7 pattern-based fake exporters):
+#
+#   1. victoriametrics — third backend under test, scrapes the same fake
+#      exporters so all three backends (ASAP / Prom / VM) see an identical
+#      input stream. Comparable 1d retention.
+#
+#   2. queryengine override — adds a cold-store bind-mount at /cold-store.
+#      The §5.2 ColdFallback (LocalFsColdStore) reads raw JSONL fixtures
+#      from this dir to answer ad-hoc queries that the sketch path can't
+#      serve. Pre-populate with `benchmarks/scripts/seed_cold_store.py`
+#      before running the ad-hoc workload.
+#
+# CI / GHCR image override: kept compatible with the existing
+# accuracy_performance.yml workflow — pass ASAP_IMAGE_TAG to swap pinned
+# release images for branch-built images.
 
 services:
+  # NOTE: the asap-quickstart base file pins these to v0.2.0 release tags.
+  # Setting ASAP_IMAGE_TAG (e.g. sha-abcdef0) overrides them for CI runs.
   asap-planner-rs:
-    image: ghcr.io/projectasap/asap-planner-rs:${ASAP_IMAGE_TAG}
+    image: ghcr.io/projectasap/asap-planner-rs:${ASAP_IMAGE_TAG:-v0.2.0}
 
   asap-summary-ingest:
-    image: ghcr.io/projectasap/asap-summary-ingest:${ASAP_IMAGE_TAG}
+    image: ghcr.io/projectasap/asap-summary-ingest:${ASAP_IMAGE_TAG:-v0.2.0}
 
+  # ---------------------------------------------------------------------------
+  # QueryEngine override — adds cold-store mount + env knob.
+  #
+  # The cold-store volume is a directory under benchmarks/ (gitignored);
+  # seed_cold_store.py writes JSONL parts into the layout the ColdFallback
+  # adapter expects (raw/<metric>/YYYY/MM/DD/HH/part-NNNNNN.jsonl).
+  #
+  # ASAP_COLD_STORE_ROOT is a placeholder env var until main.rs grows a
+  # --cold-store-root CLI flag. For now AdapterConfig::prometheus_promql
+  # (the path actually wired in main.rs) gives us the Prometheus-only
+  # fallback chain — the cold tier kicks in once the binary reads the env.
+  # The bind-mount + seed script are correct as-is so the harness will
+  # work without re-touching this YAML when that flag lands.
+  # ---------------------------------------------------------------------------
   queryengine:
-    image: ghcr.io/projectasap/asap-query-engine:${ASAP_IMAGE_TAG}
+    image: ghcr.io/projectasap/asap-query-engine:${ASAP_IMAGE_TAG:-v0.2.0}
+    volumes:
+      - asap-planner-output:/asap-planner-output:ro
+      - ./asap-quickstart/output/queryengine:/app/outputs
+      - ./benchmarks/cold-store:/cold-store
+    environment:
+      - RUST_LOG=INFO
+      - RUST_BACKTRACE=1
+      - ASAP_COLD_STORE_ROOT=/cold-store
+
+  # ---------------------------------------------------------------------------
+  # VictoriaMetrics — third backend under test.
+  # Scrapes the same fake exporters Prometheus does, with comparable
+  # retention. Exposes the standard 8428 query port.
+  # ---------------------------------------------------------------------------
+  victoriametrics:
+    image: victoriametrics/victoria-metrics:v1.106.1
+    container_name: asap-victoriametrics
+    hostname: victoriametrics
+    networks:
+      - asap-network
+    ports:
+      - "8428:8428"
+    volumes:
+      - victoriametrics-data:/storage
+      - ./benchmarks/config/vmagent-scrape.yml:/etc/vm/scrape.yml:ro
+    command:
+      - "-storageDataPath=/storage"
+      - "-retentionPeriod=1d"
+      - "-promscrape.config=/etc/vm/scrape.yml"
+      - "-httpListenAddr=:8428"
+    healthcheck:
+      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:8428/health || exit 1"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+    depends_on:
+      - fake-exporter-constant
+      - fake-exporter-linear-up
+      - fake-exporter-linear-down
+      - fake-exporter-sine
+      - fake-exporter-sine-noise
+      - fake-exporter-step
+      - fake-exporter-exp-up
+    restart: no
+
+volumes:
+  victoriametrics-data:
diff --git a/benchmarks/queries/adhoc_suite.json b/benchmarks/queries/adhoc_suite.json
@@ -0,0 +1,54 @@
+{
+  "_comment": "Workload W2 — ad-hoc PromQL queries that exercise the ASAP fallback path. Per docs/01-getting-started/architecture.md, ASAP cannot serve regex matchers, label_replace, exact histograms, or non-quantile range functions from sketches; they fall through to the cold-store first (LocalFsColdStore via ColdFallback) and finally to Prometheus. All queries here also work against the Prometheus and VictoriaMetrics baselines for direct comparison.",
+  "queries": [
+    {
+      "id": "regex_pattern_match",
+      "expr": "sensor_reading{pattern=~\"sine.*\"}",
+      "kind": "regex_matcher",
+      "fallback": "cold_store",
+      "rationale": "Regex label matchers are not pushed into sketch keys; ColdFallback scans raw samples in the cold tier."
+    },
+    {
+      "id": "label_replace_relabel",
+      "expr": "label_replace(sensor_reading, \"new_label\", \"$1\", \"region\", \"region(.*)\")",
+      "kind": "label_manipulation",
+      "fallback": "prometheus",
+      "rationale": "label_replace is a label-rewrite operator that has no sketch-side equivalent — forwarded straight to Prometheus."
+    },
+    {
+      "id": "rate_over_5m",
+      "expr": "rate(sensor_reading[5m])",
+      "kind": "range_function",
+      "fallback": "prometheus",
+      "rationale": "rate() needs per-sample positions; sketches store summaries, not raw points."
+    },
+    {
+      "id": "increase_over_1m",
+      "expr": "increase(sensor_reading[1m])",
+      "kind": "range_function",
+      "fallback": "prometheus",
+      "rationale": "Same as rate — counter-difference over a window."
+    },
+    {
+      "id": "exact_histogram_q99",
+      "expr": "histogram_quantile(0.99, sum by (le) (rate(sensor_reading_bucket[5m])))",
+      "kind": "exact_histogram",
+      "fallback": "prometheus",
+      "rationale": "Exact histogram quantiles need _bucket series; ASAP serves quantiles via DDSketch instead, this path is forwarded."
+    },
+    {
+      "id": "regex_negative_match",
+      "expr": "sum(sensor_reading{service!~\"svc[0-3]\"})",
+      "kind": "regex_matcher",
+      "fallback": "cold_store",
+      "rationale": "Negative regex matcher — same fallback path as positive regex."
+    },
+    {
+      "id": "raw_lookup_specific_series",
+      "expr": "sensor_reading{pattern=\"sine\",region=\"region0\"}",
+      "kind": "instant_vector_selector",
+      "fallback": "cold_store",
+      "rationale": "Bare instant vector lookup — ColdFallback v1 scope per s3_adapter.rs handles this exactly: per-series latest value within Prometheus's 5m lookback delta."
+    }
+  ]
+}