eval: three-way query harness — ASAP vs Prometheus vs VictoriaMetrics - #66
Merged
Merged
Conversation
End-to-end benchmark for the central paper claim: backend query latency
and throughput for ASAP (sketch + ad-hoc S3 fallback) vs Prometheus alone
vs VictoriaMetrics alone, on the same input stream.
* benchmarks/docker-compose.yml — augments asap-quickstart with a
victoriametrics service + cold-store bind-mount + queryengine env
knob. The asap-quickstart compose stays untouched; this is an
override.
* benchmarks/config/vmagent-scrape.yml — VM scrape config matching the
Prom config 1:1 so they ingest the same series.
* benchmarks/queries/adhoc_suite.json — 7 ad-hoc PromQL queries
(regex matchers, label_replace, rate, increase, exact histogram,
negative regex, raw selector) that fall outside the sketch capability.
* benchmarks/scripts/run_asap_workloads.py — runs both promql_suite
(W1) and adhoc_suite (W2) against ASAP queryengine; outputs
latency/error/fallback per query.
* benchmarks/scripts/run_prom.py / run_vm.py — same suites against
Prometheus and VictoriaMetrics baselines (W3).
* benchmarks/scripts/run_concurrency_sweep.py — ThreadPoolExecutor
hammer at C ∈ {1, 4, 16, 64} for 60s per backend; emits the
throughput-vs-concurrency CSV.
* benchmarks/scripts/compare_three_way.py — Tables 1/2/3 + capability
matrix; writes benchmarks/reports/three_way_eval.md.
* benchmarks/scripts/seed_cold_store.py — pre-populates 7×3×3 = 63
series under cold-store/raw/<metric>/YYYY/MM/DD/HH/part-NNNNNN.jsonl
in the byte-identical S3 layout, so the W2 fallback path returns
non-empty during the bench. Output dir gitignored.
* benchmarks/run_full_eval.sh — orchestrator: docker-compose up,
wait_for_stack, ingest_wait, run all 4 runners + concurrency sweep,
compare_three_way; --down to tear down, --skip-sweep to skip C-sweep.
Smoke: docker compose config --quiet exits 0 on the merged compose;
all Python scripts compile; seed_cold_store.py generates 315 JSONL
samples in the expected layout; compare_three_way.py survives missing
inputs gracefully; orchestrator passes bash -n.
Known caveat (de-scoped per the user's review): asap-query-engine's
main.rs builds AdapterConfig::prometheus_promql(...) only.
ColdFallback + LocalFsColdStore types and the
prometheus_promql_with_cold constructor exist, but no CLI flag wires
ASAP_COLD_STORE_ROOT into them. Until that flag lands, the W2 ASAP
results reflect only the Prometheus-forwarding leg, not the local-FS
cold tier. Bind-mount, seed script, and adhoc suite are correct
as-is — the harness exercises the cold path the moment the flag
lands in main.rs.
2 tasks
zzylol
added a commit
that referenced
this pull request
Apr 30, 2026
* §4 Serialization format versioning tests — mark done with #65 and document the three landed tests (schema / backfill / part_meta) including the no-data-corruption assertion. Adds a defense-in-depth follow-up note for a live-restart docker-compose variant. * §1 Cold-query fallback latency follow-up — point at #66 (benchmarks/run_full_eval.sh). The runner exists and can produce the ≤2× P99 number; only blocker now is the ASAP_COLD_STORE_ROOT CLI flag wiring in asap-query-engine main.rs (de-scoped per the paper review — flagged here so we don't lose the dependency chain). §2 (accuracy profile) and §5 (correctness proofs) intentionally untouched — those weren't in scope for this round.
8 tasks
zzylol
added a commit
that referenced
this pull request
May 25, 2026
The benchmarks/ harness (docker-compose override + 11 scripts + query suites) was built entirely on the removed asap-quickstart Arroyo/Kafka/ asap-summary-ingest stack — its compose override merges with an asap-quickstart base that no longer exists in this repo, so it can't run against the current OTLP architecture. No .github/workflows references it (the README's accuracy_performance.yml mention was stale). Removed the whole directory (21 files) and cleaned up the dangling references: - README.md / architecture.md repo trees: dropped the benchmarks/ line (and the now-empty docs/02-components/ entry). - TODO.md: the latency-target item now notes the old #66 harness was removed and a runner for the current OTLP stack is still needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
End-to-end benchmark for the central paper claim: backend query latency and throughput for ASAP (sketch + ad-hoc S3 fallback) vs Prometheus alone vs VictoriaMetrics alone, on the same input stream.
benchmarks/docker-compose.yml— augmentsasap-quickstart(untouched) with VictoriaMetrics + a cold-store bind-mount + a queryengine env knob.benchmarks/config/vmagent-scrape.yml— VM scrape config matching Prom 1:1.benchmarks/queries/adhoc_suite.json— 7 ad-hoc PromQL queries (regex matchers, label_replace, rate, increase, exact histogram, negative regex, raw selector) that fall outside the sketch capability.run_asap_workloads,run_prom,run_vm,run_concurrency_sweep) — same suites across all three backends; sweep does C ∈ {1, 4, 16, 64} per backend.compare_three_way.py— Tables 1/2/3 (per-query latency, sketch error vs Prom, throughput-vs-C) + capability matrix.seed_cold_store.py— pre-populates 63 JSONL series in the S3 layout so the W2 fallback path returns non-empty.run_full_eval.sh— orchestrator with--downand--skip-sweep.Test plan
docker compose config --quietexits 0 on the merged composeseed_cold_store.pygenerates 315 JSONL samples in the expected layoutcompare_three_way.pysurvives missing inputsbash -nrun_full_eval.shinvocation (~3–5 min stack-up + ~5 min sweep) — deferred so reviewers can sanity-check the harness firstKnown caveat (de-scoped per the user's review)
asap-query-engine'smain.rsbuildsAdapterConfig::prometheus_promql(...)only.ColdFallback+LocalFsColdStoretypes and theprometheus_promql_with_coldconstructor exist, but no CLI flag wiresASAP_COLD_STORE_ROOTinto them. Until that flag lands, the W2 ASAP results reflect only the Prometheus-forwarding leg, not the local-FS cold tier. Bind-mount, seed script, and adhoc suite are correct as-is — the harness will exercise the cold path the moment the flag is added inmain.rs.🤖 Generated with Claude Code