fix(deploy): accuracy-validation harness (per-series quantile + cumulativetodelta + seed) - #398
Merged
Merged
Conversation
… ASAPQuery-backend #299) Static bootstrap counterpart to ASAPQuery-backend PR #299. Without this static fix, the asap-otel agent's BOOTSTRAP config (used before the controller's OpAMP push lands) lacks cumulativetodelta and sends cumulative-temporality Counter values to the backend's SumAccumulator — triggering the ~300× per-window quadratic blowup that bug #298 documented. After OpAMP push (typed-stage-split emit from #299), the running config has it; this static yaml just keeps the pre-OpAMP boot window correct too. `match_type: strict` keeps the processor a no-op for gauges (http_requests_total_latency_ms etc. — quantile workloads unaffected). b0/b1 baseline configs are intentionally untouched: VictoriaMetrics expects cumulative (Prometheus convention). Closes the ASAPCollector half of #298. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…W config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
fix/per-series-sketches-for-quantile-accuracy
branch
from
May 19, 2026 00:23
db80139 to
89717c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bundle of deploy-side changes for the accuracy-validation work that landed across:
Commits
feat(demo): accuracy-validation harness— EXPORTER_SEED + baseline PRWadd_metric_suffixes: false+ per-series quantile sketches + run_demo.sh wiringfix(deploy): cumulativetodelta upstream of agent routing— static-bootstrap counterpart to ASAPQuery-backend mvp v6 phase D: fake-exporter freshness probes + measure_freshness.py #299fix(multinode): WARMUP_S=60 + grouping_labels + queries-e2e wave queries— harness-tuning + quantile-shape replacement (max-by-zone)Accuracy results post all fixes
quantile_over_time(0.5, X[5m])quantile_over_time(0.99, X[5m])max by (zone) (quantile_over_time(0.99, X[5m]))sum by (zone) (http_requests_total)b0_rate × window_secsTest plan
sum by (zone) (http_requests_total)returns ~60M (stable per-window) vs pre-fix ~770M (cubic blowup)