mvp v6 phase D: fake-exporter freshness probes + measure_freshness.py - #299
Merged
Merged
Conversation
Adds three Float64Counter probes — http_freshness_probe_{raw,warm,
archive} — whose cumulative value equals the unix_ms of the most
recent emission. The replay client polls last_over_time(<probe>[10s])
and computes delta = response_ts_ms - observed_value to measure
end-to-end freshness through each of the three serving paths.
Mechanic: each tick Add()s (now_ms - last_emit_ms). SDK cumulative
temporality means the wire-side value is exactly last_emit_ms, so
the very first observation is also a usable timestamp.
Env knobs:
EXPORTER_FRESHNESS_PROBES on | off (default on)
EXPORTER_FRESHNESS_PROBE_HZ tick rate, Hz (default 1)
Routing (metric name → tier) is owned by the agent/gateway pipeline
configs from Phase C overlay; the fake-exporter is path-agnostic.
Tests cover: timestamp invariant within scheduler-jitter slack,
EXPORTER_FRESHNESS_PROBES=off kill switch, monotonicity across two
Collect() calls, and a guardrail that the canonical name list
matches the spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a poll-only mode to measure_freshness.py that matches the
v6 spec: the fake-exporter is the producer (probes.go); this script
just polls a Prometheus-style /api/v1/query endpoint, extracts
last_over_time(<probe>[10s]), and computes
delta_ms = response_ts_ms - observed_value.
CLI:
--query-endpoint URL
--probe {http_freshness_probe_raw|warm|archive}
--path-label STR
--duration SECS
--poll-interval-ms INT
--output CSV
Output schema: path,sample_ts_ms,observed_ts_ms,delta_ms.
Stderr summary line on exit: count, p50, p99.
The pre-existing v4 mode (--baseline / --otlp-http / --query / --out)
is preserved so callers on origin/main keep working — mode is selected
at runtime from the flag set, mixing the two is rejected with a clear
error. Phase E will retire v4 once run_mvp_demo.sh stops calling the
legacy form.
15 unit tests cover query parsing (success / empty / error / bad json
/ non-success), end-to-end CSV write against an in-process HTTP stub,
empty-result handling, mode-selection edge cases, and the summary
formatter for 0 / 1 / many samples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drives the three v6 freshness probe paths sequentially against
their respective query endpoints:
raw → http://prometheus-b0:9090 (B0 Prometheus)
warm → http://backend:8080 (sketch warm tier)
archive → http://backend:8080 (Gorilla-archive)
Each path produces $OUT_DIR/freshness/{raw,warm,archive}.csv via
measure_freshness.py. Sequential by design — every invocation is
poll-only (the fake-exporter has been emitting all three probes
the whole time), and serializing makes the stderr summary lines
easy to read in the demo log.
Endpoints are overrideable via flags (or the ASAP_FRESHNESS_*_ENDPOINT
env vars) for host-mode runs against published ports. Phase E wires
the eventual v6 driver to call this script after the soak window;
this commit only lands the helper, since run_mvp_demo.sh is currently
locked by the v5 daemonized run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 6, 2026
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…W config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
4 tasks
zzylol
added a commit
that referenced
this pull request
May 19, 2026
… ASAPQuery-backend #299) Static bootstrap counterpart to ASAPQuery-backend PR #299. Without this static fix, the asap-otel agent's BOOTSTRAP config (used before the controller's OpAMP push lands) lacks cumulativetodelta and sends cumulative-temporality Counter values to the backend's SumAccumulator — triggering the ~300× per-window quadratic blowup that bug #298 documented. After OpAMP push (typed-stage-split emit from #299), the running config has it; this static yaml just keeps the pre-OpAMP boot window correct too. `match_type: strict` keeps the processor a no-op for gauges (http_requests_total_latency_ms etc. — quantile workloads unaffected). b0/b1 baseline configs are intentionally untouched: VictoriaMetrics expects cumulative (Prometheus convention). Closes the ASAPCollector half of #298. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…W config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…ativetodelta + seed) (#398) * fix(deploy): cumulativetodelta upstream of agent routing (paired with ASAPQuery-backend #299) Static bootstrap counterpart to ASAPQuery-backend PR #299. Without this static fix, the asap-otel agent's BOOTSTRAP config (used before the controller's OpAMP push lands) lacks cumulativetodelta and sends cumulative-temporality Counter values to the backend's SumAccumulator — triggering the ~300× per-window quadratic blowup that bug #298 documented. After OpAMP push (typed-stage-split emit from #299), the running config has it; this static yaml just keeps the pre-OpAMP boot window correct too. `match_type: strict` keeps the processor a no-op for gauges (http_requests_total_latency_ms etc. — quantile workloads unaffected). b0/b1 baseline configs are intentionally untouched: VictoriaMetrics expects cumulative (Prometheus convention). Closes the ASAPCollector half of #298. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(demo): accuracy-validation harness — EXPORTER_SEED + baseline PRW config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(
http_freshness_probe_{raw,warm,archive}) at 1 Hz. Each is aFloat64Counter whose cumulative value equals the unix_ms of the
most recent emission, so
last_over_time(<probe>[10s])returns atimestamp directly. Disabled with
EXPORTER_FRESHNESS_PROBES=off,rate via
EXPORTER_FRESHNESS_PROBE_HZ.measure_freshness.pygains a v6 poll-only mode matching thespec CLI (
--query-endpoint / --probe / --path-label / --duration / --poll-interval-ms / --output) and v6 CSV schema(
path,sample_ts_ms,observed_ts_ms,delta_ms). v4 mode is preservedfor run_mvp_demo.sh on origin/main; mode is selected at runtime
from the flag set.
run_freshness_phase.shdrives the three probe paths sequentiallyand writes
$OUT_DIR/freshness/{raw,warm,archive}.csv. Endpointsare overrideable via flags /
ASAP_FRESHNESS_*_ENDPOINTenv vars.Routing (probe metric name → tier) is owned by the agent / gateway
pipeline configs from Phase C overlay; the fake-exporter and the
script are both path-agnostic. See
deploy/configs/mvp-v6-freshness-probes.yamlfor the schema anddocs/spec-mvp-v6-controller-driven-multi-stage-demo.md§⑥ for theprotocol.
run_mvp_demo.shandmvp_report.pyare intentionally not touched —the v5 demo is daemonized using them right now (PID 1945011 in
/tmp/asap-collector-v5-worktree). Phase E will wire the v6 demodriver to call
run_freshness_phase.shand extendmvp_report.pyto consume the v6 CSV schema.
Test plan
go test ./deploy/fake-exporter/...— 4 new probe tests +pre-existing tests all pass; verifies timestamp invariant within
scheduler-jitter slack, EXPORTER_FRESHNESS_PROBES=off kill switch,
monotonicity across two Collect()s, and the canonical name list
matches the spec.
pytest deploy/scripts/tests/— 15 unit tests covering queryparsing (success / empty / error / bad json / non-success), an
end-to-end CSV write against an in-process HTTP stub, empty-result
handling, mode-selection edge cases, and the summary formatter for
0 / 1 / many samples.
run_freshness_phase.shagainst a stubPrometheus that returns observed=now-100ms; all three CSVs land
with delta_ms ≈ 100.
docker build -f deploy/docker/Dockerfile.fake-exporter --build-context sketchlib-go=... -t asap/fake-exporter:dev .succeeds — the rebuild will be picked up next time the v6 demo
runs.
🤖 Generated with Claude Code