Skip to content

mvp v6 phase D: fake-exporter freshness probes + measure_freshness.py - #299

Merged
zzylol merged 3 commits into
mainfrom
mvp/v6-phase-d-impl
May 6, 2026
Merged

zzylol merged 3 commits into
mainfrom
mvp/v6-phase-d-impl

Conversation

@zzylol

@zzylol zzylol commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fake-exporter now emits the three v6 freshness probes
    (http_freshness_probe_{raw,warm,archive}) at 1 Hz. Each is a
    Float64Counter whose cumulative value equals the unix_ms of the
    most recent emission, so last_over_time(<probe>[10s]) returns a
    timestamp directly. Disabled with EXPORTER_FRESHNESS_PROBES=off,
    rate via EXPORTER_FRESHNESS_PROBE_HZ.
  • measure_freshness.py gains a v6 poll-only mode matching the
    spec CLI (--query-endpoint / --probe / --path-label / --duration / --poll-interval-ms / --output) and v6 CSV schema
    (path,sample_ts_ms,observed_ts_ms,delta_ms). v4 mode is preserved
    for run_mvp_demo.sh on origin/main; mode is selected at runtime
    from the flag set.
  • run_freshness_phase.sh drives the three probe paths sequentially
    and writes $OUT_DIR/freshness/{raw,warm,archive}.csv. Endpoints
    are overrideable via flags / ASAP_FRESHNESS_*_ENDPOINT env vars.

Routing (probe metric name → tier) is owned by the agent / gateway
pipeline configs from Phase C overlay; the fake-exporter and the
script are both path-agnostic. See
deploy/configs/mvp-v6-freshness-probes.yaml for the schema and
docs/spec-mvp-v6-controller-driven-multi-stage-demo.md §⑥ for the
protocol.

run_mvp_demo.sh and mvp_report.py are intentionally not touched —
the v5 demo is daemonized using them right now (PID 1945011 in
/tmp/asap-collector-v5-worktree). Phase E will wire the v6 demo
driver to call run_freshness_phase.sh and extend mvp_report.py
to consume the v6 CSV schema.

Test plan

  • go test ./deploy/fake-exporter/... — 4 new probe tests +
    pre-existing tests all pass; verifies timestamp invariant within
    scheduler-jitter slack, EXPORTER_FRESHNESS_PROBES=off kill switch,
    monotonicity across two Collect()s, and the canonical name list
    matches the spec.
  • pytest deploy/scripts/tests/ — 15 unit tests covering query
    parsing (success / empty / error / bad json / non-success), an
    end-to-end CSV write against an in-process HTTP stub, empty-result
    handling, mode-selection edge cases, and the summary formatter for
    0 / 1 / many samples.
  • End-to-end smoke of run_freshness_phase.sh against a stub
    Prometheus that returns observed=now-100ms; all three CSVs land
    with delta_ms ≈ 100.
  • docker build -f deploy/docker/Dockerfile.fake-exporter --build-context sketchlib-go=... -t asap/fake-exporter:dev .
    succeeds — the rebuild will be picked up next time the v6 demo
    runs.

🤖 Generated with Claude Code

zzylol and others added 3 commits May 6, 2026 17:16
Adds three Float64Counter probes — http_freshness_probe_{raw,warm,
archive} — whose cumulative value equals the unix_ms of the most
recent emission. The replay client polls last_over_time(<probe>[10s])
and computes delta = response_ts_ms - observed_value to measure
end-to-end freshness through each of the three serving paths.

Mechanic: each tick Add()s (now_ms - last_emit_ms). SDK cumulative
temporality means the wire-side value is exactly last_emit_ms, so
the very first observation is also a usable timestamp.

Env knobs:
  EXPORTER_FRESHNESS_PROBES        on | off (default on)
  EXPORTER_FRESHNESS_PROBE_HZ      tick rate, Hz (default 1)

Routing (metric name → tier) is owned by the agent/gateway pipeline
configs from Phase C overlay; the fake-exporter is path-agnostic.

Tests cover: timestamp invariant within scheduler-jitter slack,
EXPORTER_FRESHNESS_PROBES=off kill switch, monotonicity across two
Collect() calls, and a guardrail that the canonical name list
matches the spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a poll-only mode to measure_freshness.py that matches the
v6 spec: the fake-exporter is the producer (probes.go); this script
just polls a Prometheus-style /api/v1/query endpoint, extracts
last_over_time(<probe>[10s]), and computes
delta_ms = response_ts_ms - observed_value.

CLI:
  --query-endpoint URL
  --probe {http_freshness_probe_raw|warm|archive}
  --path-label STR
  --duration SECS
  --poll-interval-ms INT
  --output CSV

Output schema: path,sample_ts_ms,observed_ts_ms,delta_ms.
Stderr summary line on exit: count, p50, p99.

The pre-existing v4 mode (--baseline / --otlp-http / --query / --out)
is preserved so callers on origin/main keep working — mode is selected
at runtime from the flag set, mixing the two is rejected with a clear
error. Phase E will retire v4 once run_mvp_demo.sh stops calling the
legacy form.

15 unit tests cover query parsing (success / empty / error / bad json
/ non-success), end-to-end CSV write against an in-process HTTP stub,
empty-result handling, mode-selection edge cases, and the summary
formatter for 0 / 1 / many samples.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drives the three v6 freshness probe paths sequentially against
their respective query endpoints:

  raw      → http://prometheus-b0:9090   (B0 Prometheus)
  warm     → http://backend:8080         (sketch warm tier)
  archive  → http://backend:8080         (Gorilla-archive)

Each path produces $OUT_DIR/freshness/{raw,warm,archive}.csv via
measure_freshness.py. Sequential by design — every invocation is
poll-only (the fake-exporter has been emitting all three probes
the whole time), and serializing makes the stderr summary lines
easy to read in the demo log.

Endpoints are overrideable via flags (or the ASAP_FRESHNESS_*_ENDPOINT
env vars) for host-mode runs against published ports. Phase E wires
the eventual v6 driver to call this script after the soak window;
this commit only lands the helper, since run_mvp_demo.sh is currently
locked by the v5 daemonized run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 0fff0b7 into main May 6, 2026
@zzylol
zzylol deleted the mvp/v6-phase-d-impl branch May 9, 2026 18:00
zzylol added a commit that referenced this pull request May 19, 2026
…W config + per-series quantile

Bundle of supporting changes for apples-to-apples accuracy validation
of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the
quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend
PR #297 + #299, ASAPCollector cumulativetodelta commit above).

Changes:

1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series
   PRNG (math/rand) deterministically so all 3 arms in a sequential
   run emit identical latency value sequences. Without it, cross-arm
   sampling noise masked DDSketch ε behavior in the accuracy report
   (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine
   error). Default seed of 42 in run_demo.sh; back-compat when unset
   (auto-random, original behavior). Per-series PRNG uses
   `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so
   distinct producers + series get distinct sequences.

2. **b0/b1 PRW exporter config — add_metric_suffixes: false** —
   VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric
   names with `WithUnit("ms")` annotation (Prometheus naming convention).
   Asap tier preserves the original name. Without this fix, the
   accuracy comparison probe queries `http_requests_total_latency_ms`
   land on different metric names per tier — apples-to-oranges.

3. **mvp-workload.yaml** (singlenode + multinode) — dropped
   `grouping_labels: [zone]` from quantile-metric entries
   (http_requests_total_latency_ms, request_size_bytes). Per-series
   DDSketch / KLL sketches preserve PromQL's per-series semantics so
   `quantile_over_time(...)` returns comparable per-series rows in
   both asap and baseline. Counter-metric entries retain
   grouping_labels: [zone] (Sum aggregation is mergeable; no semantic
   asymmetry there). The KLL override on the latency entry is also
   tracked here for the DDSketch-vs-KLL accuracy comparison documented
   in mvp_smoke_test_findings.md.

4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers.

Accuracy results post all fixes (multinode all-arms):
- p50 quantile: 0.4-0.6% rel-err vs baseline ✅
- p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance)
- max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch)
- sum-by-zone/rate/topk: post-#299 returns delta-window semantics
  (not directly comparable to baseline cumulative as raw numbers, but
  bug-correct; ratio drops from ~300× to ~1× when normalized).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 19, 2026
… ASAPQuery-backend #299)

Static bootstrap counterpart to ASAPQuery-backend PR #299. Without
this static fix, the asap-otel agent's BOOTSTRAP config (used before
the controller's OpAMP push lands) lacks cumulativetodelta and sends
cumulative-temporality Counter values to the backend's SumAccumulator
— triggering the ~300× per-window quadratic blowup that bug #298
documented. After OpAMP push (typed-stage-split emit from #299), the
running config has it; this static yaml just keeps the pre-OpAMP
boot window correct too.

`match_type: strict` keeps the processor a no-op for gauges
(http_requests_total_latency_ms etc. — quantile workloads unaffected).
b0/b1 baseline configs are intentionally untouched: VictoriaMetrics
expects cumulative (Prometheus convention).

Closes the ASAPCollector half of #298.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 19, 2026
…W config + per-series quantile

Bundle of supporting changes for apples-to-apples accuracy validation
of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the
quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend
PR #297 + #299, ASAPCollector cumulativetodelta commit above).

Changes:

1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series
   PRNG (math/rand) deterministically so all 3 arms in a sequential
   run emit identical latency value sequences. Without it, cross-arm
   sampling noise masked DDSketch ε behavior in the accuracy report
   (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine
   error). Default seed of 42 in run_demo.sh; back-compat when unset
   (auto-random, original behavior). Per-series PRNG uses
   `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so
   distinct producers + series get distinct sequences.

2. **b0/b1 PRW exporter config — add_metric_suffixes: false** —
   VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric
   names with `WithUnit("ms")` annotation (Prometheus naming convention).
   Asap tier preserves the original name. Without this fix, the
   accuracy comparison probe queries `http_requests_total_latency_ms`
   land on different metric names per tier — apples-to-oranges.

3. **mvp-workload.yaml** (singlenode + multinode) — dropped
   `grouping_labels: [zone]` from quantile-metric entries
   (http_requests_total_latency_ms, request_size_bytes). Per-series
   DDSketch / KLL sketches preserve PromQL's per-series semantics so
   `quantile_over_time(...)` returns comparable per-series rows in
   both asap and baseline. Counter-metric entries retain
   grouping_labels: [zone] (Sum aggregation is mergeable; no semantic
   asymmetry there). The KLL override on the latency entry is also
   tracked here for the DDSketch-vs-KLL accuracy comparison documented
   in mvp_smoke_test_findings.md.

4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers.

Accuracy results post all fixes (multinode all-arms):
- p50 quantile: 0.4-0.6% rel-err vs baseline ✅
- p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance)
- max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch)
- sum-by-zone/rate/topk: post-#299 returns delta-window semantics
  (not directly comparable to baseline cumulative as raw numbers, but
  bug-correct; ratio drops from ~300× to ~1× when normalized).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 19, 2026
…ativetodelta + seed) (#398)

* fix(deploy): cumulativetodelta upstream of agent routing (paired with ASAPQuery-backend #299)

Static bootstrap counterpart to ASAPQuery-backend PR #299. Without
this static fix, the asap-otel agent's BOOTSTRAP config (used before
the controller's OpAMP push lands) lacks cumulativetodelta and sends
cumulative-temporality Counter values to the backend's SumAccumulator
— triggering the ~300× per-window quadratic blowup that bug #298
documented. After OpAMP push (typed-stage-split emit from #299), the
running config has it; this static yaml just keeps the pre-OpAMP
boot window correct too.

`match_type: strict` keeps the processor a no-op for gauges
(http_requests_total_latency_ms etc. — quantile workloads unaffected).
b0/b1 baseline configs are intentionally untouched: VictoriaMetrics
expects cumulative (Prometheus convention).

Closes the ASAPCollector half of #298.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(demo): accuracy-validation harness — EXPORTER_SEED + baseline PRW config + per-series quantile

Bundle of supporting changes for apples-to-apples accuracy validation
of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the
quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend
PR #297 + #299, ASAPCollector cumulativetodelta commit above).

Changes:

1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series
   PRNG (math/rand) deterministically so all 3 arms in a sequential
   run emit identical latency value sequences. Without it, cross-arm
   sampling noise masked DDSketch ε behavior in the accuracy report
   (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine
   error). Default seed of 42 in run_demo.sh; back-compat when unset
   (auto-random, original behavior). Per-series PRNG uses
   `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so
   distinct producers + series get distinct sequences.

2. **b0/b1 PRW exporter config — add_metric_suffixes: false** —
   VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric
   names with `WithUnit("ms")` annotation (Prometheus naming convention).
   Asap tier preserves the original name. Without this fix, the
   accuracy comparison probe queries `http_requests_total_latency_ms`
   land on different metric names per tier — apples-to-oranges.

3. **mvp-workload.yaml** (singlenode + multinode) — dropped
   `grouping_labels: [zone]` from quantile-metric entries
   (http_requests_total_latency_ms, request_size_bytes). Per-series
   DDSketch / KLL sketches preserve PromQL's per-series semantics so
   `quantile_over_time(...)` returns comparable per-series rows in
   both asap and baseline. Counter-metric entries retain
   grouping_labels: [zone] (Sum aggregation is mergeable; no semantic
   asymmetry there). The KLL override on the latency entry is also
   tracked here for the DDSketch-vs-KLL accuracy comparison documented
   in mvp_smoke_test_findings.md.

4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers.

Accuracy results post all fixes (multinode all-arms):
- p50 quantile: 0.4-0.6% rel-err vs baseline ✅
- p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance)
- max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch)
- sum-by-zone/rate/topk: post-#299 returns delta-window semantics
  (not directly comparable to baseline cumulative as raw numbers, but
  bug-correct; ratio drops from ~300× to ~1× when normalized).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant