mvp: 5-sketch workload — fake-exporter emits 4 new metrics, replay covers all query classes - #338
Merged
Merged
Conversation
…vers all query classes Adds the four new metric families that close the gap between the existing DDSketch + Sum coverage and the full 5-sketch MVP demo (issue #46). All four are gated by EXPORTER_FIVE_SKETCH=on/off (default on) and respect the existing PER_AGENT_CARDINALITY / EXPORTER_FREQ_HZ producer knobs. New metrics (deploy/fake-exporter/five_sketch_workload.go): - request_size_bytes Gauge, log-normal 100B–10KB (KLL) - unique_users_per_min Counter, rotating user-id pool (HLL) - top_endpoint_qps Counter, Zipfian endpoint pool (CountSketch) - endpoint_request_freq Counter, same Zipfian shape (CountMinSketch) Workload spec (deploy/configs/mvp-workload.yaml): Adds four entries with sketch_family_override + target_path hints for the controller's planner (round-tripped silently through the current WorkloadEntry struct so the parallel capability_matching agent's planner can pick them up without a schema rename). Replay coverage (deploy/scripts/run_mvp_demo.sh, promql_replay.py): Bumps replay rotation from 3 to 7 queries (quantile, sum_rate, kll-quantile, count_unique, topk, frequency) and QPS from 5 to 8 so per-class samples ≥340 over the 300s soak (≥100 floor for accuracy reduction). Adds the `frequency` kind to the replay client's validator so `rate(...[5m])` results are accepted. Tests: - go test ./deploy/fake-exporter/... — 7 tests pass (3 new for the five-sketch workload + 4 pre-existing). - python3 -m pytest deploy/scripts/tests/test_promql_replay.py — 9 tests pass covering load_queries kind validation + the five PromQL result shapes (vector / scalar / topk-vector / rate-vector / sum-by-vector / json-error). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…eries ε PR #338 added frequency to the replay client's QUERY_KINDS but the reducer silently skipped frequency rows because parse_query had no `rate()` regex. This adds: - `_RATE_RE` matching `rate(metric[5m])` shape - parse_query returns ("frequency", {"metric": ...}) - new `extract_per_series` helper: PromQL vector → {labels-key → value} - frequency branch in reduce_cell_via_archive: pair warm vs archive per-series, compute mean absolute additive error normalised by truth total — comparable column with rel_err for other kinds - frequency branch in archive_miss fallback path so warm_answer is still captured for diagnostics Refs #46. Closes the gap PR #338 flagged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…eries ε (#343) PR #338 added frequency to the replay client's QUERY_KINDS but the reducer silently skipped frequency rows because parse_query had no `rate()` regex. This adds: - `_RATE_RE` matching `rate(metric[5m])` shape - parse_query returns ("frequency", {"metric": ...}) - new `extract_per_series` helper: PromQL vector → {labels-key → value} - frequency branch in reduce_cell_via_archive: pair warm vs archive per-series, compute mean absolute additive error normalised by truth total — comparable column with rel_err for other kinds - frequency branch in archive_miss fallback path so warm_answer is still captured for diagnostics Refs #46. Closes the gap PR #338 flagged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 9, 2026
…352) The five-sketch workload's HLL inner-label fan-out (user_id × outer label set) was the dominant SDK output contributor, pushing the post-PR-#338 demo to ~2.46 MB/s and blowing up agent → gateway bandwidth. Lower the two defaults that drive that fan-out: - EXPORTER_FIVE_SKETCH_USER_POOL: 1000 -> 100 (floor 500 -> 50). HLL still has a non-trivial active-user cardinality to estimate (100 distinct ids), and the rotate window keeps the per-minute semantic intact. - EXPORTER_CARDINALITY (binary default): 1000 -> 500 so direct-binary use also matches the demo script's PER_AGENT_CARDINALITY=500 default. With N_PRODUCERS=10 the aggregate gateway cardinality lands at ~5K series (was 10K-25K+). top_endpoint_qps / endpoint_request_freq stay at 50 endpoints (already small); request_size_bytes is scalar. PER_AGENT_CARDINALITY=500 in run_mvp_demo.sh and the mvp-multi-stage compose env propagate already match the new binary default; mvp-workload.yaml does not pin cardinality. Verified with go test ./deploy/fake-exporter/... and a local binary smoke (startup log shows cardinality=500 users=100 endpoints=50). Refs #46. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
May 9, 2026
… (#46) (#355) Root cause: PR #353 spawned six gorillas3 instances (one per per-family pipeline under the 5-sketch routing topology), each with its own in-memory windowState. With the static placeholder's 60 s flush interval and PR #338's fake-exporter cardinality, the agent buffered the full per-pipeline working set in RAM BEFORE the first flush ticker fired — peak crossed the 1.5 GiB per-agent memory limit, the agent was OOM-killed, and zero TSDB blocks ever reached MinIO. PR #354's flat-slice OOB-tolerance approach compounded the problem at flush time (O(N_samples) flat slice + interleaved Head chunk creation across every series) but the agent never lived long enough for that path to fire. Result in /tmp/asap-mvp-rerun-bug34/asap/measurements/accuracy.csv: 1713 archive_miss / 0 archive_ok (was 343 archive_ok pre-#354). Two-part fix: 1) opentelemetry-collector-contrib-patch/processor/gorillas3processor: replace PR #354's flat-sort-and-skip path with a series-visit- ordered append. Each series' points are still sorted ascending locally, but series are now visited in ascending order of each series' EARLIEST sample timestamp. That guarantees the very first `app.Append(...)` carries the global minimum, anchoring the appender's `minValidTime = globalMin - chunkRange/2` so every other in-window sample passes the OOB check — without ever allocating an O(N_samples) flat slice or interleaving Head series creation. Memory footprint at flush is bounded by `max(series points, Head per-series state)` instead of total sample count. PR #354's defensive ErrOutOfBounds tolerance (`gorillas3_tsdb_oob_samples_dropped_total` counter + warn log, no rollback) is retained for genuinely late-arriving samples. 2) deploy/configs/asap-otel-agent-b6-asap-single-sketch.yaml: reduce gorillas3.window_interval from 60 s → 5 s. With six per-pipeline windowStates instead of one, the per-instance in-flight buffer is the dominant memory pressure. 5 s caps the per-pipeline window peak at ~150 MiB, putting the steady-state aggregate inside the agent's 1.5 GiB budget. tsdb_block_duration stays at 60 s so the on-S3 layout still matches the Thanos store-gateway sync interval; flushes just roll smaller sub-blocks that thanos-compact will merge. Verification (90 s soak diagnose, repeated): before: mc ls myminio/asap-gorilla-tsdb/ → empty curl -s http://localhost:19092/api/v1/labels → {"data":["__name__"]} (no metric labels surfaced) after: mc ls myminio/asap-gorilla-tsdb/ → 27 blocks across all six pipelines (countsketch_path, countminsketch_path, ddsketch_path, hll_path, kll_path, raw_passthrough) curl -s http://localhost:19092/api/v1/label/__name__/values → ["endpoint_request_freq","http_freshness_probe_archive", "http_freshness_probe_raw","http_freshness_probe_warm", "http_requests_total","http_requests_total_latency_ms", "request_size_bytes","top_endpoint_qps", "unique_users_per_min"] all 5 sketched metrics + raw + 3 freshness probes. Coverage: TestTSDBBlockBuilder_HighCardinalityMemoryBound — 200-series / 60-sample window with non-overlapping per-series time slices, so the global minimum lives on a different series from the global maximum. Asserts 0 OOB drops + every sample lands in the block. Pre-this-PR's algorithm would still have passed correctness but allocated the O(N) flat slice; this test pins the visit-order invariant going forward. Existing TSDB OOB regressions (RotatingCardinalityNoOOB, WideTimestampSpanNoOOB, FlushTSDB_OOBDoesNotCrashProcessor) all still pass with the new visit-order path — same OOB-tolerance contract, different memory footprint. Closes #46. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the gap between today's DDSketch + Sum-only producer signal and the
full 5-sketch MVP demo (issue #46). Four new metrics are added to the
fake-exporter, gated by
EXPORTER_FIVE_SKETCH=on/off(default on); thecontroller's workload registry is extended with planner hints; and the
replay client's rotation now exercises all six query classes.
Six metrics × six query classes (verbatim contract from issue #46)
http_requests_totalsum by (zone) (rate(http_requests_total[5m]))http_latency_ms(http_requests_total_latency_ms)quantile_over_time(0.99, http_requests_total_latency_ms[1m])request_size_bytesquantile_over_time(0.99, request_size_bytes[1m])unique_users_per_mincount(unique_users_per_min)top_endpoint_qpstopk(5, top_endpoint_qps)endpoint_request_freqrate(endpoint_request_freq[5m])The first two existed before this PR; the last four are new.
Producer changes (
deploy/fake-exporter/)five_sketch_workload.go— wires the four new instruments and startsper-outer-series goroutines reusing the same zone/rack/node/pod label
schema as
runSynthetic. Distribution shapes:request_size_bytes: log-normal in [100B, 10KB] — heavy right tailso KLL's rank-error metric is the natural oracle.
unique_users_per_min: rotating user-id pool of 500–2000 (envEXPORTER_FIVE_SKETCH_USER_POOL), rotating every 60s by default(env
EXPORTER_FIVE_SKETCH_USER_ROTATE) so the active set HLLestimates actually slides.
top_endpoint_qps/endpoint_request_freq: Zipfian (s=1.2)over 50 endpoints (env
EXPORTER_FIVE_SKETCH_ENDPOINTS/EXPORTER_FIVE_SKETCH_ZIPF_S) so heavy hitters dominate.main.go— callsstartFiveSketchWorkloadfromrunSyntheticalongside the existing counter + gauge so all six metrics ride the
same SDK View / PeriodicReader pipeline.
EXPORTER_FIVE_SKETCH=offshort-circuits all four; the existingDDSketch + Sum signal continues unchanged.
Workload spec (
deploy/configs/mvp-workload.yaml)Adds four entries (entries 5–8) keyed by the new metric names with
sketch_family_override+target_pathhints (KLL / HLL / CountSketch/ CountMinSketch, all routed
target_path: warm). The currentWorkloadEntrystruct incontroller/src/config/workloads.rsdoesn'tread those two keys yet —
serdeignores unknown fields by default,so the YAML round-trips silently until the parallel
capability_matching + planner work picks them up. Pre-existing entries
(entries 1–4) are untouched.
Replay client (
deploy/scripts/promql_replay.py)QUERY_KINDSextended withfrequencysorate(...[5m])queriesare validated rather than rejected.
result_type+resultverbatim across vector / scalar / matrix shapes, so thereducer keys on
kindto pick the right oracle.MVP demo driver (
deploy/scripts/run_mvp_demo.sh)replay-queries.jsonrotation grows from 3 → 7 queries covering allsix families.
samples/class, ≥100 floor for the accuracy reducer).
Test coverage
go test ./deploy/fake-exporter/...— 3 new tests for thefive-sketch workload (all-metrics-emit, EXPORTER_FIVE_SKETCH=off
kill switch, env-string parsing) on top of the existing 4. All pass.
python3 -m pytest deploy/scripts/tests/test_promql_replay.py— 9new tests covering
load_querieskind validation + run_query result-shape parsing for vector / scalar / topk-vector / rate-vector /
sum-by-vector / json-error. All pass.
Test plan
go test ./deploy/fake-exporter/...python3 -m pytest deploy/scripts/tests/test_promql_replay.pymvp-workload.yamlparses (yaml.safe_load→ 8 entries)replay-queries.jsonparses (json.load→ 7 queries)lands its
sketch_family_overridereaderBoundaries respected
This PR only touches
deploy/fake-exporter/,deploy/configs/mvp-workload.yaml, and the replay client +demo-driver scripts. No controller / planner / mvp_report.py edits, so
no conflict with the three parallel agents.
🤖 Generated with Claude Code