fix: static agent placeholder uses 5-sketch routing-connector pipeline (was single ddsketch) - #353
Merged
Merged
Conversation
…e (was single ddsketch) PR #350's report flagged that OpAMP RemoteConfig push isn't applied by the agent at runtime — the effective-config preview keeps showing the static bootstrap YAML loaded from the volume mount, so the running pipeline is whatever this static file declares. Previously the placeholder declared a single [gorillas3, ddsketch, batch] pipeline, so agents only ran ONE sketch (DDSketch) regardless of the controller plan. KLL / HLL / CountSketch / CountMinSketch processors never ran. This PR rewrites the static placeholder to mirror the typed-emit shape from controller/src/config/stage_config.rs::emit_edge_yaml_5sketch_routing: all 5 sketch processors loaded at top level, the OTel v0.106 routing connector under connectors: (NOT processors:), and 7 named pipelines (entry + raw_passthrough + 5 per-family paths). Each per-sketch pipeline runs gorillas3 first so the cold-tier write happens on raw samples before the sketch processor mutates the stream. The agent boots cleanly with no config-load errors — verified with `asap-otel validate` (exit 0) and a minimal compose stack run that reports "Everything is ready. Begin running and processing data." with all 5 sketch processors listed under their respective pipelines. Smoke test in deploy/configs/tests/ parses the YAML and pins: - All 5 sketch processors present at top level - routing in connectors: (not processors:) - All named pipelines (entry + raw_passthrough + 5 per-family) - Each per-sketch pipeline has gorillas3 first - routing.default_pipelines == [metrics/raw_passthrough] - context: metric on every routing table entry Refs #46. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 9, 2026
… (#46) (#355) Root cause: PR #353 spawned six gorillas3 instances (one per per-family pipeline under the 5-sketch routing topology), each with its own in-memory windowState. With the static placeholder's 60 s flush interval and PR #338's fake-exporter cardinality, the agent buffered the full per-pipeline working set in RAM BEFORE the first flush ticker fired — peak crossed the 1.5 GiB per-agent memory limit, the agent was OOM-killed, and zero TSDB blocks ever reached MinIO. PR #354's flat-slice OOB-tolerance approach compounded the problem at flush time (O(N_samples) flat slice + interleaved Head chunk creation across every series) but the agent never lived long enough for that path to fire. Result in /tmp/asap-mvp-rerun-bug34/asap/measurements/accuracy.csv: 1713 archive_miss / 0 archive_ok (was 343 archive_ok pre-#354). Two-part fix: 1) opentelemetry-collector-contrib-patch/processor/gorillas3processor: replace PR #354's flat-sort-and-skip path with a series-visit- ordered append. Each series' points are still sorted ascending locally, but series are now visited in ascending order of each series' EARLIEST sample timestamp. That guarantees the very first `app.Append(...)` carries the global minimum, anchoring the appender's `minValidTime = globalMin - chunkRange/2` so every other in-window sample passes the OOB check — without ever allocating an O(N_samples) flat slice or interleaving Head series creation. Memory footprint at flush is bounded by `max(series points, Head per-series state)` instead of total sample count. PR #354's defensive ErrOutOfBounds tolerance (`gorillas3_tsdb_oob_samples_dropped_total` counter + warn log, no rollback) is retained for genuinely late-arriving samples. 2) deploy/configs/asap-otel-agent-b6-asap-single-sketch.yaml: reduce gorillas3.window_interval from 60 s → 5 s. With six per-pipeline windowStates instead of one, the per-instance in-flight buffer is the dominant memory pressure. 5 s caps the per-pipeline window peak at ~150 MiB, putting the steady-state aggregate inside the agent's 1.5 GiB budget. tsdb_block_duration stays at 60 s so the on-S3 layout still matches the Thanos store-gateway sync interval; flushes just roll smaller sub-blocks that thanos-compact will merge. Verification (90 s soak diagnose, repeated): before: mc ls myminio/asap-gorilla-tsdb/ → empty curl -s http://localhost:19092/api/v1/labels → {"data":["__name__"]} (no metric labels surfaced) after: mc ls myminio/asap-gorilla-tsdb/ → 27 blocks across all six pipelines (countsketch_path, countminsketch_path, ddsketch_path, hll_path, kll_path, raw_passthrough) curl -s http://localhost:19092/api/v1/label/__name__/values → ["endpoint_request_freq","http_freshness_probe_archive", "http_freshness_probe_raw","http_freshness_probe_warm", "http_requests_total","http_requests_total_latency_ms", "request_size_bytes","top_endpoint_qps", "unique_users_per_min"] all 5 sketched metrics + raw + 3 freshness probes. Coverage: TestTSDBBlockBuilder_HighCardinalityMemoryBound — 200-series / 60-sample window with non-overlapping per-series time slices, so the global minimum lives on a different series from the global maximum. Asserts 0 OOB drops + every sample lands in the block. Pre-this-PR's algorithm would still have passed correctness but allocated the O(N) flat slice; this test pins the visit-order invariant going forward. Existing TSDB OOB regressions (RotatingCardinalityNoOOB, WideTimestampSpanNoOOB, FlushTSDB_OOBDoesNotCrashProcessor) all still pass with the new visit-order path — same OOB-tolerance contract, different memory footprint. Closes #46. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
PR #350's report flagged a real OpAMP-push gap: the controller emits the typed 5-sketch routing-connector YAML correctly, but the agent never APPLIES the OpAMP RemoteConfig push at runtime. The effective-config preview keeps showing the static bootstrap YAML loaded from the volume mount (see
deploy/docker-compose/mvp-multi-stage.ymlAGENT_CONFIG_A/AGENT_CONFIG_Bdefaults).Previously the placeholder declared a single
[gorillas3, ddsketch, batch]pipeline, so agents only ran ONE sketch (DDSketch) regardless of the controller plan. KLL / HLL / CountSketch / CountMinSketch processors never ran. The §3 architecture story ("one agent → five concurrent sketch pipelines → backend serves five concurrent query families") was broken at runtime.What
Rewrite
deploy/configs/asap-otel-agent-b6-asap-single-sketch.yamlto mirror the typed-emit shape fromcontroller/src/config/stage_config.rs::emit_edge_yaml_5sketch_routing:ddsketch,KLL,HLL,countsketch,countmin).routingunderconnectors:(NOTprocessors:) — OTel v0.106 removed the deprecatedroutingprocessor.metrics:+metrics/raw_passthrough+ 5 per-family paths.gorillas3first so cold-tier write happens on raw samples before sketch mutation.Residual half of #350 (out of scope here)
The controller emit uses
*processor-suffixed component names (ddsketchprocessor,kllprocessor, ...) and decorates each block withaggregation_id+sketch_kindkeys. None of those are accepted by the patched contrib build's component registry (verified viaasap-otel componentsandmapstructure:tags), so the controller-emitted YAML wouldn't load on agents either. That's the residual half of #350 — fix in the controller emitter, separate PR.Verification
asap-otel validate --config asap-otel-agent-b6-asap-single-sketch.yaml→ exit 0.docker compose ... up -d controller minio gateway agent-a) →Everything is ready. Begin running and processing data.Each per-family pipeline logsStarting <sketch> processor+Starting gorillas3 processor. Noerror|invalid|cannot|fatal|paniclines.Test plan
python3 deploy/configs/tests/test_static_placeholder_5sketch_routing.py— 8/8 pass.routinginconnectors:(notprocessors:).gorillas3and ends withbatch.routing.default_pipelines == [metrics/raw_passthrough].context: metricon every routing table entry.asap-otel validateexits 0.Everything is ready.Refs #46.
🤖 Generated with Claude Code