Skip to content

fix: static agent placeholder uses 5-sketch routing-connector pipeline (was single ddsketch) - #353

Merged
zzylol merged 1 commit into
mainfrom
fix/static-placeholder-5sketch-routing
May 9, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/static-placeholder-5sketch-routing

Conversation

@zzylol

@zzylol zzylol commented May 9, 2026

Copy link
Copy Markdown
Contributor

Why

PR #350's report flagged a real OpAMP-push gap: the controller emits the typed 5-sketch routing-connector YAML correctly, but the agent never APPLIES the OpAMP RemoteConfig push at runtime. The effective-config preview keeps showing the static bootstrap YAML loaded from the volume mount (see deploy/docker-compose/mvp-multi-stage.yml AGENT_CONFIG_A / AGENT_CONFIG_B defaults).

Previously the placeholder declared a single [gorillas3, ddsketch, batch] pipeline, so agents only ran ONE sketch (DDSketch) regardless of the controller plan. KLL / HLL / CountSketch / CountMinSketch processors never ran. The §3 architecture story ("one agent → five concurrent sketch pipelines → backend serves five concurrent query families") was broken at runtime.

What

Rewrite deploy/configs/asap-otel-agent-b6-asap-single-sketch.yaml to mirror the typed-emit shape from controller/src/config/stage_config.rs::emit_edge_yaml_5sketch_routing:

  • All 5 sketch processors at top level (ddsketch, KLL, HLL, countsketch, countmin).
  • routing under connectors: (NOT processors:) — OTel v0.106 removed the deprecated routingprocessor.
  • 7 named pipelines: entry metrics: + metrics/raw_passthrough + 5 per-family paths.
  • Each per-sketch pipeline runs gorillas3 first so cold-tier write happens on raw samples before sketch mutation.
service:
  pipelines:
    metrics:                     { receivers: [otlp], exporters: [routing] }
    metrics/raw_passthrough:     { receivers: [routing], processors: [gorillas3, batch], ...}
    metrics/ddsketch_path:       { receivers: [routing], processors: [gorillas3, ddsketch, batch], ...}
    metrics/kll_path:            { receivers: [routing], processors: [gorillas3, KLL, batch], ...}
    metrics/hll_path:            { receivers: [routing], processors: [gorillas3, HLL, batch], ...}
    metrics/countsketch_path:    { receivers: [routing], processors: [gorillas3, countsketch, batch], ...}
    metrics/countminsketch_path: { receivers: [routing], processors: [gorillas3, countmin, batch], ...}

Residual half of #350 (out of scope here)

The controller emit uses *processor-suffixed component names (ddsketchprocessor, kllprocessor, ...) and decorates each block with aggregation_id + sketch_kind keys. None of those are accepted by the patched contrib build's component registry (verified via asap-otel components and mapstructure: tags), so the controller-emitted YAML wouldn't load on agents either. That's the residual half of #350 — fix in the controller emitter, separate PR.

Verification

  • asap-otel validate --config asap-otel-agent-b6-asap-single-sketch.yaml → exit 0.
  • Stack boot smoke (docker compose ... up -d controller minio gateway agent-a) → Everything is ready. Begin running and processing data. Each per-family pipeline logs Starting <sketch> processor + Starting gorillas3 processor. No error|invalid|cannot|fatal|panic lines.

Test plan

  • python3 deploy/configs/tests/test_static_placeholder_5sketch_routing.py — 8/8 pass.
    • All 5 sketch processors present at top level.
    • routing in connectors: (not processors:).
    • All 7 named pipelines declared.
    • Each per-sketch pipeline starts with gorillas3 and ends with batch.
    • routing.default_pipelines == [metrics/raw_passthrough].
    • context: metric on every routing table entry.
    • Routing table targets only declared pipelines.
  • asap-otel validate exits 0.
  • Minimal stack agent boot reports Everything is ready.

Refs #46.

🤖 Generated with Claude Code

…e (was single ddsketch)

PR #350's report flagged that OpAMP RemoteConfig push isn't applied by
the agent at runtime — the effective-config preview keeps showing the
static bootstrap YAML loaded from the volume mount, so the running
pipeline is whatever this static file declares.

Previously the placeholder declared a single
[gorillas3, ddsketch, batch] pipeline, so agents only ran ONE sketch
(DDSketch) regardless of the controller plan. KLL / HLL / CountSketch /
CountMinSketch processors never ran.

This PR rewrites the static placeholder to mirror the typed-emit shape
from controller/src/config/stage_config.rs::emit_edge_yaml_5sketch_routing:
all 5 sketch processors loaded at top level, the OTel v0.106 routing
connector under connectors: (NOT processors:), and 7 named pipelines
(entry + raw_passthrough + 5 per-family paths). Each per-sketch
pipeline runs gorillas3 first so the cold-tier write happens on raw
samples before the sketch processor mutates the stream.

The agent boots cleanly with no config-load errors — verified with
`asap-otel validate` (exit 0) and a minimal compose stack run that
reports "Everything is ready. Begin running and processing data."
with all 5 sketch processors listed under their respective pipelines.

Smoke test in deploy/configs/tests/ parses the YAML and pins:
  - All 5 sketch processors present at top level
  - routing in connectors: (not processors:)
  - All named pipelines (entry + raw_passthrough + 5 per-family)
  - Each per-sketch pipeline has gorillas3 first
  - routing.default_pipelines == [metrics/raw_passthrough]
  - context: metric on every routing table entry

Refs #46.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 6041950 into main May 9, 2026
zzylol added a commit that referenced this pull request May 9, 2026
… (#46) (#355)

Root cause: PR #353 spawned six gorillas3 instances (one per
per-family pipeline under the 5-sketch routing topology), each with
its own in-memory windowState. With the static placeholder's 60 s
flush interval and PR #338's fake-exporter cardinality, the agent
buffered the full per-pipeline working set in RAM BEFORE the first
flush ticker fired — peak crossed the 1.5 GiB per-agent memory
limit, the agent was OOM-killed, and zero TSDB blocks ever reached
MinIO. PR #354's flat-slice OOB-tolerance approach compounded the
problem at flush time (O(N_samples) flat slice + interleaved Head
chunk creation across every series) but the agent never lived long
enough for that path to fire.

Result in /tmp/asap-mvp-rerun-bug34/asap/measurements/accuracy.csv:
1713 archive_miss / 0 archive_ok (was 343 archive_ok pre-#354).

Two-part fix:

1) opentelemetry-collector-contrib-patch/processor/gorillas3processor:
   replace PR #354's flat-sort-and-skip path with a series-visit-
   ordered append. Each series' points are still sorted ascending
   locally, but series are now visited in ascending order of each
   series' EARLIEST sample timestamp. That guarantees the very first
   `app.Append(...)` carries the global minimum, anchoring the
   appender's `minValidTime = globalMin - chunkRange/2` so every
   other in-window sample passes the OOB check — without ever
   allocating an O(N_samples) flat slice or interleaving Head
   series creation. Memory footprint at flush is bounded by
   `max(series points, Head per-series state)` instead of total
   sample count. PR #354's defensive ErrOutOfBounds tolerance
   (`gorillas3_tsdb_oob_samples_dropped_total` counter + warn log,
   no rollback) is retained for genuinely late-arriving samples.

2) deploy/configs/asap-otel-agent-b6-asap-single-sketch.yaml:
   reduce gorillas3.window_interval from 60 s → 5 s. With six
   per-pipeline windowStates instead of one, the per-instance
   in-flight buffer is the dominant memory pressure. 5 s caps the
   per-pipeline window peak at ~150 MiB, putting the steady-state
   aggregate inside the agent's 1.5 GiB budget. tsdb_block_duration
   stays at 60 s so the on-S3 layout still matches the Thanos
   store-gateway sync interval; flushes just roll smaller sub-blocks
   that thanos-compact will merge.

Verification (90 s soak diagnose, repeated):
  before: mc ls myminio/asap-gorilla-tsdb/ → empty
          curl -s http://localhost:19092/api/v1/labels →
            {"data":["__name__"]} (no metric labels surfaced)
  after:  mc ls myminio/asap-gorilla-tsdb/ → 27 blocks across all
          six pipelines (countsketch_path, countminsketch_path,
          ddsketch_path, hll_path, kll_path, raw_passthrough)
          curl -s http://localhost:19092/api/v1/label/__name__/values →
            ["endpoint_request_freq","http_freshness_probe_archive",
             "http_freshness_probe_raw","http_freshness_probe_warm",
             "http_requests_total","http_requests_total_latency_ms",
             "request_size_bytes","top_endpoint_qps",
             "unique_users_per_min"]
          all 5 sketched metrics + raw + 3 freshness probes.

Coverage:
  TestTSDBBlockBuilder_HighCardinalityMemoryBound — 200-series /
  60-sample window with non-overlapping per-series time slices, so
  the global minimum lives on a different series from the global
  maximum. Asserts 0 OOB drops + every sample lands in the block.
  Pre-this-PR's algorithm would still have passed correctness but
  allocated the O(N) flat slice; this test pins the visit-order
  invariant going forward.

Existing TSDB OOB regressions (RotatingCardinalityNoOOB,
WideTimestampSpanNoOOB, FlushTSDB_OOBDoesNotCrashProcessor) all
still pass with the new visit-order path — same OOB-tolerance
contract, different memory footprint.

Closes #46.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the fix/static-placeholder-5sketch-routing branch May 9, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant