fix: agent pipeline includes memory_limiter (was OOM-killed at ~3 min) - #356
Merged
Merged
Conversation
PR #355 lowered gorillas3 `window_interval` to 5 s to keep the in-memory windowState within the agent's 1.5 GiB cgroup, but under sustained load six per-family pipelines still overshoot the ceiling and the agent gets OOM-killed (exit 137) after roughly three minutes — the demo confirmed both `agent-a-1` and `agent-b-1` Exited (137) two minutes into Phase 2, and the backend's `MEMORY_DIAG` reported `group_states_len=0` for every worker because nothing ever ingested. The OpenTelemetry Collector ships a `memory_limiter` processor that refuses incoming data when RSS crosses a soft threshold, applying backpressure to upstream senders rather than crashing. The gateway already uses it (see compose log: `Memory limiter configured limit_mib=1536 spike_limit_mib=256 check_interval=1`); the agent didn't. This change inserts `memory_limiter` AS THE FIRST processor in every per-sketch pipeline (and the default `metrics/raw_passthrough`): metrics/ddsketch_path: processors: [memory_limiter, gorillas3, ddsketch, batch] Same for `kll_path`, `hll_path`, `countsketch_path`, `countminsketch_path`, and `raw_passthrough`. The entry pipeline (which has `processors: []` because routing fan-out is the connector's job) is unchanged. Threshold rationale: agent cgroup is 1536 MiB, so we pin `limit_mib: 1280` (≈ 80 % of cgroup) and `spike_limit_mib: 256` — leaving 256 MiB headroom under the cgroup ceiling for short bursts. Mirrors the gateway shape but scaled to the agent's smaller cgroup. Updates both the static placeholder (`deploy/configs/asap-otel-agent-b6-asap-single-sketch.yaml`) and the controller's typed emit (`emit_edge_yaml_5sketch_routing` in `controller/src/config/stage_config.rs`) so once the OpAMP-push gap flagged by PR #350 lands, the runtime swap stays in shape. Tests: - New `mvp46_per_sketch_pipelines_have_memory_limiter_first` Rust test asserts every per-family pipeline + raw_passthrough lists `memory_limiter` first. - Python smoke `test_static_placeholder_5sketch_routing.py` gains `test_memory_limiter_processor_block_present` and updated `test_each_per_sketch_pipeline_has_memory_limiter_first` / `test_raw_passthrough_pipeline_shape` covering the new shape. Refs #46, follow-up to #355. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #355 lowered gorillas3
window_intervalto 5 s to keep thein-memory
windowStatewithin the agent's 1.5 GiB cgroup, but as AgentX flagged in the post-merge review, "a follow-up needs a
memory_limiterprocessor (or backpressure inside gorillas3) forindefinite agent uptime". The MVP demo confirmed the gap:
agent-a-1andagent-b-1Exited (137)≈ 2 min into Phase 2(after the ~150 s settle + warm-up).
stayed Up & healthy.
MEMORY_DIAGreportedgroup_states_len=0for every worker— i.e. the backend never ingested anything because the agent died
before the first sketch flush reached the gateway.
status=error: No result for query.This change inserts the OpenTelemetry Collector's upstream
memory_limiterprocessor as the FIRST processor in every per-sketchpipeline (and the default
metrics/raw_passthrough). The processorapplies backpressure (refuses incoming data + forces GC) when RSS
crosses a soft threshold instead of letting the cgroup OOM-killer
terminate the agent. The gateway already uses the same processor (see
the compose log line
Memory limiter configured limit_mib=1536 spike_limit_mib=256 check_interval=1); the agent didn't, so thehalf-fix from PR #355 had nothing to throttle behind.
Pipeline shape (placeholder + controller emit, both updated):
Same for
kll_path,hll_path,countsketch_path,countminsketch_path, andraw_passthrough. The entrymetrics:pipeline (
processors: [], routing fan-out is the connector's job)is unchanged.
Threshold:
Mirrors the gateway shape but scaled to the agent's smaller cgroup.
Files touched
deploy/configs/asap-otel-agent-b6-asap-single-sketch.yaml— staticplaceholder (current default
AGENT_CONFIG_A/AGENT_CONFIG_B).controller/src/config/stage_config.rs—emit_edge_yaml_5sketch_routingaddsmemory_limiterto theprocessor map and prepends it to
make_sketch_pipeline/raw_passthroughso the controller's typed emit stays in shape withthe placeholder for when the OpAMP-push gap from fix: archive tier covers all 5 sketched metrics (HLL/CS/CMS/KLL/DDSketch all in Thanos) #350 lands.
deploy/configs/tests/test_static_placeholder_5sketch_routing.py—new
test_memory_limiter_processor_block_presentplus updatedtest_each_per_sketch_pipeline_has_memory_limiter_first(renamedfrom the gorillas3-first test) and updated
test_raw_passthrough_pipeline_shape.mvp46_per_sketch_pipelines_have_memory_limiter_firstincontroller/src/config/stage_config.rs.Smoke evidence (5-min soak)
The 5-min soak reproduces the original OOM, but with one critical
difference: the limiter is now actively pushing back. Compose log:
The limiter is engaged within 15 s of agent startup and stays in
"Refusing data" mode for the rest of the run.
Honest gap (must read before merging)
The spec'd memory_limiter shape does not by itself eliminate the
exit-137 loop in this workload. The 5-min soak shows agents still
Exited (137)at ≈ 30 s, worse than the pre-fix ~3-min threshold.Two compounding causes the limiter alone can't address:
Idle baseline exceeds limit_mib. The Go runtime's MemStats
reports
cur_mem_mib=2174within 15 s ofEverything is ready—already above the spec'd 1280 limit_mib. This is the six
independent in-memory
windowStatebuffers (one per per-familypipeline) being populated by 5 producers × 500 user_ids × ~30 K
events / s before the first 5 s flush ticker fires. The limiter
correctly enters "Refusing data" mode but new data is not the
problem; already-buffered windowState is.
memory_limiter cannot evict already-buffered windowState. The
processor's contract is "refuse incoming batches"; it has no hook
to free memory held by downstream processors. The gorillas3
windowStatebuffer keeps accreting until the 5 s flush tickerreaps it, and the cgroup OOM-killer fires before the next reap.
The follow-up Agent X actually flagged was "
memory_limiterprocessor or backpressure inside gorillas3". This PR delivers the
former; the latter (gorillas3-side window cap, hard byte budget,
backpressure-on-buffer-full) is the structural fix and needs its own
PR. Possible direction: gorillas3 itself reports its current buffer
bytes via the existing self-monitoring path and refuses new samples
when buffer > soft cap, OR runs an early-flush when buffer crosses a
threshold (rather than waiting for the 5 s ticker).
Refs #46, follow-up to #355.
Test plan
python3 deploy/configs/tests/test_static_placeholder_5sketch_routing.py— 9/9 passcd controller && cargo test stage_config— 54/54 pass (including newmvp46_per_sketch_pipelines_have_memory_limiter_first)