Conversation
…o 1 shared pipeline (R1) The b6 5-sketch routing config ran a gorillas3 (cold-tier TSDB archive) processor on every routed pipeline (raw_passthrough + 5 sketch paths) = 6 instances, each buffering windowState/TSDB blocks — the dominant agent-memory cost. Add a single `metrics/cold_archive` pipeline holding the only gorillas3, and use the routing connector's multi-target support to fan every metric to BOTH its warm-tier path AND cold_archive (default_pipelines includes cold_archive too). Cold coverage is preserved for all 6 metrics — the accuracy harness's `X-ASAP-Engine: thanos_archive` ground truth reads the same TSDB-blocks-on-MinIO this one gorillas3 writes — collapsing 6 gorillas3 instances to 1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Superseded by the mvp-singlenode deletion (this config lives in the deleted tree). Closing; the single-shared-cold-archive idea carries into the asapedge mvp-multinode rewrite. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
R1: single shared
gorillas3cold-archive (6 → 1)What. The mvp-singlenode b6 agent config ran a
gorillas3cold-tier (TSDB→MinIO) archive processor on all 6 routed pipelines (raw_passthrough+ the 5 sketch paths). This collapses them to one sharedmetrics/cold_archivepipeline, fed via the routing connector's multi-target fan-out so every metric still reaches the cold archive.Why. The 6
gorillas3instances — each buffering windowState/TSDB blocks — are the dominant agent-memory cost (drove agent RSS to ~100% of cap even at low cardinality).Cold coverage preserved. Each routing entry now targets
[<sketch>_path, metrics/cold_archive], anddefault_pipelinesincludescold_archive, so the singlegorillas3archives the raw samples for all 6 metrics.accuracy_reduce.py'sX-ASAP-Engine: thanos_archiveground truth reads the same TSDB-blocks-on-MinIO this instance writes → noarchive_miss. Thegorillas3settings (window_interval, block_format, tsdb_block_duration, bucket/endpoint) are carried over verbatim.Config-only change. Before merge, confirm on the mvp-singlenode stack:
accuracy_reducereturns noarchive_missfor any of the 6 metrics (cold coverage intact).routingconnectoraccepts multi-targetpipelines:lists at boot (no boot error).Scope note: this branch is R1 only — the heap-wiring (
emit_heap/item_label) andmemory_limiter/OOM changes that co-live in this file in the working tree are intentionally excluded here; they belong to their own PRs.🤖 Generated with Claude Code