fix(agent): split chained sketch pipelines so windowed processors don't drop each other's output - #209
Closed
zzylol wants to merge 1 commit into
Closed
fix(agent): split chained sketch pipelines so windowed processors don't drop each other's output#209zzylol wants to merge 1 commit into
zzylol wants to merge 1 commit into
Conversation
…'t drop each other's output
## Bug
When an agent config chains two windowed sketch processors —
e.g. b3-delta's `[ddsketch, HLL, batch]` — the agent silently
emits nothing. Symptom from Prometheus self-telemetry: ddsketch
processor reports `output_metric_points_total > 0`, but the
agent's `otelcol_exporter_sent_metric_points_total` never
appears (and the gateway shows 0 receiver_accepted on its
gRPC transport).
## Root cause
Both `ddsketchprocessor` and `hllprocessor` (and the other
windowed sketch processors) implement `ConsumeMetrics` like:
```go
case ModeWindow:
p.accumulateIntoWindow(md)
return nil // <-- inputs are NOT forwarded
```
In window mode the processor is its OWN producer — it accumulates
inputs into its windowed state and emits on its own timer via
`emitWindowAndReset` → `nextConsumer.ConsumeMetrics(...)`. It does
not forward inputs synchronously.
So when the pipeline is `[otlp -> ddsketch, HLL, batch -> otlp/gateway]`:
1. otlp receiver delivers raw fake-exporter metrics → ddsketch.ConsumeMetrics.
2. ddsketch accumulates, returns nil. HLL **never sees the raw inputs**.
3. ddsketch's window timer fires → emits 1000 typed-DDSketch
metrics → calls HLL.ConsumeMetrics(typed DDSketch payload).
4. HLL.accumulateIntoWindow has only `case MetricTypeGauge:` and
`case MetricTypeHLLSketch:`. The typed DDSketch metrics fall
through and HLL **silently drops them** (no default case, no
pass-through to nextConsumer).
5. batch processor sees nothing.
(Confirmed live: ddsketch in/out=18000/2000, HLL in/out=2000/0,
batch processor reports only `metadata_cardinality=1` — it's been
instantiated but never received items.)
## Fix
Split the chained pipeline into per-sketch pipelines. The OTLP
receiver fans out to every pipeline that references it, so
`metrics/ddsketch` and `metrics/hll` each see every incoming
batch independently and process it without stepping on each
other:
```yaml
service:
pipelines:
metrics/ddsketch:
receivers: [otlp]
processors: [ddsketch, batch]
exporters: [otlp/gateway]
metrics/hll:
receivers: [otlp]
processors: [HLL, batch]
exporters: [otlp/gateway]
```
Applied to all three agent configs that chained two windowed
sketch processors:
- `sketchcol-agent-b2-full.yaml` (paper baseline B2 — full sketch)
- `sketchcol-agent-b3-delta.yaml` (paper baseline B3 — delta)
- `sketchcol-agent-b4-tunable.yaml` (paper baseline B4 — variable window)
Each config now has a comment block at the pipeline section
explaining why chaining doesn't work, so the next person to add
a sketch processor doesn't recreate the bug.
## Verification
End-to-end with `b3-delta` after fix:
| Stage | Before | After |
|---|---|---|
| ddsketch processor output | 2000 | 2000 |
| HLL processor output | 0 | (independent pipeline, n/a) |
| Agent exporter sent | 0 / metric absent | 2000 |
| Gateway receiver accepted (gRPC) | 0 / metric absent | 2000 |
| Gateway exporter sent to backend | 0 | 1000 (first batch) |
| Backend OTLP ingest | 0 routed | 1000 routed, 0 decode-failed |
Backend log on first window:
```
OTLP modified-proto Ddsketch received
(metric=http_requests_total_latency_ms_quantile, dps=1000); decoder is PR B
OTLP modified-proto sketch ingest:
1000 routed, 0 decode-failed (fallback), 0 unconfigured
```
The PromQL query path matches inference patterns and looks up
the right `agg_id` in the precompute store; remaining gaps
(empty store on lookup, decode-fail on second-window deltas)
are downstream of the agent pipeline fix and out of scope here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 1, 2026
So a single pipeline like `[ddsketch, hll, batch]` actually works.
## Bug
ddsketch / kll / hll processors in `mode: window` returned `nil`
from `ConsumeMetrics` — they accumulated inputs into their
windowed state and dropped the input batch on the floor without
forwarding it to the next consumer:
```go
case ModeWindow:
p.accumulateIntoWindow(md)
return nil // <-- inputs go nowhere
```
In a chained pipeline `[ddsketch, hll, batch]`, that means:
1. otlp delivers raw `http_requests_total` → ddsketch.ConsumeMetrics.
ddsketch accumulates, returns nil. **HLL never sees the raw input.**
2. ddsketch's window timer fires → it emits typed-DDSketch metrics
to its nextConsumer (HLL.ConsumeMetrics).
3. HLL.accumulateIntoWindow has only `case MetricTypeGauge` and
`case MetricTypeHLLSketch` in its switch — typed DDSketch
metrics fall through and HLL **silently drops them**.
4. batch sees nothing.
Net: with multiple windowed sketch processors in one pipeline,
nothing reaches the exporter.
countminsketchprocessor / countsketchprocessor already do the
right thing — their ModeWindow paths return `md` (forward
inputs) by default, with an opt-in `drop_original` flag. The
other three were the outliers.
## Fix
Match the CMS/CountSketch pattern: forward `md` unchanged after
accumulating. Same one-liner in all three processors:
```go
case ModeWindow:
p.accumulateIntoWindow(md)
return p.nextConsumer.ConsumeMetrics(ctx, md)
```
Now the chain works as expected:
- `ddsketch.ConsumeMetrics(raw)`: accumulate, forward raw to HLL.
- `HLL.ConsumeMetrics(raw)`: accumulate, forward raw to batch.
- `batch.ConsumeMetrics(raw)`: batch & ship.
- ddsketch tick fires → typed-DDSketch → HLL (no match, forward) → batch.
- HLL tick fires → typed-HLL → batch.
So both DDSketch and HLL sketches reach the exporter in addition
to the original raw inputs. Users who don't want the raw inputs
shipped can add a `filter` processor at the end of the pipeline.
## Verification
End-to-end with `b3-delta` agent
(`processors: [ddsketch, HLL, batch]`, `delta_transmission: true`,
`window_duration: 60s`):
```
W1 (16:21:09):
Ddsketch received: dps=1000
Hllsketch received: dps=1000 ← HLL now reaches backend
ingest: 1000 routed, 0 decode-failed, 1000 unconfigured
W2 (16:22:09):
Ddsketch received: dps=1000 (delta)
ingest: 1000 routed, 0 decode-failed, 1000 unconfigured
```
Pre-fix: HLL never reached backend; only ddsketch's outputs were
emitted (and only after #210's DDSketch delta wire fix).
## Supersedes #209
PR #209 worked around this bug at the config level by splitting
`[ddsketch, HLL, batch]` into two parallel pipelines
(`metrics/ddsketch` and `metrics/hll`). That goes against the
goal of one pipeline that supports any controller-chosen
combination of sketches. With this fix, the original chained
config works directly. #209 can be closed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
Contributor
Author
|
Superseded by #211 — the proper fix is at the processor level (forward inputs through windowed processors), so the original chained |
zzylol
added a commit
that referenced
this pull request
May 1, 2026
) So a single pipeline like `[ddsketch, hll, batch]` actually works. ## Bug ddsketch / kll / hll processors in `mode: window` returned `nil` from `ConsumeMetrics` — they accumulated inputs into their windowed state and dropped the input batch on the floor without forwarding it to the next consumer: ```go case ModeWindow: p.accumulateIntoWindow(md) return nil // <-- inputs go nowhere ``` In a chained pipeline `[ddsketch, hll, batch]`, that means: 1. otlp delivers raw `http_requests_total` → ddsketch.ConsumeMetrics. ddsketch accumulates, returns nil. **HLL never sees the raw input.** 2. ddsketch's window timer fires → it emits typed-DDSketch metrics to its nextConsumer (HLL.ConsumeMetrics). 3. HLL.accumulateIntoWindow has only `case MetricTypeGauge` and `case MetricTypeHLLSketch` in its switch — typed DDSketch metrics fall through and HLL **silently drops them**. 4. batch sees nothing. Net: with multiple windowed sketch processors in one pipeline, nothing reaches the exporter. countminsketchprocessor / countsketchprocessor already do the right thing — their ModeWindow paths return `md` (forward inputs) by default, with an opt-in `drop_original` flag. The other three were the outliers. ## Fix Match the CMS/CountSketch pattern: forward `md` unchanged after accumulating. Same one-liner in all three processors: ```go case ModeWindow: p.accumulateIntoWindow(md) return p.nextConsumer.ConsumeMetrics(ctx, md) ``` Now the chain works as expected: - `ddsketch.ConsumeMetrics(raw)`: accumulate, forward raw to HLL. - `HLL.ConsumeMetrics(raw)`: accumulate, forward raw to batch. - `batch.ConsumeMetrics(raw)`: batch & ship. - ddsketch tick fires → typed-DDSketch → HLL (no match, forward) → batch. - HLL tick fires → typed-HLL → batch. So both DDSketch and HLL sketches reach the exporter in addition to the original raw inputs. Users who don't want the raw inputs shipped can add a `filter` processor at the end of the pipeline. ## Verification End-to-end with `b3-delta` agent (`processors: [ddsketch, HLL, batch]`, `delta_transmission: true`, `window_duration: 60s`): ``` W1 (16:21:09): Ddsketch received: dps=1000 Hllsketch received: dps=1000 ← HLL now reaches backend ingest: 1000 routed, 0 decode-failed, 1000 unconfigured W2 (16:22:09): Ddsketch received: dps=1000 (delta) ingest: 1000 routed, 0 decode-failed, 1000 unconfigured ``` Pre-fix: HLL never reached backend; only ddsketch's outputs were emitted (and only after #210's DDSketch delta wire fix). ## Supersedes #209 PR #209 worked around this bug at the config level by splitting `[ddsketch, HLL, batch]` into two parallel pipelines (`metrics/ddsketch` and `metrics/hll`). That goes against the goal of one pipeline that supports any controller-chosen combination of sketches. With this fix, the original chained config works directly. #209 can be closed. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an agent config chains two windowed sketch processors — e.g. b3-delta's `[ddsketch, HLL, batch]` — the agent silently emits nothing. Symptom: ddsketch's `output_metric_points_total` grows but `exporter_sent_metric_points_total` never appears. This was the actual blocker for warm-tier sketch verification.
Root cause
Both `ddsketchprocessor` and `hllprocessor` (and the other windowed sketch processors) handle window mode like:
```go
case ModeWindow:
p.accumulateIntoWindow(md)
return nil // <-- inputs are NOT forwarded
```
In window mode the processor is its OWN producer — it accumulates inputs into windowed state and emits on its own timer. It does not forward inputs synchronously.
So the chain `[otlp -> ddsketch, HLL, batch -> otlp/gateway]` actually behaves like this:
Live confirmation before the fix: ddsketch input/output = 18000/2000, HLL input/output = 2000/0, batch processor reports only `metadata_cardinality=1` — instantiated but never received items.
Fix
Split into per-sketch pipelines. The OTLP receiver fans out to every pipeline that references it, so each sketch processor sees every incoming batch independently:
```yaml
service:
pipelines:
metrics/ddsketch:
receivers: [otlp]
processors: [ddsketch, batch]
exporters: [otlp/gateway]
metrics/hll:
receivers: [otlp]
processors: [HLL, batch]
exporters: [otlp/gateway]
```
Applied to all three configs that chained windowed sketch processors:
Each comes with a comment block explaining the gotcha so the next person adding a sketch processor doesn't recreate the bug.
Verification
End-to-end with `b3-delta` after the fix:
Backend log on first agent window flush:
```
OTLP modified-proto Ddsketch received
(metric=http_requests_total_latency_ms_quantile, dps=1000); decoder is PR B
OTLP modified-proto sketch ingest:
1000 routed, 0 decode-failed (fallback), 0 unconfigured
```
PromQL query path matches inference patterns and looks up the right `agg_id` in the precompute store. Remaining no-result on `quantile_over_time(...)` is downstream and out of scope here — see follow-ups below.
Two adjacent issues found during e2e (separate from this PR)
These are independent of this PR — agent → gateway → backend OTLP ingest works.
Test plan
🤖 Generated with Claude Code