Skip to content

fix(agent): split chained sketch pipelines so windowed processors don't drop each other's output - #209

Closed
zzylol wants to merge 1 commit into
mainfrom
fix/agent-windowed-sketch-pipeline-chain
Closed

zzylol wants to merge 1 commit into
mainfrom
fix/agent-windowed-sketch-pipeline-chain

Conversation

@zzylol

@zzylol zzylol commented May 1, 2026

Copy link
Copy Markdown
Contributor

Summary

When an agent config chains two windowed sketch processors — e.g. b3-delta's `[ddsketch, HLL, batch]` — the agent silently emits nothing. Symptom: ddsketch's `output_metric_points_total` grows but `exporter_sent_metric_points_total` never appears. This was the actual blocker for warm-tier sketch verification.

Root cause

Both `ddsketchprocessor` and `hllprocessor` (and the other windowed sketch processors) handle window mode like:

```go
case ModeWindow:
p.accumulateIntoWindow(md)
return nil // <-- inputs are NOT forwarded
```

In window mode the processor is its OWN producer — it accumulates inputs into windowed state and emits on its own timer. It does not forward inputs synchronously.

So the chain `[otlp -> ddsketch, HLL, batch -> otlp/gateway]` actually behaves like this:

  1. otlp delivers raw fake-exporter metrics → `ddsketch.ConsumeMetrics`.
  2. ddsketch accumulates, returns nil. HLL never sees the raw inputs.
  3. ddsketch's window timer fires → emits 1000 typed-DDSketch metrics → calls `HLL.ConsumeMetrics(typed DDSketch payload)`.
  4. `HLL.accumulateIntoWindow`'s switch only handles `MetricTypeGauge` and `MetricTypeHLLSketch`. Typed DDSketch metrics fall through and HLL silently drops them (no default case, no pass-through to nextConsumer).
  5. batch processor sees nothing.

Live confirmation before the fix: ddsketch input/output = 18000/2000, HLL input/output = 2000/0, batch processor reports only `metadata_cardinality=1` — instantiated but never received items.

Fix

Split into per-sketch pipelines. The OTLP receiver fans out to every pipeline that references it, so each sketch processor sees every incoming batch independently:

```yaml
service:
pipelines:
metrics/ddsketch:
receivers: [otlp]
processors: [ddsketch, batch]
exporters: [otlp/gateway]
metrics/hll:
receivers: [otlp]
processors: [HLL, batch]
exporters: [otlp/gateway]
```

Applied to all three configs that chained windowed sketch processors:

  • `sketchcol-agent-b2-full.yaml` (paper baseline B2)
  • `sketchcol-agent-b3-delta.yaml` (paper baseline B3)
  • `sketchcol-agent-b4-tunable.yaml` (paper baseline B4)

Each comes with a comment block explaining the gotcha so the next person adding a sketch processor doesn't recreate the bug.

Verification

End-to-end with `b3-delta` after the fix:

Stage Before After
ddsketch processor output 2000 2000
HLL processor output 0 (in same pipeline) n/a (independent pipeline)
Agent exporter sent 0 / metric absent 2000
Gateway receiver accepted (gRPC) 0 / metric absent 2000
Gateway exporter sent to backend 0 1000 (first batch)
Backend OTLP ingest 0 routed 1000 routed, 0 decode-failed

Backend log on first agent window flush:
```
OTLP modified-proto Ddsketch received
(metric=http_requests_total_latency_ms_quantile, dps=1000); decoder is PR B
OTLP modified-proto sketch ingest:
1000 routed, 0 decode-failed (fallback), 0 unconfigured
```

PromQL query path matches inference patterns and looks up the right `agg_id` in the precompute store. Remaining no-result on `quantile_over_time(...)` is downstream and out of scope here — see follow-ups below.

Two adjacent issues found during e2e (separate from this PR)

  1. Delta-encoded sketches fail to decode at backend. Second window's flush logs `1000 routed, 1000 decode-failed (fallback)`. The agent has `delta_transmission: true` and emits proto_full only for the first window, then proto_delta for subsequent windows. The backend's modified-OTLP DDSketch decoder doesn't yet handle `DDSketchEncodingProtoDelta`. (See the `ddsketchprocessor.decodeDDSketchDataPoint` stub which already documents this.)
  2. Precompute store empty on query. Even after first-window data routes successfully (1000 routed, 0 decode-failed), `Query execution failed: No precomputed outputs found for metric`. Possibly a windowing/timestamp alignment issue between the agent's 60s window emit and the backend's 30s tumbling window in `backend-streaming.yaml`. Worth a focused look at the precompute engine's window flush + query lookup timestamps.

These are independent of this PR — agent → gateway → backend OTLP ingest works.

Test plan

  • `docker compose ... up -d` with the new b3-delta config — agent's exporter starts sending after first 60s window.
  • Gateway gRPC receiver accepts the typed sketch payload (transport=grpc, accepted_metric_points_total grows).
  • Backend OTLP receiver decodes and routes typed DDSketch payloads to precompute engine.
  • Once delta-decode + store-window issues are addressed, full PromQL flow returns expected envelope.

🤖 Generated with Claude Code

…'t drop each other's output

## Bug

When an agent config chains two windowed sketch processors —
e.g. b3-delta's `[ddsketch, HLL, batch]` — the agent silently
emits nothing. Symptom from Prometheus self-telemetry: ddsketch
processor reports `output_metric_points_total > 0`, but the
agent's `otelcol_exporter_sent_metric_points_total` never
appears (and the gateway shows 0 receiver_accepted on its
gRPC transport).

## Root cause

Both `ddsketchprocessor` and `hllprocessor` (and the other
windowed sketch processors) implement `ConsumeMetrics` like:

```go
case ModeWindow:
    p.accumulateIntoWindow(md)
    return nil          // <-- inputs are NOT forwarded
```

In window mode the processor is its OWN producer — it accumulates
inputs into its windowed state and emits on its own timer via
`emitWindowAndReset` → `nextConsumer.ConsumeMetrics(...)`. It does
not forward inputs synchronously.

So when the pipeline is `[otlp -> ddsketch, HLL, batch -> otlp/gateway]`:

1. otlp receiver delivers raw fake-exporter metrics → ddsketch.ConsumeMetrics.
2. ddsketch accumulates, returns nil. HLL **never sees the raw inputs**.
3. ddsketch's window timer fires → emits 1000 typed-DDSketch
   metrics → calls HLL.ConsumeMetrics(typed DDSketch payload).
4. HLL.accumulateIntoWindow has only `case MetricTypeGauge:` and
   `case MetricTypeHLLSketch:`. The typed DDSketch metrics fall
   through and HLL **silently drops them** (no default case, no
   pass-through to nextConsumer).
5. batch processor sees nothing.

(Confirmed live: ddsketch in/out=18000/2000, HLL in/out=2000/0,
batch processor reports only `metadata_cardinality=1` — it's been
instantiated but never received items.)

## Fix

Split the chained pipeline into per-sketch pipelines. The OTLP
receiver fans out to every pipeline that references it, so
`metrics/ddsketch` and `metrics/hll` each see every incoming
batch independently and process it without stepping on each
other:

```yaml
service:
  pipelines:
    metrics/ddsketch:
      receivers: [otlp]
      processors: [ddsketch, batch]
      exporters: [otlp/gateway]
    metrics/hll:
      receivers: [otlp]
      processors: [HLL, batch]
      exporters: [otlp/gateway]
```

Applied to all three agent configs that chained two windowed
sketch processors:
- `sketchcol-agent-b2-full.yaml` (paper baseline B2 — full sketch)
- `sketchcol-agent-b3-delta.yaml` (paper baseline B3 — delta)
- `sketchcol-agent-b4-tunable.yaml` (paper baseline B4 — variable window)

Each config now has a comment block at the pipeline section
explaining why chaining doesn't work, so the next person to add
a sketch processor doesn't recreate the bug.

## Verification

End-to-end with `b3-delta` after fix:

| Stage | Before | After |
|---|---|---|
| ddsketch processor output | 2000 | 2000 |
| HLL processor output | 0 | (independent pipeline, n/a) |
| Agent exporter sent | 0 / metric absent | 2000 |
| Gateway receiver accepted (gRPC) | 0 / metric absent | 2000 |
| Gateway exporter sent to backend | 0 | 1000 (first batch) |
| Backend OTLP ingest | 0 routed | 1000 routed, 0 decode-failed |

Backend log on first window:
```
OTLP modified-proto Ddsketch received
  (metric=http_requests_total_latency_ms_quantile, dps=1000); decoder is PR B
OTLP modified-proto sketch ingest:
  1000 routed, 0 decode-failed (fallback), 0 unconfigured
```

The PromQL query path matches inference patterns and looks up
the right `agg_id` in the precompute store; remaining gaps
(empty store on lookup, decode-fail on second-window deltas)
are downstream of the agent pipeline fix and out of scope here.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 1, 2026
So a single pipeline like `[ddsketch, hll, batch]` actually works.

## Bug

ddsketch / kll / hll processors in `mode: window` returned `nil`
from `ConsumeMetrics` — they accumulated inputs into their
windowed state and dropped the input batch on the floor without
forwarding it to the next consumer:

```go
case ModeWindow:
    p.accumulateIntoWindow(md)
    return nil   // <-- inputs go nowhere
```

In a chained pipeline `[ddsketch, hll, batch]`, that means:

1. otlp delivers raw `http_requests_total` → ddsketch.ConsumeMetrics.
   ddsketch accumulates, returns nil. **HLL never sees the raw input.**
2. ddsketch's window timer fires → it emits typed-DDSketch metrics
   to its nextConsumer (HLL.ConsumeMetrics).
3. HLL.accumulateIntoWindow has only `case MetricTypeGauge` and
   `case MetricTypeHLLSketch` in its switch — typed DDSketch
   metrics fall through and HLL **silently drops them**.
4. batch sees nothing.

Net: with multiple windowed sketch processors in one pipeline,
nothing reaches the exporter.

countminsketchprocessor / countsketchprocessor already do the
right thing — their ModeWindow paths return `md` (forward
inputs) by default, with an opt-in `drop_original` flag. The
other three were the outliers.

## Fix

Match the CMS/CountSketch pattern: forward `md` unchanged after
accumulating. Same one-liner in all three processors:

```go
case ModeWindow:
    p.accumulateIntoWindow(md)
    return p.nextConsumer.ConsumeMetrics(ctx, md)
```

Now the chain works as expected:

- `ddsketch.ConsumeMetrics(raw)`: accumulate, forward raw to HLL.
- `HLL.ConsumeMetrics(raw)`: accumulate, forward raw to batch.
- `batch.ConsumeMetrics(raw)`: batch & ship.
- ddsketch tick fires → typed-DDSketch → HLL (no match, forward) → batch.
- HLL tick fires → typed-HLL → batch.

So both DDSketch and HLL sketches reach the exporter in addition
to the original raw inputs. Users who don't want the raw inputs
shipped can add a `filter` processor at the end of the pipeline.

## Verification

End-to-end with `b3-delta` agent
(`processors: [ddsketch, HLL, batch]`, `delta_transmission: true`,
`window_duration: 60s`):

```
W1 (16:21:09):
  Ddsketch received: dps=1000
  Hllsketch received: dps=1000   ← HLL now reaches backend
  ingest: 1000 routed, 0 decode-failed, 1000 unconfigured
W2 (16:22:09):
  Ddsketch received: dps=1000   (delta)
  ingest: 1000 routed, 0 decode-failed, 1000 unconfigured
```

Pre-fix: HLL never reached backend; only ddsketch's outputs were
emitted (and only after #210's DDSketch delta wire fix).

## Supersedes #209

PR #209 worked around this bug at the config level by splitting
`[ddsketch, HLL, batch]` into two parallel pipelines
(`metrics/ddsketch` and `metrics/hll`). That goes against the
goal of one pipeline that supports any controller-chosen
combination of sketches. With this fix, the original chained
config works directly. #209 can be closed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol

zzylol commented May 1, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #211 — the proper fix is at the processor level (forward inputs through windowed processors), so the original chained [ddsketch, HLL, batch] config works directly without needing per-pipeline split. The user's design goal is one pipeline that supports any controller-chosen combination of sketches, which #211 enables.

@zzylol zzylol closed this May 1, 2026
zzylol added a commit that referenced this pull request May 1, 2026
)

So a single pipeline like `[ddsketch, hll, batch]` actually works.

## Bug

ddsketch / kll / hll processors in `mode: window` returned `nil`
from `ConsumeMetrics` — they accumulated inputs into their
windowed state and dropped the input batch on the floor without
forwarding it to the next consumer:

```go
case ModeWindow:
    p.accumulateIntoWindow(md)
    return nil   // <-- inputs go nowhere
```

In a chained pipeline `[ddsketch, hll, batch]`, that means:

1. otlp delivers raw `http_requests_total` → ddsketch.ConsumeMetrics.
   ddsketch accumulates, returns nil. **HLL never sees the raw input.**
2. ddsketch's window timer fires → it emits typed-DDSketch metrics
   to its nextConsumer (HLL.ConsumeMetrics).
3. HLL.accumulateIntoWindow has only `case MetricTypeGauge` and
   `case MetricTypeHLLSketch` in its switch — typed DDSketch
   metrics fall through and HLL **silently drops them**.
4. batch sees nothing.

Net: with multiple windowed sketch processors in one pipeline,
nothing reaches the exporter.

countminsketchprocessor / countsketchprocessor already do the
right thing — their ModeWindow paths return `md` (forward
inputs) by default, with an opt-in `drop_original` flag. The
other three were the outliers.

## Fix

Match the CMS/CountSketch pattern: forward `md` unchanged after
accumulating. Same one-liner in all three processors:

```go
case ModeWindow:
    p.accumulateIntoWindow(md)
    return p.nextConsumer.ConsumeMetrics(ctx, md)
```

Now the chain works as expected:

- `ddsketch.ConsumeMetrics(raw)`: accumulate, forward raw to HLL.
- `HLL.ConsumeMetrics(raw)`: accumulate, forward raw to batch.
- `batch.ConsumeMetrics(raw)`: batch & ship.
- ddsketch tick fires → typed-DDSketch → HLL (no match, forward) → batch.
- HLL tick fires → typed-HLL → batch.

So both DDSketch and HLL sketches reach the exporter in addition
to the original raw inputs. Users who don't want the raw inputs
shipped can add a `filter` processor at the end of the pipeline.

## Verification

End-to-end with `b3-delta` agent
(`processors: [ddsketch, HLL, batch]`, `delta_transmission: true`,
`window_duration: 60s`):

```
W1 (16:21:09):
  Ddsketch received: dps=1000
  Hllsketch received: dps=1000   ← HLL now reaches backend
  ingest: 1000 routed, 0 decode-failed, 1000 unconfigured
W2 (16:22:09):
  Ddsketch received: dps=1000   (delta)
  ingest: 1000 routed, 0 decode-failed, 1000 unconfigured
```

Pre-fix: HLL never reached backend; only ddsketch's outputs were
emitted (and only after #210's DDSketch delta wire fix).

## Supersedes #209

PR #209 worked around this bug at the config level by splitting
`[ddsketch, HLL, batch]` into two parallel pipelines
(`metrics/ddsketch` and `metrics/hll`). That goes against the
goal of one pipeline that supports any controller-chosen
combination of sketches. With this fix, the original chained
config works directly. #209 can be closed.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the fix/agent-windowed-sketch-pipeline-chain branch May 9, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant