Skip to content

fix(gateway): unbreak boot post-#206 + simplify merge config - #207

Merged
zzylol merged 1 commit into
mainfrom
fix/gateway-boot-and-merge-config
May 1, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/gateway-boot-and-merge-config

Conversation

@zzylol

@zzylol zzylol commented May 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Caught when running the e2e for the first time after #206 merged. Two boot bugs in the gateway configs land here, plus a scope-correction on the "DD/KLL/HLL merge processors follow-up" I named in #206.

1. Gateway didn't boot — prometheusremotewriteexporter not in the patched build

The OCB manifest at opentelemetry-collector-contrib-patch/cmd/sketchcollector/builder-config.yaml ships only otlpexporter, otlphttpexporter, debugexporter, prometheusexporter — no PRW. gateway.yaml's "non-active fallback" PRW block fails the config decode on startup:

'exporters' unknown type: "prometheusremotewrite"
  (valid values: [otlphttp prometheus debug otlp])

Removed the block. Comment in its place documents that adding PRW would need a manifest edit + rebuild.

2. service.telemetry.metrics.address is deprecated in v0.141

v0.141 only accepts the readers: form (the agent configs already use it). Same fix applied in all three gateway configs:

metrics:
  level: detailed
  readers:
    - pull:
        exporter:
          prometheus:
            host: 0.0.0.0
            port: 8890

3. The DD/KLL/HLL "merge processors" follow-up I named in #206 isn't needed

I had this wrong in the #206 description. The existing five sketch processors already handle merge for the typed wire format — each one's input switch dispatches both raw inputs AND typed sketch inputs into the same windowed accumulator. Code references:

  • processor/countminsketchprocessor/processor.go:269 — case pmetric.MetricTypeCountMinSketch: decodes incoming typed CMS payloads via inboundDecodeCMS.
  • processor/ddsketchprocessor/processor.go:206 — consumeDDSketchDataPoints decodes incoming typed DDSketch payloads.

Same in the other three. Configuring the regular ddsketch processor at the gateway with the same window the agent used produces a windowed cross-agent merge. No new merge processors needed.

The legacy countminsketchmergeprocessor / countsketchmergeprocessor handle a DIFFERENT wire format (Gauge data points with sketch_payload byte attributes — pre-typed-Metric.data). They're not used by today's typed-wire e2e.

gateway-aggregate-from-sketches.yaml now uses ddsketch (same as agent) with KLL / HLL / CS / CMS commented in for opt-in, plus a comment block explaining the dual-mode dispatch.

e2e probe

With the patched gateway up and these fixes applied, sent a test OTLP probe (HTTP, Gauge metric test_probe) from inside the docker network. It traversed:

  • Gateway OTLP HTTP receiver: transport="http" accepted_metric_points=1
  • Gateway batch processor: 1 batch, send size 1
  • Gateway OTLP exporter to backend: sent_metric_points=1
  • Backend OTLP gRPC receiver: OTLP ingest: received 1 resource metrics, 1 total data points (transport=gRPC)
  • Backend ingest router: OTLP parse example: test_probe {} @1714525000000000000ns = 42

End-to-end OTLP round-trip works, confirming the patched-gateway preservation is correct. The typed-sketch path uses the same OTLP framing → inherits the preservation.

(Aside, not in scope for this PR: separately observed the agent's [ddsketch, batch] pipeline producing output_metric_points_total=4000 from ddsketch but no exporter_sent_metric_points_total on the wire — looks like an agent-side pipeline-wiring issue between ddsketch and batch processors. Orthogonal to the gateway preservation. Worth a separate look.)

Test plan

  • gateway.yaml parses against patched build's component set (no PRW component).
  • All three gateway configs use the v0.141 readers: telemetry syntax.
  • gateway-aggregate-from-sketches.yaml uses ddsketch from the regular processor set, not the legacy merge processors.
  • Stack comes up cleanly with GATEWAY_CONFIG=gateway.yaml (default).
  • Probe traversal confirmed end-to-end (test_probe Gauge through agent → gateway → backend).
  • Real sketch traffic verification deferred — the agent's [ddsketch, batch] flow has an unrelated issue.

🤖 Generated with Claude Code

Caught when running the e2e for the first time after #206 merged.
Three gateway configs landed in #206 referenced components that
the patched `asap/sketchcol:dev` build doesn't include:

1. **`prometheusremotewriteexporter` is not in the patched build.**
   The OCB manifest at
   `opentelemetry-collector-contrib-patch/cmd/sketchcollector/builder-config.yaml`
   only ships `otlpexporter`, `otlphttpexporter`, `debugexporter`,
   `prometheusexporter` — no PRW. `gateway.yaml`'s `prometheusremotewrite/backend`
   block (kept as a "non-active fallback") fails the config decode
   on startup:
       'exporters' unknown type: "prometheusremotewrite"
       (valid values: [otlphttp prometheus debug otlp])
   Removed the block. Comment in its place documents that the PRW
   exporter would need to be added to the OCB manifest + rebuild;
   sketch-native pipelines don't need it.

2. **`service.telemetry.metrics.address` is the deprecated v0.140
   shorthand.** v0.141 only accepts the `readers:` form (matches
   what the agent configs already use). Same pattern in all three
   gateway configs:
       readers:
         - pull:
             exporter:
               prometheus:
                 host: 0.0.0.0
                 port: 8890

3. **`gateway-aggregate-from-sketches.yaml` was using the wrong
   processors.** I claimed in #206 that DD/KLL/HLL needed new
   merge processors as a follow-up. That was wrong. The existing
   five sketch processors already handle merge for the typed wire
   format — each one's input switch dispatches both raw inputs
   AND typed sketch inputs into the same windowed accumulator
   (see `processor/countminsketchprocessor/processor.go:269`
   `case pmetric.MetricTypeCountMinSketch:` and
   `processor/ddsketchprocessor/processor.go:206`
   `consumeDDSketchDataPoints`). Configuring the regular
   `ddsketch` processor at the gateway with the same window the
   agent used produces a windowed cross-agent merge.

   The legacy `countminsketchmergeprocessor` /
   `countsketchmergeprocessor` are for the OLD Gauge-with-payload
   wire format only (where sketch bytes were stuffed into a
   `sketch_payload` byte attribute on a Gauge data point). The
   typed wire format (`Metric.data` variants 13–17) supersedes
   that. The follow-up PR I said I'd open isn't needed.

   This config now uses `ddsketch` (same as the agent) with KLL /
   HLL / CountSketch / CMS commented in for opt-in, and a
   prominent comment block explaining the dual-mode dispatch.

PROGRESS.md item #1 updated to reflect both the boot fix and the
merge-processors clarification, plus a record of the e2e probe
that proved the gateway preservation works (Gauge probe via
HTTP traversed agent-tier → gateway → backend OTLP receiver
intact).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 95a05de into main May 1, 2026
@zzylol
zzylol deleted the fix/gateway-boot-and-merge-config branch May 1, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant