fix(gateway): unbreak boot post-#206 + simplify merge config - #207
Merged
Merged
Conversation
Caught when running the e2e for the first time after #206 merged. Three gateway configs landed in #206 referenced components that the patched `asap/sketchcol:dev` build doesn't include: 1. **`prometheusremotewriteexporter` is not in the patched build.** The OCB manifest at `opentelemetry-collector-contrib-patch/cmd/sketchcollector/builder-config.yaml` only ships `otlpexporter`, `otlphttpexporter`, `debugexporter`, `prometheusexporter` — no PRW. `gateway.yaml`'s `prometheusremotewrite/backend` block (kept as a "non-active fallback") fails the config decode on startup: 'exporters' unknown type: "prometheusremotewrite" (valid values: [otlphttp prometheus debug otlp]) Removed the block. Comment in its place documents that the PRW exporter would need to be added to the OCB manifest + rebuild; sketch-native pipelines don't need it. 2. **`service.telemetry.metrics.address` is the deprecated v0.140 shorthand.** v0.141 only accepts the `readers:` form (matches what the agent configs already use). Same pattern in all three gateway configs: readers: - pull: exporter: prometheus: host: 0.0.0.0 port: 8890 3. **`gateway-aggregate-from-sketches.yaml` was using the wrong processors.** I claimed in #206 that DD/KLL/HLL needed new merge processors as a follow-up. That was wrong. The existing five sketch processors already handle merge for the typed wire format — each one's input switch dispatches both raw inputs AND typed sketch inputs into the same windowed accumulator (see `processor/countminsketchprocessor/processor.go:269` `case pmetric.MetricTypeCountMinSketch:` and `processor/ddsketchprocessor/processor.go:206` `consumeDDSketchDataPoints`). Configuring the regular `ddsketch` processor at the gateway with the same window the agent used produces a windowed cross-agent merge. The legacy `countminsketchmergeprocessor` / `countsketchmergeprocessor` are for the OLD Gauge-with-payload wire format only (where sketch bytes were stuffed into a `sketch_payload` byte attribute on a Gauge data point). The typed wire format (`Metric.data` variants 13–17) supersedes that. The follow-up PR I said I'd open isn't needed. This config now uses `ddsketch` (same as the agent) with KLL / HLL / CountSketch / CMS commented in for opt-in, and a prominent comment block explaining the dual-mode dispatch. PROGRESS.md item #1 updated to reflect both the boot fix and the merge-processors clarification, plus a record of the e2e probe that proved the gateway preservation works (Gauge probe via HTTP traversed agent-tier → gateway → backend OTLP receiver intact). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Caught when running the e2e for the first time after #206 merged. Two boot bugs in the gateway configs land here, plus a scope-correction on the "DD/KLL/HLL merge processors follow-up" I named in #206.
1. Gateway didn't boot —
prometheusremotewriteexporternot in the patched buildThe OCB manifest at
opentelemetry-collector-contrib-patch/cmd/sketchcollector/builder-config.yamlships onlyotlpexporter,otlphttpexporter,debugexporter,prometheusexporter— no PRW.gateway.yaml's "non-active fallback" PRW block fails the config decode on startup:Removed the block. Comment in its place documents that adding PRW would need a manifest edit + rebuild.
2.
service.telemetry.metrics.addressis deprecated in v0.141v0.141 only accepts the
readers:form (the agent configs already use it). Same fix applied in all three gateway configs:3. The DD/KLL/HLL "merge processors" follow-up I named in #206 isn't needed
I had this wrong in the #206 description. The existing five sketch processors already handle merge for the typed wire format — each one's input switch dispatches both raw inputs AND typed sketch inputs into the same windowed accumulator. Code references:
processor/countminsketchprocessor/processor.go:269—case pmetric.MetricTypeCountMinSketch:decodes incoming typed CMS payloads viainboundDecodeCMS.processor/ddsketchprocessor/processor.go:206—consumeDDSketchDataPointsdecodes incoming typed DDSketch payloads.Same in the other three. Configuring the regular
ddsketchprocessor at the gateway with the same window the agent used produces a windowed cross-agent merge. No new merge processors needed.The legacy
countminsketchmergeprocessor/countsketchmergeprocessorhandle a DIFFERENT wire format (Gauge data points withsketch_payloadbyte attributes — pre-typed-Metric.data). They're not used by today's typed-wire e2e.gateway-aggregate-from-sketches.yamlnow usesddsketch(same as agent) with KLL / HLL / CS / CMS commented in for opt-in, plus a comment block explaining the dual-mode dispatch.e2e probe
With the patched gateway up and these fixes applied, sent a test OTLP probe (HTTP, Gauge metric
test_probe) from inside the docker network. It traversed:transport="http"accepted_metric_points=1OTLP ingest: received 1 resource metrics, 1 total data points (transport=gRPC)OTLP parse example: test_probe {} @1714525000000000000ns = 42End-to-end OTLP round-trip works, confirming the patched-gateway preservation is correct. The typed-sketch path uses the same OTLP framing → inherits the preservation.
(Aside, not in scope for this PR: separately observed the agent's
[ddsketch, batch]pipeline producingoutput_metric_points_total=4000from ddsketch but noexporter_sent_metric_points_totalon the wire — looks like an agent-side pipeline-wiring issue between ddsketch and batch processors. Orthogonal to the gateway preservation. Worth a separate look.)Test plan
gateway.yamlparses against patched build's component set (no PRW component).readers:telemetry syntax.gateway-aggregate-from-sketches.yamlusesddsketchfrom the regular processor set, not the legacy merge processors.GATEWAY_CONFIG=gateway.yaml(default).[ddsketch, batch]flow has an unrelated issue.🤖 Generated with Claude Code