feat(deploy): warm-tier OTLP end-to-end — gateway → backend OTLP, alt query_engine_rust image - #205
Merged
Merged
Conversation
… query_engine_rust image
Closes the warm-tier sketch path in the deploy stack. Previously
the gateway's `prometheusremotewrite/backend` exporter dropped
typed sketch data points (DDSketch / KLLSketch / HLLSketch /
CountSketch / CountMinSketch on Metric.data tags 13–17), so the
e2e flow only ran through the cold-tier path. Companion PR
ASAPQuery-backend#69 lands the matching OTLP receiver on the
backend.
Both follow-ups from ASAPCollector#204 PROGRESS.md item 1 are
included:
(a) **OTLP on the existing precompute_engine binary.**
- `gateway.yaml`: replace `prometheusremotewrite/backend` in
the metrics pipeline with `otlp/backend` (gRPC →
`backend:4317`, insecure TLS, sending_queue enabled). PRW
exporter retained as a non-active fallback for legacy
clients / direct PRW writers.
- `base.yml`: backend command gets `--enable-otel-ingest
--otel-grpc-port=4317 --otel-http-port=4318`. OTLP ports
stay intra-docker (gateway → backend); 19090/19091 host
mappings are unchanged.
(b) **Alternative query_engine_rust image.**
- `Dockerfile.backend.queryengine`: same multi-stage build
as `Dockerfile.backend` but ships `query_engine_rust` (the
workspace's default binary) instead of `precompute_engine`.
Use when the deploy needs the binary's extra subsystems:
controller capability-miss callbacks, query tracker,
schema eviction, backfill worker — none of which
`precompute_engine` wires up.
- `queryengine-overlay.yml`: compose overlay swapping the
backend image to `asap/query-backend-queryengine:dev` with
the larger required flag set (`--config`,
`--query-language=promql`, `--lock-strategy=per-key`,
`--prometheus-scrape-interval=30`, `--output-dir`,
`--http-port=8088`, `--controller-endpoint=...`,
`--enable-otel-ingest`, etc.). Same OTLP wire, port
remapped on the host so 19091 still serves PromQL across
both backend variants.
PROGRESS.md follow-up #1 marked done; P3 limitation note
updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
zzylol
added a commit
that referenced
this pull request
May 1, 2026
…es (#206) Real fix for the warm-tier sketch-drop the e2e harness has been working around. #205's yaml-only swap (PRW → OTLP exporter) didn't solve it because the drop wasn't at the exporter — it was at the gateway's pdata *unmarshal* step. Stock `otel/opentelemetry-collector-contrib:0.108.0`'s pdata only knows the standard `Metric.data` OneOf variants (Gauge / Sum / Histogram / ExpHistogram / Summary). Tags 13–17 (DDSketch / KLLSketch / HLLSketch / CountSketch / CountMinSketch) hit the `default:` arm at `opentelemetry-collector/pdata/internal/generated_proto_metric.go:1201` which calls `proto.ConsumeUnknown(...)`. That advances past the bytes without storing them — there's no `XXX_unrecognized` field on the Metric struct to catch them — so by the time any exporter sees the metric, the typed sketch payload is gone. Fix: run the gateway from the ASAP-patched OTel collector — the same `asap/sketchcol:dev` image agents already use, whose pdata is generated from the patched proto and round-trips tags 13–17 intact. base.yml: - gateway service: image stock 0.108 → asap/sketchcol:dev. - Mounted config selectable via $GATEWAY_CONFIG env var (mirrors $AGENT_CONFIG). Default = gateway.yaml. - Self-telemetry host port 18889:8889 → 18890:8890 (the patched build doesn't include a prometheus exporter on 8889; self- telemetry has been on 8890 since #205, intra-docker scrape was already pointed there). deploy/configs/: - gateway.yaml — pure forwarder (default). Updated comments to document the patched-build dependency + cross-reference the two aggregate variants. - gateway-aggregate-from-raw.yaml — agents send raw OTLP; gateway runs DDSketch processor at gateway tier (KLL / HLL / CS / CMS blocks present but commented out — uncomment to add). Use when sketch placement is at the gateway tier per the controller's plan. - gateway-aggregate-from-sketches.yaml — agents already sketched; gateway runs countminsketchmerge + countsketchmerge to reconstruct full state from delta payloads. DD / KLL / HLL pass through unchanged (merge processors for those don't exist yet; backend handles per-agent merging via the accumulator's merge_into path). Builder manifests (both `builder-config.yaml` and the e2e-harness-trimmed `builder-config-sketches.yaml`): - Add countminsketchmergeprocessor + countsketchmergeprocessor. Their factories expose `countminsketchmerge` / `countsketchmerge` as the OTel component types — the names referenced in the new gateway-aggregate-from-sketches.yaml. PROGRESS.md follow-up #1 rewritten to document the real root cause and the real fix; the yaml-only changes from #205 are still in main and remain correct (just not sufficient alone). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the warm-tier sketch path in the deploy stack. Follows up on the recently-merged #204; companion PR ASAPQuery-backend#69 lands the matching OTLP receiver on the backend.
With #204's runtime path landed, sketches were still getting dropped at the OTel gateway because
prometheusremotewritecan't translate the typedMetric.datavariants — so the e2e only flowed through the cold tier. This PR closes that, in both forms #204's PROGRESS.md item 1 named.(a) OTLP on the existing
precompute_enginebinary — default pathgateway.yaml: metrics pipeline now exports throughotlp/backend(gRPC →backend:4317, insecure, sending_queue on) instead ofprometheusremotewrite/backend. PRW exporter retained as a non-active fallback for legacy clients / direct PRW writers (it's not inservice.pipelines.metrics.exporters).base.yml: backend command picks up--enable-otel-ingest --otel-grpc-port=4317 --otel-http-port=4318. OTLP stays intra-docker; the 19090/19091 host mappings are unchanged.(b) Alternative
query_engine_rustimage — opt-in via overlayDockerfile.backend.queryengine(new): same multi-stage build asDockerfile.backendbut shipsquery_engine_rust(the workspace's default binary). Use when the deploy needs the binary's extra subsystems — controller capability-miss callbacks, query tracker, schema eviction, backfill worker — thatprecompute_enginedoesn't wire up.queryengine-overlay.yml(new): compose overlay that swaps the backend image toasap/query-backend-queryengine:devand provides the larger required flag set. Same OTLP wire on:4317, host port 19091 still serves PromQL across both backend variants.PROGRESS.md
Follow-up #1 marked done. P3 row's limitation note updated.
Why both, not just (a)?
(a)solves the e2e goal end-to-end with the focused binary the harness drives today.(b)is the path forward when you want the controller in-loop on capability misses (which closes the loop on the project's stated North Star — the controller decides where each sketch is computed — for queries the agent's current sketch can't answer). They share the same OTLP receiver on the backend side, so the wire format is identical.Test plan
docker compose -f base.yml -f agents-N1.yml -f baseline-b3-delta.yml -f e2e-overlay.yml config --quietparses cleanly-f queryengine-overlay.ymlappended also parses cleanlygateway.yamlparses; pipeline exporters list =[otlp/backend]histogram_quantile(0.5, ...)on a DDSketch agg, observe non-cold-fallback log line on the backend.asap/query-backend-queryengine:devonce, bring up the stack withqueryengine-overlay.yml, repeat the warm-tier check.🤖 Generated with Claude Code