mvp phase ε.1.5: sketchotap + sketchtelegraf Mode 3 emitters - #322
Merged
Merged
Conversation
Add per-runtime emit functions mirroring `emit_edge_yaml`'s three-mode dispatch from Phase ε.1, so all three edge runtimes (sketchcollector / sketchotap / sketchtelegraf) can produce wire bytes from a single typed L5 EdgeStageConfig. Mode 3 (RawAtEdgePrometheusArchive) routes raw metrics directly to Prometheus's archive in each runtime's idiom. * `stage_config_otap::emit_otap_dag_yaml` produces an otap-dataflow DAG YAML (`version: otel_dataflow/v1`) with `receiver:otlp` → optional `processor:asap_sketches` → `exporter:otlp_grpc` (Modes 1/2) or `exporter:otlp_http` to Prometheus's `/api/v1/otlp/v1/metrics` (Mode 3). * `stage_config_telegraf::emit_telegraf_toml` produces Telegraf TOML with `[[inputs.opentelemetry]]` → optional `[[processors.allsketches]]` → `[[outputs.opentelemetry]]` to gateway (Modes 1/2) or `[[outputs.http]]` with `data_format = "prometheusremotewrite"` to Prometheus's `/api/v1/write` (Mode 3). Telegraf's shipped `outputs.opentelemetry` is gRPC-only with no HTTP/protobuf alternative; remote-write lands in the same Prometheus TSDB the OTel-collector path lands in via OTLP HTTP, so the archive contents are identical (only the wire framing differs). * New `AgentRuntime` enum (Sketchcollector / Sketchotap / Sketchtelegraf) + `emit_for_runtime` dispatcher reads the `X-Agent-Runtime` OpAMP header and dispatches accordingly. Defaults to Sketchcollector when absent so legacy agents keep working. Tests: 16 new unit tests covering each runtime's three-mode shape and runtime-dispatch behavior; the OTAP DAG round-trips through a serde_yaml stub (the otap-df-config crate would add a large dependency footprint to controller), and the Telegraf TOML round-trips through a purpose-built minimal validator that catches malformed table headers / unbalanced quotes. Pre-existing 10 controller test failures are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Note that all three edge runtimes (sketchcollector / sketchotap / sketchtelegraf) support all three Phase ε.1 BindModes via dedicated emit functions, with the Telegraf Mode 3 path using `outputs.http` + prometheusremotewrite as a documented deviation from the OTel collector's `otlphttp/prometheus` (Telegraf has no OTLP-HTTP serializer in our shipped version). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase ε.1.5 — re-dispatch the Phase ε.1 three-mode emit path across all three edge runtimes. Phase ε.1 (#319) shipped
BindMode+WireCostTable+SketchExpr::RawAtEdgePrometheusArchive+ theemit_edge_yamlMode 3 branch for the OTel-collector runtime; this PR mirrors that forsketchotap(otap-dataflow Rust) andsketchtelegraf.controller/src/config/stage_config_otap.rs—emit_otap_dag_yamlproduces an otap-dataflow DAG (version: otel_dataflow/v1),receiver:otlp→ optionalprocessor:asap_sketches→exporter:otlp_grpcto gateway (Modes 1/2) orexporter:otlp_httpto Prometheus's/api/v1/otlp/v1/metrics(Mode 3).controller/src/config/stage_config_telegraf.rs—emit_telegraf_tomlproduces[[inputs.opentelemetry]]→ optional[[processors.allsketches]]→[[outputs.opentelemetry]]to gateway (Modes 1/2) or[[outputs.http]]withdata_format = "prometheusremotewrite"to Prometheus's/api/v1/write(Mode 3). Telegraf'soutputs.opentelemetryis gRPC-only with noprotocol = "http/protobuf"field; remote-write lands in the same Prometheus TSDB so the storage outcome is identical (only the wire framing differs).AgentRuntimeenum +emit_for_runtimedispatcher incontroller/src/config/mod.rs. ReadsX-Agent-RuntimeOpAMP header (sketchcollector/sketchotap/sketchtelegraf); defaults toSketchcollectorwhen absent so legacy agents keep working.AgentRuntimedid NOT exist on main — this is a new enum, parallel to the existingopamp::AgentRole.docs/control-plane-design.mddocuments the per-runtime emit paths.Test plan
cargo test --release -p controller— 549 passed, 10 pre-existing failures unchangedserde_yaml::from_str::<OtapDagStub>, Telegraf via the in-tree minimal TOML validatoremit_for_runtime(Sketchcollector, ...)is identical to directemit_edge_yaml(regression guard)Mode 3 exporter shapes (one-line per runtime)
sketchcollector(existing):exporters.otlphttp/prometheus.metrics_endpoint = "${ASAP_PROMETHEUS_OTLP_URL:-http://prometheus:9090/api/v1/otlp/v1/metrics}"sketchotap(new): nodeexporterof typeexporter:otlp_httpwithmetrics_endpoint: http://prometheus:9090/api/v1/otlp/v1/metricssketchtelegraf(new):[[outputs.http]]url = "http://prometheus:9090/api/v1/write"+data_format = "prometheusremotewrite"(Telegraf-specific deviation — see PR body)Plugin compatibility notes for Phase 3.3
outputs.opentelemetryplugin in our shipped tree (telegraf/plugins/outputs/opentelemetry/opentelemetry.go) is gRPC-only with no HTTP/protobuf option. Phase 3.3 demo runs that exercisesketchtelegrafMode 3 must point Prometheus's--web.enable-remote-write-receiverflag (or use the dedicated remote-write endpoint) — Prometheus accepts both OTLP HTTP at/api/v1/otlp/v1/metricsand remote-write at/api/v1/writeand stores them in the same TSDB.processor:asap_sketchesURN in the emitted OTAP DAG is the contract Phase 3.3'ssketchotapbinary must register against (vialinkme::distributed_slice(OTAP_PROCESSOR_FACTORIES)); the plugin source lives inotap-patch/plugins/asap_sketches/perdocs/design-asap-otap-rust-integration.md.🤖 Generated with Claude Code