Skip to content

mvp phase ε.1.5: sketchotap + sketchtelegraf Mode 3 emitters - #322

Merged
zzylol merged 2 commits into
mainfrom
mvp/phase-eps.1.5-otap-telegraf-mode3
May 7, 2026
Merged

zzylol merged 2 commits into
mainfrom
mvp/phase-eps.1.5-otap-telegraf-mode3

Conversation

@zzylol

@zzylol zzylol commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase ε.1.5 — re-dispatch the Phase ε.1 three-mode emit path across all three edge runtimes. Phase ε.1 (#319) shipped BindMode + WireCostTable + SketchExpr::RawAtEdgePrometheusArchive + the emit_edge_yaml Mode 3 branch for the OTel-collector runtime; this PR mirrors that for sketchotap (otap-dataflow Rust) and sketchtelegraf.

  • New controller/src/config/stage_config_otap.rs — emit_otap_dag_yaml produces an otap-dataflow DAG (version: otel_dataflow/v1), receiver:otlp → optional processor:asap_sketches → exporter:otlp_grpc to gateway (Modes 1/2) or exporter:otlp_http to Prometheus's /api/v1/otlp/v1/metrics (Mode 3).
  • New controller/src/config/stage_config_telegraf.rs — emit_telegraf_toml produces [[inputs.opentelemetry]] → optional [[processors.allsketches]] → [[outputs.opentelemetry]] to gateway (Modes 1/2) or [[outputs.http]] with data_format = "prometheusremotewrite" to Prometheus's /api/v1/write (Mode 3). Telegraf's outputs.opentelemetry is gRPC-only with no protocol = "http/protobuf" field; remote-write lands in the same Prometheus TSDB so the storage outcome is identical (only the wire framing differs).
  • New AgentRuntime enum + emit_for_runtime dispatcher in controller/src/config/mod.rs. Reads X-Agent-Runtime OpAMP header (sketchcollector / sketchotap / sketchtelegraf); defaults to Sketchcollector when absent so legacy agents keep working. AgentRuntime did NOT exist on main — this is a new enum, parallel to the existing opamp::AgentRole.
  • 16 new unit tests covering each runtime's three modes + the dispatcher; OTAP DAGs round-trip through a serde_yaml stub, Telegraf TOML through a purpose-built minimal validator that catches malformed table headers / unbalanced quotes.
  • docs/control-plane-design.md documents the per-runtime emit paths.

Test plan

  • cargo test --release -p controller — 549 passed, 10 pre-existing failures unchanged
  • Mode 1 / Mode 2 / Mode 3 snapshot tests for each new runtime emitter
  • Mode 3 outputs are parseable: OTAP via serde_yaml::from_str::<OtapDagStub>, Telegraf via the in-tree minimal TOML validator
  • emit_for_runtime(Sketchcollector, ...) is identical to direct emit_edge_yaml (regression guard)

Mode 3 exporter shapes (one-line per runtime)

  • sketchcollector (existing): exporters.otlphttp/prometheus.metrics_endpoint = "${ASAP_PROMETHEUS_OTLP_URL:-http://prometheus:9090/api/v1/otlp/v1/metrics}"
  • sketchotap (new): node exporter of type exporter:otlp_http with metrics_endpoint: http://prometheus:9090/api/v1/otlp/v1/metrics
  • sketchtelegraf (new): [[outputs.http]] url = "http://prometheus:9090/api/v1/write" + data_format = "prometheusremotewrite" (Telegraf-specific deviation — see PR body)

Plugin compatibility notes for Phase 3.3

  • Telegraf outputs.opentelemetry plugin in our shipped tree (telegraf/plugins/outputs/opentelemetry/opentelemetry.go) is gRPC-only with no HTTP/protobuf option. Phase 3.3 demo runs that exercise sketchtelegraf Mode 3 must point Prometheus's --web.enable-remote-write-receiver flag (or use the dedicated remote-write endpoint) — Prometheus accepts both OTLP HTTP at /api/v1/otlp/v1/metrics and remote-write at /api/v1/write and stores them in the same TSDB.
  • The processor:asap_sketches URN in the emitted OTAP DAG is the contract Phase 3.3's sketchotap binary must register against (via linkme::distributed_slice(OTAP_PROCESSOR_FACTORIES)); the plugin source lives in otap-patch/plugins/asap_sketches/ per docs/design-asap-otap-rust-integration.md.

🤖 Generated with Claude Code

zzylol and others added 2 commits May 7, 2026 18:22
Add per-runtime emit functions mirroring `emit_edge_yaml`'s three-mode
dispatch from Phase ε.1, so all three edge runtimes (sketchcollector /
sketchotap / sketchtelegraf) can produce wire bytes from a single typed
L5 EdgeStageConfig. Mode 3 (RawAtEdgePrometheusArchive) routes raw
metrics directly to Prometheus's archive in each runtime's idiom.

* `stage_config_otap::emit_otap_dag_yaml` produces an otap-dataflow DAG
  YAML (`version: otel_dataflow/v1`) with `receiver:otlp` →
  optional `processor:asap_sketches` → `exporter:otlp_grpc` (Modes 1/2)
  or `exporter:otlp_http` to Prometheus's `/api/v1/otlp/v1/metrics`
  (Mode 3).

* `stage_config_telegraf::emit_telegraf_toml` produces Telegraf TOML
  with `[[inputs.opentelemetry]]` → optional
  `[[processors.allsketches]]` → `[[outputs.opentelemetry]]` to gateway
  (Modes 1/2) or `[[outputs.http]]` with `data_format =
  "prometheusremotewrite"` to Prometheus's `/api/v1/write` (Mode 3).
  Telegraf's shipped `outputs.opentelemetry` is gRPC-only with no
  HTTP/protobuf alternative; remote-write lands in the same Prometheus
  TSDB the OTel-collector path lands in via OTLP HTTP, so the archive
  contents are identical (only the wire framing differs).

* New `AgentRuntime` enum (Sketchcollector / Sketchotap /
  Sketchtelegraf) + `emit_for_runtime` dispatcher reads the
  `X-Agent-Runtime` OpAMP header and dispatches accordingly. Defaults
  to Sketchcollector when absent so legacy agents keep working.

Tests: 16 new unit tests covering each runtime's three-mode shape and
runtime-dispatch behavior; the OTAP DAG round-trips through a
serde_yaml stub (the otap-df-config crate would add a large dependency
footprint to controller), and the Telegraf TOML round-trips through a
purpose-built minimal validator that catches malformed table
headers / unbalanced quotes.

Pre-existing 10 controller test failures are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Note that all three edge runtimes (sketchcollector / sketchotap /
sketchtelegraf) support all three Phase ε.1 BindModes via dedicated
emit functions, with the Telegraf Mode 3 path using `outputs.http` +
prometheusremotewrite as a documented deviation from the OTel
collector's `otlphttp/prometheus` (Telegraf has no OTLP-HTTP
serializer in our shipped version).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 9ecc1aa into main May 7, 2026
@zzylol
zzylol deleted the mvp/phase-eps.1.5-otap-telegraf-mode3 branch May 9, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant