feat(opampextension): apply pushed RemoteConfig via disk-write + restart (#381) - #383
Merged
Merged
Conversation
…art (ASAPCollector#381) Patches the upstream opampextension's `onMessage` callback to handle `msg.RemoteConfig` — closes the residual half of PR #350. Background. The agent connects to the controller's OpAMP server, receives `AgentRemoteConfig` messages, but the upstream extension's onMessage handler only processes `AgentIdentification` and `CustomMessage`. RemoteConfig is silently dropped. Result: the controller's elaborate typed-stage-split runtime YAML (routing connector + 5 sketch processors + gorillas3) is generated, pushed, and ignored — every MVP demo since runs the agent's static bootstrap YAML, the central blocker tracked in this repo's #381. Patch (overlay at `opentelemetry-collector-contrib-patch/extension/opampextension/`): * `config.go` — new `RemoteConfigPath string` field (mapstructure key `remote_config_path`). When empty, the extension stays "report-only" (upstream behavior preserved). When set, the extension applies pushed configs by writing to that path. * `opamp_agent.go`: - `onMessage` now dispatches `msg.RemoteConfig` to a new `processRemoteConfig` method when both `msg.RemoteConfig` and `cfg.RemoteConfigPath` are non-nil/non-empty. - `Start` advertises `AcceptsRemoteConfig` capability when `RemoteConfigPath` is set. The OpAMP client library at `client/internal/receivedprocessor.go:83` silently drops `RemoteConfig` without this capability — that was the first dead-end on debug. - `processRemoteConfig` picks the first non-empty ConfigFile body (preferring `""` / `"asap-otel"` keys), YAML-validates it, writes directly to disk (NOT write-tmp+rename — Linux returns EBUSY on `renameat2` for Docker bind-mounted files), reports `RemoteConfigStatuses_APPLIED`, sleeps 200ms to flush the status, and `os.Exit(0)`. Docker `restart=unless-stopped` / systemd `Restart=always` is expected to bring the collector back up loading the freshly-written YAML. Non-code files (`auth.go`, `factory.go`, `logger.go`, etc.) are verbatim copies of the upstream extension. OCB's per-extension module replace requires the WHOLE module directory under the replace path; we can't just override two files. builder-config.yaml: adds `path: ./extension/opampextension` beneath the existing `gomod:` line so OCB redirects to the patch overlay. Mirrors the pattern every other patched processor uses (ddsketchprocessor, kllprocessor, …). End-to-end verification (single-node MVP smoke test under /mydata/mvp-smoke-test/): * Before patch: agent stays on bootstrap YAML; `restartCount=0`; backend sees sketches keyed by full wire-attr fingerprint. * After patch: agent applies pushed config and exits; `restartCount=1+`; host-side mounted `agent.yaml` is overwritten with the controller-emitted typed-stage-split runtime YAML. The agent's post-restart behavior surfaces downstream issues (gorillas3 S3 bucket not provisioned in smoke setup, memory limiter trips on the 5-sketch-pipeline workload) — those are separate blockers, not in scope for this PR. The OpAMP-apply half of #381 is what this PR closes. Companion: ASAPQuery-backend `b1-5-controller-emit-literal-env-vars` branch — switches the controller's gorillas3 YAML emit from bash-style `${VAR:-default}` (rejected by OTel's confmap URI parser) to literal values resolved from the controller's env at emit time, and threads `remote_config_path: /etc/otel/config.yaml` into every emitted opamp extension block so OpAMP-apply persists across the restart loop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 18, 2026
…locks (#391) PR #383 (and #389 follow-up) added the `RemoteConfigPath` field to the patched opampextension: when set, the extension applies pushed AgentRemoteConfig by writing the YAML to disk and exiting, so a restart-policy-enabled supervisor (Docker `restart: unless-stopped`, systemd, k8s) brings the agent back up with the new config. The mvp-smoke-test agent uses this field via its overlay-mounted `agent.yaml`, but the 13 stock agent yamls in `deploy/mvp-singlenode/configs/asap-otel-agent-*.yaml` still ship without it, so OpAMP-apply silently no-ops in stock single-node deploys (the controller's push is received, the extension reports back, but nothing is persisted across the implied restart — the next boot reloads the same static bootstrap yaml). This change adds `remote_config_path: /etc/otel/config.yaml` (the same path the collector boots from via `--config=/etc/otel/config.yaml`, and the same convention the smoke-test uses) to the `extensions.opamp:` block of every stock yaml that already declares one. Files touched (13): - asap-otel-agent-allsketches.yaml - asap-otel-agent-b0-prometheus.yaml - asap-otel-agent-b0-victoriametrics.yaml - asap-otel-agent-b1-serf-prometheus.yaml - asap-otel-agent-b3-delta-direct.yaml - asap-otel-agent-b5-gorilla-prometheus.yaml - asap-otel-agent-b5-gorilla-victoriametrics.yaml - asap-otel-agent-b6-asap-single-sketch.yaml - asap-otel-agent-b6-gorilla-s3.yaml - asap-otel-agent-cms-direct.yaml - asap-otel-agent-cs-direct.yaml - asap-otel-agent-hll-direct.yaml - asap-otel-agent-kll-direct.yaml The 6 stock yamls WITHOUT an `extensions.opamp:` block (`b0a-raw-stream`, `b0b-raw-batched`, `b2-full`, `b3-delta`, `b4-tunable`, `kafka-fragment`) are intentionally left alone — adding `remote_config_path` would require also wiring up `extensions:` + `service.extensions:`, which is out of scope for this sweep. OPERATIONAL CAVEAT (flagged inline + in each yaml's comment): the stock `agents-N{1,10,100}.yml` compose overlays mount `/etc/otel/config.yaml` read-only (`:ro` suffix on the bind mount) and do NOT set a restart policy on the agent service. As-shipped, the extension's disk-write will fail with EROFS and the agent process will exit without coming back. To actually exercise OpAMP-apply persistence in a stock deploy, the operator must additionally: 1. Drop `:ro` from the agent's config volume mount (so the opampextension can write back to the same path the collector boots from), and 2. Add `restart: unless-stopped` (or equivalent) to the agent service so the container respawns after the extension exits. The smoke-test's `mvp-smoke-test/compose/smoke-overlay.yml` does both via `volumes: !override` + `restart: unless-stopped` — that is the reference pattern. The compose-side wiring update is deferred to a follow-up so this PR stays narrowly scoped to the agent yaml schema. YAML validity verified with `python3 -c "import yaml; ..."` over all 19 stock agent yamls; the 13 modified files parse with `extensions.opamp.remote_config_path == "/etc/otel/config.yaml"`. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #381 — the residual half of PR #350 where the agent connects to OpAMP but ignores incoming
AgentRemoteConfigmessages.The patch adds a new
RemoteConfigPath string(mapstructure:"remote_config_path") field to the opampextensionConfig. When empty, the extension stays "report-only" (upstream behavior preserved). When set,onMessagenow dispatchesmsg.RemoteConfigto a newprocessRemoteConfigmethod, andStartadvertises theAcceptsRemoteConfigcapability — without this the OpAMP client library atclient/internal/receivedprocessor.go:83silently drops incoming RemoteConfig.processRemoteConfigpicks the first non-empty ConfigFile body (preferring""/"asap-otel"keys), YAML-validates it, writes directly to disk (NOT write-tmp+rename — Linux returns EBUSY onrenameat2for Docker bind-mounted files), reportsRemoteConfigStatuses_APPLIED, sleeps 200ms to flush the status, andos.Exit(0). Dockerrestart=unless-stopped/ systemdRestart=alwaysbrings the collector back up loading the freshly-written YAML. Thebuilder-config.yamlchange addspath: ./extension/opampextensionunder the existinggomod:line so OCB resolves opampextension from the patch overlay — mirrors the pattern every other patched processor uses.Test plan
bash build_asap_otel.sh --skip-patchesbuilds clean"Applied OpAMP RemoteConfig"stringdocker build -f deploy/docker/Dockerfile.asap-otelproduces a valid image/mydata/mvp-smoke-test/run_smoke.sh:restartCountgoes from 0 → 1+, hostagent.yamlis overwritten with the controller's emitted runtime YAML (size changes from 5345 → 4053 bytes)Companion PR
ASAPQuery-backend
b1-5-controller-emit-literal-env-varsresolves env vars literally in the controller's gorillas3 emit so the agent doesn't crash-loop on${VAR:-default}after applying.🤖 Generated with Claude Code