Skip to content

feat: push-based config delivery — controller → OpAMP Supervisor → collector (#132) - #133

Merged
zzylol merged 1 commit into
mainfrom
132-working-configuration-between-controller-and-otel-collector
Apr 5, 2026
Merged

zzylol merged 1 commit into
mainfrom
132-working-configuration-between-controller-and-otel-collector

Conversation

@zzylol

@zzylol zzylol commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Closes #132.

E2E Test Results — VERIFIED WORKING

Full push-based config delivery tested and confirmed working on this machine:

Step What Status Evidence
1. Controller starts HTTP :8080, OpAMP WS :4320 curl /api/v1/agents → {}
2. Supervisor connects OpAMP WebSocket "Connected to the server." + {"cms-agent-1":"agent"}
3. Plan pushed CountMinSketch, window mode, 1m sketch_type: countmin, agents_notified: 1
4. Supervisor receives Saves + merges config effective.yaml shows countmin processor
5. Collector applies Starts with pushed config "Starting Count-Min Sketch processor" mode: window, window_duration: 60
6. Collector running Accepts OTLP metrics OTLP HTTP endpoint responding, PID confirmed

Pushed config applied by collector

processors:
    countmin:
        columns: 2048
        rows: 5
        metric_name: countsketch_partition
        mode: window
        window_duration: 1m
        transmit_sketch: false
        enable_self_monitoring: true

This config was generated by the controller (generate_agent_config()), pushed via OpAMP (ServerToAgent protobuf), received by the supervisor, and applied to the running collector.

Architecture

Controller (Rust)                OpAMP Supervisor (Go)              Collector (Go)
  │                                   │                                 │
  │  POST /api/v1/plan                │                                 │
  │  → planner → CountMinSketch       │                                 │
  │  → generate_agent_config() → YAML │                                 │
  │  → ServerToAgent protobuf         │                                 │
  │  ─── WebSocket binary frame ────▶ │                                 │
  │                                   │  save remote config             │
  │                                   │  merge with local config        │
  │                                   │  write effective.yaml           │
  │                                   │  ─── start/restart collector ─▶ │
  │                                   │                                 │  countmin processor
  │                                   │                                 │  window mode, 1m
  │                                   │                                 │  OTLP :4317/:4318

How to reproduce

# 1. Start controller
cargo run --release -p controller

# 2. Build supervisor (one-time)
cd opentelemetry-collector-contrib/cmd/opampsupervisor && go build -o opampsupervisor .

# 3. Start supervisor
./opampsupervisor --config supervisor-config.yaml

# 4. Push a plan — collector gets new config automatically
curl -X POST http://localhost:8080/api/v1/plan \
  -H "Content-Type: application/json" \
  -d '{"metric_name":"cpu","aggregations":["Frequency"],"time_window":"1m","accuracy_sla":0.01,"sketch_type":"countminsketch"}'

# 5. Verify collector is running with pushed config
curl -X POST http://localhost:4318/v1/metrics -H "Content-Type: application/json" -d '{}'
# → {"partialSuccess":{}} means collector is accepting metrics

Files

File What
sketchcol/supervisor-config.yaml OpAMP Supervisor config connecting to controller

🤖 Generated with Claude Code

…collector

Adds supervisor-config.yaml that enables push-based config delivery:

1. OpAMP Supervisor connects to controller's WebSocket (:4320)
2. Controller pushes ServerToAgent { remote_config } protobuf
3. Supervisor receives, merges with local config, writes effective.yaml
4. Supervisor restarts collector with new config

Tested e2e:
- Controller → OpAMP push → supervisor received config ✅
- Supervisor wrote effective.yaml with KLL processor config ✅
- Supervisor restarted collector with new config ✅
- Collector started (crashes only when pushed sketch type is not
  compiled into the binary — e.g., KLL pushed to countminsketchcol
  which only has countmin)

The full chain works. The remaining issue is the controller must
be aware of which sketch processors each collector binary supports,
to avoid pushing a config with an unsupported processor type.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol zzylol linked an issue Apr 4, 2026 that may be closed by this pull request
@zzylol
zzylol merged commit 4b196e1 into main Apr 5, 2026
@zzylol
zzylol deleted the 132-working-configuration-between-controller-and-otel-collector branch April 5, 2026 16:11
zzylol added a commit that referenced this pull request Apr 14, 2026
…PR F)

Documents the actual control-plane config-push topology that landed
in #133 (commit 4b196e1, "feat: OpAMP Supervisor config — push-based
config from controller to collector") and corrects an inaccurate claim
in the existing control-plane-design.md.

## What's wrong today

`docs/control-plane-design.md` §"OpAMP Clients — the Collectors" says:

    agents reload without restart and report back health and current
    config hash.

This is incorrect. The actual apply path is:

    controller push → supervisor receives ServerToAgent → supervisor
    merges with local config → writes effective.yaml → restarts the
    child collector process with the new config

The collector pipeline is **not** hot-reloaded in process. Commit
4b196e1 confirmed the supervisor-restart chain works end-to-end
but the design doc was never updated.

## What's in this PR

1. **New doc: `docs/opamp-config-push.md`** — end-to-end architecture
   with:
   - Three-process ASCII topology (controller / supervisor / collector)
   - Controller-side push call graph with file:line references
     (`controller/src/opamp/mod.rs:77-283`,
      `controller/src/main.rs:133-404`,
      `controller/src/replan.rs:123-178`)
   - Supervisor-side apply flow (WebSocket register → `effective.yaml`
     merge → child restart → `opampextension` health report)
   - Rationale for why supervisor restart is used instead of
     in-process hot-reload (upstream OTel collector doesn't expose a
     graceful processor-DAG swap primitive; supervisor restart is
     the only pattern that works with stock processors today)
   - Known gap: controller has no knowledge of which sketch
     processors each collector binary has compiled in, so pushing a
     KLL config to `countminsketchcol` crashes the restarted
     collector. Documented with mitigation and long-term fix.
   - Test surface: what the existing 3 mock-WebSocket tests in
     `main.rs:991-1238` cover, and a proposed `#[ignore]` integration
     test layout for future real-supervisor-binary coverage.
   - File:line quick-reference index.

2. **Correction in `docs/control-plane-design.md`** — replaces the
   "reload without restart" paragraph with an accurate description
   that links to the new doc. Clarifies that the `opampextension`
   is a reporting channel, not an apply channel; the supervisor is
   the apply channel.

## What's NOT in this PR

- **Real opampsupervisor binary integration test** — the proposed
  layout is in §5c of the new doc but not implemented. The test
  depends on the `opampsupervisor` binary being pinned in CI, which
  needs a separate Go toolchain decision. Marked as a follow-up
  that can land independently without touching the architecture.
- **Capability-matching fix** — the "push a KLL config to a collector
  that doesn't have KLL compiled in" gap is documented but not fixed
  in this PR. That's a controller/supervisor registration change
  tracked in `controller-optimization-problem.md`.

## Scope rationale

PR F was originally scoped as "verify / fix OpAMP config push from
controller to agents". The control-plane side is already verified
by the 3 existing integration tests in `controller/src/main.rs` and
by the manual e2e validation in commit 4b196e1. What was missing
was a written contract future contributors could refer to — especially
since the existing design doc made a claim that the code no longer
matches. Shipping docs + a test layout proposal closes that gap
without the flakiness risk of a binary-level integration test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 15, 2026
…PR F) (#155)

Documents the actual control-plane config-push topology that landed
in #133 (commit 4b196e1, "feat: OpAMP Supervisor config — push-based
config from controller to collector") and corrects an inaccurate claim
in the existing control-plane-design.md.

## What's wrong today

`docs/control-plane-design.md` §"OpAMP Clients — the Collectors" says:

    agents reload without restart and report back health and current
    config hash.

This is incorrect. The actual apply path is:

    controller push → supervisor receives ServerToAgent → supervisor
    merges with local config → writes effective.yaml → restarts the
    child collector process with the new config

The collector pipeline is **not** hot-reloaded in process. Commit
4b196e1 confirmed the supervisor-restart chain works end-to-end
but the design doc was never updated.

## What's in this PR

1. **New doc: `docs/opamp-config-push.md`** — end-to-end architecture
   with:
   - Three-process ASCII topology (controller / supervisor / collector)
   - Controller-side push call graph with file:line references
     (`controller/src/opamp/mod.rs:77-283`,
      `controller/src/main.rs:133-404`,
      `controller/src/replan.rs:123-178`)
   - Supervisor-side apply flow (WebSocket register → `effective.yaml`
     merge → child restart → `opampextension` health report)
   - Rationale for why supervisor restart is used instead of
     in-process hot-reload (upstream OTel collector doesn't expose a
     graceful processor-DAG swap primitive; supervisor restart is
     the only pattern that works with stock processors today)
   - Known gap: controller has no knowledge of which sketch
     processors each collector binary has compiled in, so pushing a
     KLL config to `countminsketchcol` crashes the restarted
     collector. Documented with mitigation and long-term fix.
   - Test surface: what the existing 3 mock-WebSocket tests in
     `main.rs:991-1238` cover, and a proposed `#[ignore]` integration
     test layout for future real-supervisor-binary coverage.
   - File:line quick-reference index.

2. **Correction in `docs/control-plane-design.md`** — replaces the
   "reload without restart" paragraph with an accurate description
   that links to the new doc. Clarifies that the `opampextension`
   is a reporting channel, not an apply channel; the supervisor is
   the apply channel.

## What's NOT in this PR

- **Real opampsupervisor binary integration test** — the proposed
  layout is in §5c of the new doc but not implemented. The test
  depends on the `opampsupervisor` binary being pinned in CI, which
  needs a separate Go toolchain decision. Marked as a follow-up
  that can land independently without touching the architecture.
- **Capability-matching fix** — the "push a KLL config to a collector
  that doesn't have KLL compiled in" gap is documented but not fixed
  in this PR. That's a controller/supervisor registration change
  tracked in `controller-optimization-problem.md`.

## Scope rationale

PR F was originally scoped as "verify / fix OpAMP config push from
controller to agents". The control-plane side is already verified
by the 3 existing integration tests in `controller/src/main.rs` and
by the manual e2e validation in commit 4b196e1. What was missing
was a written contract future contributors could refer to — especially
since the existing design doc made a claim that the code no longer
matches. Shipping docs + a test layout proposal closes that gap
without the flakiness risk of a binary-level integration test.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

working configuration between controller and otel collector

1 participant