feat(monitor): hot-reload the CDM coordinator's monitor set (no restart)

[zzylol]

Problem

The CDM monitor coordinator read streaming_config.monitors() once at boot (main.rs, into MonitorCoordinator::new). But the control plane publishes the real config slightly after boot via the /api/v1/streaming-config hot-reload POST (an ArcSwap in hot_reload_config) — which updated the query engine but not the coordinator. So a monitor published post-boot never reached the coordinator: every edge registering for it was rejected as register for unconfigured monitor — ignored, and coordinated sampling stayed dead unless the data-plane was restarted with the monitor already baked into its boot config.

This is exactly the gap that forced the Fig 9 run-side workaround (seed the monitor into the boot streaming.yaml + restart the data-plane) in ProjectASAP/ASAPCollector#503.

Fix — make the coordinator reconfigurable on the live config

• MonitorCoordinator.cfgs moves behind a std::sync::RwLock (held only for brief, non-await critical sections, so it's safe in async code).
• New MonitorCoordinator::reconfigure(specs) diffs the incoming specs against the current set and applies the delta, returning (added, changed, removed):
  • added → matchable immediately (the next register lazily builds the Monitor);
  • changed (different τ/ε/window for an existing key) → evicts the stale live Monitor so the next register rebuilds it under the new spec; edges re-register every epoch, so it self-heals within one window;
  • removed → drops both the spec and any live state;
  • unchanged monitors keep their in-flight slack-countdown state untouched. Idempotent (same specs → (0,0,0)).
• main.rs spawns a watcher that polls the hot_reload_config ArcSwap (an atomic load + Arc::ptr_eq every 2 s) and calls reconfigure only on an actual swap — so a controller-pushed monitor takes effect live, with no restart and no boot-config seed.
• MonitorConfig gains PartialEq for change detection.

Tests

Two unit tests (monitor::server::reconfigure_tests), 2/2 pass:

• reconfigure_adds_changes_and_removes_specs — add + change + remove counts, monitor_count, and idempotency.
• reconfigure_evicts_live_state_for_changed_monitor — a register materializes live state; a τ change evicts it; an unconfigured key is still rejected.

Public signatures of new/monitor_count are unchanged (the harness binary is unaffected).

Impact

Removes the boot-seed-and-restart workaround for live coordinated sampling. Pairs with ASAPCollector#503 (Fig 9) and #504 (edge), and ASAPQuery-backend#377 (per-sid f_i allocation).

🤖 Generated with Claude Code

[zzylol]

Live validation (Fig 9, 8-node CloudLab) ✅

Built a data-plane image from this branch + the per-sid f_i allocation (#377) and reran Fig 9 with the seed-boot-config + restart workaround removed — the coordinator must pick the monitor up live.

Data-plane log (no seed, no restart):

--enable-monitor-coordinator set but streaming-config has no `monitors:` yet — ...
CDM monitor coordinator hot-reloaded monitors from pushed streaming-config added=1 changed=0 removed=0

The watcher applied the controller-pushed cms_point monitor ~1.5 s after boot.

Result — identical to the boot-seed run, with the workaround gone:

edge	series (rate)	learned	granted p
hot	400	✓ s0	0.0010
med	80	✓ s0	0.0022
quiet	16	✓ s0	0.0049

• hot-reload events: 1, unconfigured rejections: 0
• grants differentiate 1 : 2.2 : 4.9 vs √rate law 1 : 2.24 : 5

The coordinator now picks up a controller-pushed monitor live with no restart and no boot-config seed — confirming this removes the ASAPCollector#503 Fig 9 workaround.