Skip to content

Retire the collector-facing emission pipeline - #719

Closed
zzylol wants to merge 13 commits into
refactor/703-remove-legacy-plan-endpointsfrom
refactor/703-remove-collector-emit
Closed

zzylol wants to merge 13 commits into
refactor/703-remove-legacy-plan-endpointsfrom
refactor/703-remove-collector-emit

Conversation

@zzylol

@zzylol zzylol commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

#703 item 18, final deletion step. Stacked on #718#715#713 — review those first.

−22,510 / +607. control_plane goes from ~51k lines to 29,070; main.rs from 3,500+ to 1,108.

Why this lands as one commit

With the legacy planning endpoints gone (#718), the only remaining entry into the collector config chain was GET /api/v1/collector-config/agent, the HTTP config provider for collector bootstrap. Removing it retires the whole chain, because every part exists to serve that entry:

workloads.yaml startup pre-population
  → Analyzer / DeploymentPlanCompiler
  → PlanStore / WorkloadStore
  → Replanner (expiry + SLA-violation + backend-repost tickers)
  → Scraper (SLA violation detection)
  → OpAMP on-connect / on-disconnect hooks
  → colored_dag stage allocation + three-stage split
  → per-runtime YAML/TOML emitters

Splitting it would leave intermediate states that do not compile.

Deleted

lines
emit/stage_config.rs 7,410
physical/colored_dag/ 2,572
replan.rs 1,350
emit/backend_push.rs 1,043
physical/workload_planner.rs 980
pipeline.rs 843
deployment_cost/{delta,sketch_capability} + scoring ~1,500
emit/{agent,otap,telegraf}.rs 1,857
store/, monitor/, physical/{plan_cache,stage_split,topology}.rs ~1,600
plus the tests covering them ~3,400

Also gone: the USE_TYPED_STAGE_SPLIT kill switch (#703 item 12) — its "off" branch was the legacy emitter; RealizationProvider::stages (item 10), which had no callers; physical/topology.rs (item 11), a pure re-export.

Extracted, not deleted

Two pieces the ASAPPlanner path depends on:

  • physical::backend_stage (87 lines) keeps BackendStageConfig, BackendAggregation, BackendReadout, AggregationInput. PhysicalCompiler builds these; backend_plan::from_stage_config consumes them. Their collector-side neighbours (EdgeStageConfig, GatewayStageConfig, ExportTarget, StageId) do not come along.
  • emit::backend_wire (419 lines) keeps the storage-routing classifier and build_backend_aggregation_json. That builder stays the single source of the aggregation wire shape, so BackendPlan materializations and the JSON format keep one PolicyFingerprint identity space — the invariant from_stage_config's module doc calls load-bearing.

deployment_cost keeps only what still has consumers: online (behind /api/v1/cost-model), tco (behind /api/v1/tco), wire (post-ASAP cost model). SketchCosts and the benchmark table move into online, its only remaining caller.

Behaviour change

/metrics no longer exposes asap_active_plan_id. The gauge read PlanStore, which was legacy plan history and is gone. Every other asap_runtime_* gauge is unchanged.

Resulting surface

Production routes are now the ASAPPlanner path only:

/api/v1/physical-plan/cost-manifests
/api/v1/metricsql/physical-plan/cost-manifests
/api/v1/physical-plan/compile-and-publish
/api/v1/metricsql/physical-plan/compile-and-publish
/api/v1/clickhouse-plan/compile-and-publish
/api/v1/cost-model
/api/v1/tco
/metrics

AppState is down to opamp, online_store, runtime_samples, active_summary_catalog, backend_client. OpampServer is constructed bare and retains only publish_collector_plans, which the physical path uses.

Testing

  • cargo +1.98.0 check -p control_plane --all-targets — clean, no warnings.
  • cargo +1.98.0 test -p control_plane --lib — 387 passed, 0 failed (705 → 387; the 318 covered deleted code).
  • cargo +1.98.0 test -p data_plane --lib — 1193 passed, 0 failed, unchanged.

🤖 Generated with Claude Code

zzylol and others added 8 commits September 13, 2026 16:13
Six endpoints on the legacy planning API have no caller anywhere in the
workspace — not the data plane, not the process e2e tests, not the docs:
`/plan/auto`, `/plan/pareto`, `/plan/:metric/rollback`,
`/plan/:metric/diff`, `/agents`, and `/config/:metric`.

Removing them retires the autonomous allocation chain that existed only
to serve `/plan/auto`: `epsilon_alloc::build_auto_plan` and its knob
allocation, which pulled in `query_planning`, and through it
`sketch_selection` and `asap_tier_implement`.

`epsilon_alloc` itself stays. `derive_sample_p` and `split_budget` are
live on the physical path (`physical/compiler.rs` and `types.rs`), so
the module keeps those two and loses the rest.

`POST /api/v1/plan`, `GET /api/v1/plan/:metric` and `/api/v1/cost-model`
are untouched: the data plane's capability-miss notifier and config
fetcher call the first two, and `control_plane/tests/component_process_e2e.rs`
uses the third as its readiness probe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ss loop

`POST /api/v1/plan` and `GET /api/v1/plan/:metric` have no working
consumer. The apparent ones do not function:

* The data plane's capability-miss notifier POSTs a
  `CapabilityMissPayload` — `{kind, metric, statistics, data_range_ms,
  grouping_labels, spatial_filter_normalized}`. `handle_plan` accepts a
  `QuerySpec`, whose `accuracy_sla` carries no serde default, so every
  notification fails deserialization with `missing field accuracy_sla`.
  No control-plane handler accepts that payload at all. The non-2xx
  becomes an `Err` that `spawn_capability_miss_notify` logs at WARN and
  drops, so the loop has been a silent no-op.

* `control_plane_client::config_fetcher::ControlPlaneClient`, which polls
  `GET /plan/:metric` and posts `/plan`, is never constructed outside its
  own module.

That left `control_plane/tests/component_process_e2e.rs` as the only
real caller, and it exercises the legacy path itself.

Removing the endpoints therefore needs no rewrite. The three gaps that
would have made porting `handle_plan` onto `PhysicalCompiler` awkward —
`QuerySpec::query_string` being optional, the absent
`LifecyclePlanningInput` cost evidence, and the absent
`window_implementations` — do not have to be closed.

`AppState` loses `analyzer`, `planner`, `store`, `scraper` and
`backend_routing_cache`, all unread once the handlers are gone.

The legacy planner modules behind them stay for now: the startup
`workloads.yaml` pre-population still runs `Analyzer` and
`DeploymentPlanCompiler` to fill `workload_store`, which
`handle_bootstrap_agent_config` reads. That chain retires with the
collector-facing emitters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the refactor/703-remove-legacy-plan-endpoints branch from 7305832 to a1749b7 Compare September 13, 2026 22:37
zzylol and others added 3 commits September 13, 2026 16:38
With the legacy planning endpoints gone, the only remaining entry into the
collector config chain was `GET /api/v1/collector-config/agent`, the HTTP
config provider for collector bootstrap. Removing it retires the whole
chain in one piece, because every part of it exists to serve that entry:

  workloads.yaml startup pre-population
    -> Analyzer / DeploymentPlanCompiler
    -> PlanStore / WorkloadStore
    -> Replanner (expiry + SLA-violation + backend-repost tickers)
    -> Scraper (SLA violation detection)
    -> OpAMP on-connect / on-disconnect hooks
    -> colored_dag stage allocation and three-stage split
    -> per-runtime YAML/TOML emitters

Splitting it would leave states that do not compile, so it lands together.

Two pieces are extracted rather than deleted, because the ASAPPlanner path
depends on them:

* `physical::backend_stage` keeps `BackendStageConfig`, `BackendAggregation`,
  `BackendReadout` and `AggregationInput`. `PhysicalCompiler` builds these and
  `backend_plan::from_stage_config` consumes them. The collector-side
  neighbours (`EdgeStageConfig`, `GatewayStageConfig`, `ExportTarget`,
  `StageId`) do not come along.
* `emit::backend_wire` keeps the storage-routing classifier and
  `build_backend_aggregation_json`. That builder stays the single source of
  the aggregation wire shape, so `BackendPlan` materializations and the JSON
  format keep one `PolicyFingerprint` identity space.

`deployment_cost` keeps only what still has consumers: `online` behind
`/api/v1/cost-model`, `tco` behind `/api/v1/tco`, and `wire` for the
post-ASAP cost model. `SketchCosts` and the benchmark table move into
`online`, its only remaining caller.

Behaviour change to note: `/metrics` no longer exposes `asap_active_plan_id`.
The gauge read `PlanStore`, which was legacy plan history.

AppState is down to opamp, online_store, runtime_samples,
active_summary_catalog and backend_client.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the refactor/703-remove-collector-emit branch from 49ef7db to 354ef73 Compare September 14, 2026 00:23
@zzylol
zzylol force-pushed the refactor/703-remove-legacy-plan-endpoints branch from a1749b7 to ebb601c Compare September 14, 2026 00:27
@zzylol
zzylol force-pushed the refactor/703-remove-legacy-plan-endpoints branch from ebb601c to 9e2117d Compare September 14, 2026 00:56
@zzylol

zzylol commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Superseded. main moved substantially while this was open (#712, #714, #717 — the last renamed PhysicalCompiler, BackendLocalPlanningSnapshot, compile, planning_request and aggregation_configs), and re-applying the deletion on the current tree was cleaner than replaying this branch through rename-heavy conflicts. Replacement: refactor/703-collector-emit-v2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant