Skip to content

feat(controller-client): DcControllerClient adapter for DC /api/v1/plan - #15

Merged
zzylol merged 1 commit into
mainfrom
feat/dc-controller-client-adapter
Apr 16, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/dc-controller-client-adapter

Conversation

@zzylol

@zzylol zzylol commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds DcControllerClient next to the existing HttpControllerClient. It implements the same ControllerClient trait but translates each capability-miss notification into a DataCollector QuerySpec and POSTs to DC's /api/v1/plan endpoint. Gated by a new --controller-protocol={generic,datacollector} CLI flag (defaulting to generic, so existing deployments are unchanged).

Motivation

PR G (#11) wired the backend to notify "the controller" on capability misses via the ControllerClient trait. The concrete impl shipped was HttpControllerClient, which POSTs a backend-native CapabilityMissPayload JSON to whatever URL is given via --controller-endpoint. That payload looks like:

{
  "kind": "capability_miss",
  "metric": "http_requests_total",
  "statistics": ["Sum"],
  "data_range_ms": 60000,
  "grouping_labels": ["service"],
  "spatial_filter_normalized": "status=\"200\""
}

However, the DataCollector controller/ binary's plan endpoint (POST /api/v1/plan) expects a completely different shape — a QuerySpec:

{
  "metric_name": "http_requests_total",
  "aggregations": ["frequency"],
  "group_by_labels": ["service"],
  "time_window": "60000ms",
  "accuracy_sla": 0.95
}

Result: if you point --controller-endpoint at DC's /api/v1/plan, every notification gets a 422 because DC can't deserialize CapabilityMissPayload as QuerySpec. The field names, nesting, and semantics are all different.

This PR introduces DcControllerClient — a translation adapter that converts QueryRequirements → DC QuerySpec on the wire. It's selected via a new --controller-protocol CLI flag:

--controller-protocol values

Value Behavior When to use
generic (default) POSTs the backend-native CapabilityMissPayload JSON directly. No translation. For a future endpoint that natively accepts the backend's payload shape (e.g. a hypothetical POST /api/v1/capability-miss), or any third-party controller that speaks the backend's format. Today nothing actually consumes this format — it exists to preserve the original PR G behavior as the default so existing --controller-endpoint users aren't broken.
datacollector Translates QueryRequirements → DC QuerySpec and POSTs to /api/v1/plan. Use this when pointing at a real DataCollector controller. This is the only mode that actually works end-to-end today.

In practice: if you pass --controller-endpoint=http://controller:8080/api/v1/plan without --controller-protocol=datacollector, the backend will POST the wrong JSON shape and get 422s from DC.

Translation rules

QueryRequirements field QuerySpec field Notes
metric metric_name Direct copy
grouping_labels.labels group_by_labels Direct copy
data_range_ms time_window Formatted as "{N}ms" (e.g. 60000"60000ms"); defaults to "60s" when None (instant queries with no range)
statistics: [Quantile] aggregations: ["quantile"] DC planner picks KLL/DDSketch family
statistics: [Cardinality] aggregations: ["cardinality"] DC planner picks HLL
statistics: [Count|Sum|Increase|Rate|Min|Max|Topk] aggregations: ["frequency"] DC planner picks CountMin/CountSketch family
(fixed) accuracy_sla: 0.95 Configurable via DcControllerConfig; DC rejects values outside [0,1]

The accuracy_sla and default time window are both on DcControllerConfig, so they can be tuned later without another CLI flag.

What happens after the POST

When DcControllerClient POSTs to /api/v1/plan, the DC controller:

  1. Parses the QuerySpec via its Analyzer
  2. Runs cost-model planning (bandwidth, delta vs. full, sketch type selection)
  3. Stores the resulting plan (retrievable via GET /api/v1/plan/:metric)
  4. If OpAMP agents are connected, pushes updated collector YAML configs to agent-role and backend-role collectors

The backend's fire-and-forget call completes as soon as the POST returns 200. The query that triggered the miss has already returned (via the §5.2 fallback). Future queries benefit once the plan propagates.

Tests

Six new unit tests in drivers/query/controller_client.rs:

  • dc_build_query_spec_projects_statistics_to_aggregations — happy-path field projection
  • dc_build_query_spec_defaults_time_window_for_instant_query — instant-query fallback
  • dc_build_query_spec_maps_quantile_statistic — quantile → "quantile"
  • dc_build_query_spec_maps_cardinality_statistic — cardinality → "cardinality"
  • dc_client_round_trips_against_mock_plan_endpoint — full POST to an axum mock server asserting every QuerySpec field is present and correctly shaped
  • dc_client_reports_non_success_status — clean error propagation on 4xx/5xx

Plus live validation against a real DC controller/ binary at :18080 (Stage B's actual validation goal):

curl /api/v1/query?query=sum(nonexistent_metric)
  → backend SimpleEngine finds no compatible aggregation
  → spawn_capability_miss_notify (fire-and-forget tokio::spawn)
  → DcControllerClient translates QueryRequirements → QuerySpec
  → POST http://localhost:18080/api/v1/plan
  → DC controller handle_plan runs cost model, stores plan
  → GET /api/v1/plan/nonexistent_metric returns:
    {"metric":"nonexistent_metric","sketch_type":"countsketch","valid_until":"..."}

Test plan

  • cargo test -p query_engine_rust --lib controller_client → 11 passed
  • cargo fmt / cargo clippy -p query_engine_rust -- -D warnings clean
  • End-to-end capability-miss → DC plan creation verified against real controller binary
  • Default CLI behavior unchanged — protocol=generic remains the default, existing --controller-endpoint users are not affected

Depends on

Stacks cleanly on #14 (the CountMinState envelope decoder fix). Can merge in either order; they touch different files.

🤖 Generated with Claude Code

Adds `DcControllerClient` next to `HttpControllerClient` in
`drivers/query/controller_client.rs`. It implements the same
`ControllerClient` trait but translates each capability-miss
notification into a DataCollector `QuerySpec` and POSTs to DC's
`/api/v1/plan` endpoint. Gated by a new `--controller-protocol=
{generic,datacollector}` CLI flag (defaulting to `generic` so existing
deployments are unchanged).

Translation rules:
  - QueryRequirements.metric           -> QuerySpec.metric_name
  - QueryRequirements.grouping_labels  -> QuerySpec.group_by_labels
  - QueryRequirements.data_range_ms    -> QuerySpec.time_window ("{N}ms"),
                                          falls back to "60s" when None
  - QueryRequirements.statistics       -> QuerySpec.aggregations:
        Quantile       -> "quantile"
        Cardinality    -> "cardinality"
        Count|Sum|...  -> "frequency"
  - accuracy_sla                       -> 0.95 (configurable)

Tests: five new unit tests (field projection, instant-query fallback,
quantile/cardinality mapping, round-trip against an axum mock server
asserting every field is present, and non-success HTTP status
handling). Plus one live validation on a real DC controller at
:18080, which is the whole point of Stage B:

  curl /api/v1/query?query=sum(nonexistent_metric)
    → backend SimpleEngine capability miss
    → DcControllerClient translates and POSTs
    → DC controller handle_plan runs cost model
    → GET /api/v1/plan/nonexistent_metric returns a plan with
      sketch_type=countsketch

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit b24b06d into main Apr 16, 2026
5 of 6 checks passed
@zzylol
zzylol deleted the feat/dc-controller-client-adapter branch April 16, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant