mvp v6 phase B: controller plan -> per-stage runtime config emitter - #297
Merged
Merged
Conversation
Per-stage emitter that turns each StageConfig variant produced by
ThreeStageEmitter into the wire bytes the executor consumes:
* emit_edge_yaml — OTel YAML for the edge agent (OTLP
receiver, per-sketch processor block, OTLP exporter to gateway).
* emit_gateway_yaml — OTel YAML for the gateway (OTLP receiver,
family-specific *merge processor block, OTLP exporter to backend).
* emit_backend_config_json — JSON document matching the
ASAPQuery-backend POST /api/v1/streaming-config surface.
8 new unit tests cover round-tripping aggregations / readouts,
processor / pipeline key consistency, batch-vs-window mode, and
sketch-kind → backend-type mapping. Pure transformation; no I/O.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(Phase B step 2) Drop the #[allow(dead_code)] gates on split_typed_three_stage, typed_stage_split_enabled, and bind_workload_typed — Phase B's main::handle_plan wiring (next commit) is their first caller, so the suppressions are stale. Also points the split_typed_three_stage docstring at the new config::stage_config emitter module so future readers can follow the typed L5 pipeline end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…TYPED_STAGE_SPLIT (Phase B step 3)
When USE_TYPED_STAGE_SPLIT=1 is set, after the legacy plan push,
handle_plan additionally:
1. binds the workload to a SketchExpr via planner::rules::bind_workload_typed,
2. runs the typed L5 path via planner::stage_split::split_typed_three_stage,
3. for each (StageId, StageConfig) entry, calls the matching emitter
in config::stage_config and routes the output:
- Edge: pushes the YAML to OpAMP role-Agent.
- Gateway: logs the YAML at info! (no AgentRole::Gateway yet —
Phase C adds it + a proper push).
- Backend: logs the JSON at info! (Phase C wires the streaming-
config POST through the existing BackendClient).
Existing controller behaviour is unchanged unless the env var is set;
the legacy untyped path still runs in both branches so nothing
regresses for callers that don't opt in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 6, 2026
zzylol
added a commit
that referenced
this pull request
May 7, 2026
…314) The controller's L1→L5 pipeline (PRs #273-279) plans sketch placement, and Phase B (PR #297) added `emit_backend_config_json` for the backend's StreamingConfig. The per-metric `BackendStorageRouting` table that the backend's HTTP query handler consults on every PromQL query, however, was still hand-authored YAML at `deploy/configs/backend-storage-routing.yaml` — two sources of truth, drift between the controller's stage-split decisions and the routing config, and manual edits required when the workload changes. Phase α makes the controller THE planner: it emits a `BackendStorageRouting` JSON document as part of every plan emit, the backend hot-loads it on push, and routing flows via OpAMP push instead of YAML edits. Concretely: * `config::stage_config::emit_backend_storage_routing(metric_plans)` — for each `(metric, &BackendStageConfig)` pair the controller has planned this cycle, emit a `metrics:` row with the per-shape engine routing list. Classification rules are sourced from the L4 `sketch_algebra` outputs landing at the backend (DDSketch / KLL → warm tier for `quantile`; HLL → warm tier for `count`; Count-Sketch → warm tier for `topk`; CMS → warm tier for `count` / `point_count`). Archive-eligible shapes — `histogram_quantile`, `delta`, `deriv`, `absent`, `rate_post_hoc`, plus `topk` / `count` when no sketch claims them — are emitted on a `thanos_archive` target with an explicit `applies_to_query_shape` filter. Warm-tier slot stays the default (no filter) so unanticipated shapes route to warm rather than failing through to the archive's first-target fallback. * `BackendClient::post_storage_routing_json` — sibling of `post_streaming_config_json`. Rewrites the configured streaming-config endpoint URL's path component from `/api/v1/streaming-config` to `/api/v1/storage_routing` so operators only configure one `CONTROLLER_BACKEND_ENDPOINT` and both pushes land at the same backend host. * `main::handle_plan` — when `USE_TYPED_STAGE_SPLIT=1` and the typed L5 emitter produced a `BackendStageConfig`, also call `emit_backend_storage_routing` and POST it via the shared `BackendClient`. Same fire-and-forget contract as the existing `streaming-config` push: errors logged at WARN, the next replan cycle retries. Tests: 11 new tests (7 emitter unit / snapshot tests, 2 URL-derivation tests, 2 mock-backend integration tests). Snapshot test in `storage_routing_three_metric_snapshot_stable` pins the exact JSON shape for a DDSketch + HLL + Count-Sketch plan so accidental schema drift surfaces immediately. Pre-existing 10 controller failures unchanged (493 pass, was 482+). Phase α is gated behind `USE_TYPED_STAGE_SPLIT=1` (the existing typed L5 path). Operators can still hand-author `deploy/configs/backend-storage-routing.yaml` for dev / standalone deployments — the backend falls back to the static YAML when no JSON has been pushed yet (Part B, separate PR). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…W config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…W config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…ativetodelta + seed) (#398) * fix(deploy): cumulativetodelta upstream of agent routing (paired with ASAPQuery-backend #299) Static bootstrap counterpart to ASAPQuery-backend PR #299. Without this static fix, the asap-otel agent's BOOTSTRAP config (used before the controller's OpAMP push lands) lacks cumulativetodelta and sends cumulative-temporality Counter values to the backend's SumAccumulator — triggering the ~300× per-window quadratic blowup that bug #298 documented. After OpAMP push (typed-stage-split emit from #299), the running config has it; this static yaml just keeps the pre-OpAMP boot window correct too. `match_type: strict` keeps the processor a no-op for gauges (http_requests_total_latency_ms etc. — quantile workloads unaffected). b0/b1 baseline configs are intentionally untouched: VictoriaMetrics expects cumulative (Prometheus convention). Closes the ASAPCollector half of #298. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(demo): accuracy-validation harness — EXPORTER_SEED + baseline PRW config + per-series quantile Bundle of supporting changes for apples-to-apples accuracy validation of asap-tier vs baseline (b0/b1 VictoriaMetrics). Lands alongside the quantile-shape + cumulativetodelta engine fixes (ASAPQuery-backend PR #297 + #299, ASAPCollector cumulativetodelta commit above). Changes: 1. **fake-exporter/main.go** — EXPORTER_SEED env var seeds per-series PRNG (math/rand) deterministically so all 3 arms in a sequential run emit identical latency value sequences. Without it, cross-arm sampling noise masked DDSketch ε behavior in the accuracy report (b0 vs b1 differed by 0.04-0.4% just from random draws, not engine error). Default seed of 42 in run_demo.sh; back-compat when unset (auto-random, original behavior). Per-series PRNG uses `seed ^ hash(EXPORTER_PRODUCER_ID) ^ (seriesIdx+1)*prime` so distinct producers + series get distinct sequences. 2. **b0/b1 PRW exporter config — add_metric_suffixes: false** — VictoriaMetrics' OTLP→PRW path appends `_milliseconds` to metric names with `WithUnit("ms")` annotation (Prometheus naming convention). Asap tier preserves the original name. Without this fix, the accuracy comparison probe queries `http_requests_total_latency_ms` land on different metric names per tier — apples-to-oranges. 3. **mvp-workload.yaml** (singlenode + multinode) — dropped `grouping_labels: [zone]` from quantile-metric entries (http_requests_total_latency_ms, request_size_bytes). Per-series DDSketch / KLL sketches preserve PromQL's per-series semantics so `quantile_over_time(...)` returns comparable per-series rows in both asap and baseline. Counter-metric entries retain grouping_labels: [zone] (Sum aggregation is mergeable; no semantic asymmetry there). The KLL override on the latency entry is also tracked here for the DDSketch-vs-KLL accuracy comparison documented in mvp_smoke_test_findings.md. 4. **run_demo.sh** — pass EXPORTER_SEED=42 (default) to all producers. Accuracy results post all fixes (multinode all-arms): - p50 quantile: 0.4-0.6% rel-err vs baseline ✅ - p99 quantile: 11-12% rel-err (DDSketch ε + temporal-window variance) - max by (zone) (quantile_over_time(...)): 8-9% rel-err (DDSketch) - sum-by-zone/rate/topk: post-#299 returns delta-window semantics (not directly comparable to baseline cumulative as raw numbers, but bug-correct; ratio drops from ~300× to ~1× when normalized). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase B of the MVP v6 plan, per the Phase A audit.
The controller's
StageAllocator+ThreeStageEmitteralready produce a structuredHashMap<StageId, StageConfig>(incontroller/src/stage_split/emitter.rs), butsplit_typed_three_stagewas#[allow(dead_code)]with no callers. This PR adds the per-stage emitter that materialises eachStageConfigvariant into wire bytes and wires it intohandle_planbehind a feature flag.New module —
controller/src/config/stage_config.rsThree pure functions, modelled after the existing
config::agent/config::backendpatterns:emit_edge_yaml(&EdgeStageConfig, opamp_endpoint) -> Result<String>— OTel YAML for the edge agent: OTLP receiver on:4317/:4318, one processor perEdgeSketchProcessor(kllprocessor,ddsketchprocessor,hllprocessor,countminsketchprocessor,countsketchprocessor), OTLP exporter to gateway. Window / label-filter / aggregation_id / family-specific params are threaded through from the typed config.emit_gateway_yaml(&GatewayStageConfig, opamp_endpoint) -> Result<String>— OTel YAML for the gateway: OTLP receiver on the configured port, family-specific merge processor (kllmerge,ddsketchmerge,hllmerge,countminmerge,countsketchmerge) perGatewayMergeProcessor, OTLP exporter to backend.emit_backend_config_json(&BackendStageConfig) -> Result<serde_json::Value>— JSON document matching the ASAPQuery-backendPOST /api/v1/streaming-configAPI surface; mirrorsconfig::asapquery_backend::generate_streaming_config_yaml's shape, sourced from the typed L5BackendStageConfiginstead of aCollectionPlan.8 unit tests cover round-tripping aggregations / readouts, processor / pipeline key consistency, batch-vs-window mode, family-specific param surfaces, and
ExportTarget::Endpointpassthrough.Wiring —
controller/src/main.rsAfter the legacy plan + push,
handle_planconsultsplanner::stage_split::typed_stage_split_enabled()(USE_TYPED_STAGE_SPLIT=1). When set, it binds the workload to aSketchExpr, runssplit_typed_three_stage, and for each(StageId, StageConfig)pair calls the matching emitter:Edge→OpampServer::push_to_role(AgentRole::Agent, ...)Gateway→info!log of the YAML (noAgentRole::Gatewayexists today; Phase C adds the role + push)Backend→info!log of the JSON (Phase C wires the actualPOSTthrough the existingBackendClient)Phase A audit verdict honoured
mvp/v6-phase-b-controller-emitterbranched offorigin/main.USE_TYPED_STAGE_SPLIT=1.Notes for Phase C
USE_TYPED_STAGE_SPLIT=1(already declared asENV_USE_TYPED_STAGE_SPLITinplanner::stage_split).AgentRoleonly carriesAgent/Backendtoday. Phase C should addAgentRole::Gatewaysoemit_gateway_yamloutput can be pushed via OpAMP rather than logged.backend_client::BackendClientis currently held only by theReplanner. Phase C should plumb it ontoAppStatesohandle_plancan callclient.push_streaming_config_json(...)directly.ExportTarget::Stage(_)resolves to documented placeholders (gateway:4317,backend:4317); Phase C plumbsDeploymentConstraints::executors()for real addresses.opentelemetry-collector-contrib-patch/processor/):kllprocessor,ddsketchprocessor,hllprocessor,countminsketchprocessor,countsketchprocessorkllmerge,ddsketchmerge,hllmerge,countminmerge,countsketchmergeDDSketch,DatasketchesKLL,HLL,CountSketch,CountMinSketch— same mappingconfig::asapquery_backend::map_sketch_type_to_agg_typealready uses.Test plan
cargo test stage_config -p controller— 8 new tests pass.cargo test -p controller— 475 pass (467 baseline + 8 new); the 10 pre-existing failures (opamp::tests::push_to_*role*,api_tests::*,analyzer::tests::valid_spec, etc.) reproduce on bareorigin/mainand are unrelated.cargo check -p controllerclean.LOC: ~720 in the new emitter file (functions + 8 tests + doc), ~85 lines of
main.rswiring + planner doc updates. 5 files touched (1 new, 4 modified).🤖 Generated with Claude Code