mvp issue-46 ④ ⑥ partial — archive ground truth flowing; warm + probe routing remain - #333
Merged
Merged
Conversation
…g, warm + probe routing remain Changes: - controller typed bootstrap: extend Edge stage config with freshness probes + workload-registry archive metrics so the typed path emits gorillas3 + warm-passthrough routing (groundwork for OpAMP-on-connect future port) - thanos-store-gateway: shrink --sync-block-duration to 30s (was upstream default 3min) so demo soak window sees fresh blocks - mvp_report.py: surface real ⑥ pass/fail verdict from criterion eval, not placeholder CAPTURED/UNKNOWN rollup Verdict shift from PR #332's run: - ④ archive ground truth: 500 quantile rows archive_status=ok (was 0, all archive_miss) - ④ warm side: warm engine returns status=error for ALL 500 quantile queries — separate warm-engine pattern-matcher gap - ⑥ still UNKNOWN: agents load static placeholder via volume mount, NOT typed bootstrap; freshness-probe routing absent at agent - ① ② ③ ⑤: unchanged from PR #332 Two follow-up bugs documented: 1. Backend SimpleEngine errors on quantile_over_time and sum-instant queries (out of scope) 2. Static placeholder agent config has no routing for http_freshness_probe_* (out of scope; OpAMP-on-connect typed-config push is the durable fix)
This was referenced May 8, 2026
zzylol
added a commit
that referenced
this pull request
May 8, 2026
… pipeline (#334) `Replanner::push_config_to_agent` (and the agent-config push in `replan_metric`) called the legacy `generate_agent_config` emitter, which produces a single-pipeline DDSketch YAML with no `gorillas3` archive write, no OTTL `routing` processor, and no `metrics/warm_passthrough` pipeline. Agents that connected via the OpAMP `on_connect` callback (the post-#333 reconnection path) got that legacy YAML — so freshness-probe routing reached fresh-connect agents (criterion ⑥ on issue #46) but NOT plan-pinned reconnecting agents. Mirror `main::emit_bootstrap_typed`'s flow on the OpAMP push path: `bind_workload_typed` → `split_typed_three_stage` → pick the Edge stage → extend with freshness-probe + workload-registry archive metrics → `emit_for_runtime`. Gate it behind the existing `USE_TYPED_STAGE_SPLIT` env var so the legacy single-pipeline fallback still runs when the gate is off, matching `handle_plan` / `handle_bootstrap_agent_config`. On any typed-emit error the push falls back to the legacy emitter so it never silently drops. The bootstrap-scope demo plumbing (freshness probes + workload-registry archive metrics) is lifted out of `emit_bootstrap_typed` into a shared `config::extend_edge_with_demo_plumbing` helper; both bootstrap and replan now apply the SAME extension. The runtime defaults to `AgentRuntime::AsapOtel` since OpAMP `on_connect` doesn't surface the agent's runtime header today (matches the bootstrap default for legacy clients). Adds two unit tests in `replan::tests`: - `typed_replan_emit_includes_freshness_probe_routing` — asserts the typed path emits `gorillas3`, `routing`, `metrics/warm_passthrough`, and references `http_freshness_probe_warm`. - `legacy_path_omits_typed_processors_when_gate_off` — pins the gate semantics so a regression that always-on'd the typed path would surface here. Together with PR #329 (bootstrap GET path) this finishes the OpAMP-on-connect side of the typed emit so reconnecting agents now receive the same routed YAML as fresh-connect agents. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
May 8, 2026
Pair with ASAPQuery-backend's per-tenant `BackendStorageRouting` data-model + lookup change. Adds an explicit `tenant` field to the controller-emitted JSON document the backend's `POST /api/v1/storage_routing` endpoint consumes. * New `emit_backend_storage_routing_for_tenant(tenant, plans)` and `emit_backend_storage_routing_with_prometheus_for_tenant( tenant, plans, mode3)` entry points. Tenant lands in the emitted JSON's top-level `tenant` field; the backend's per-tenant swap routes to the named slot. * The single-tenant aliases (`emit_backend_storage_routing` / `emit_backend_storage_routing_with_prometheus`) keep emitting the `default` tenant id so existing single-tenant deploys are source-compatible. * `DEFAULT_TENANT` constant mirrors the backend's value so the controller doesn't take a build-time dep on the backend crate. * Snapshot test updated for the new `tenant: "default"` field; three new unit tests cover the per-tenant emit path. Out of scope (matches backend part 1/2): * Tenant-aware AUTH. * Per-tenant sketch state isolation. Closes the per-tenant follow-up flagged in PR #333. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…L/CountSketch/CountMinSketch) (#340) Rewrites the typed L5 edge YAML emitter so the agent loads all 5 sketch processors and the OTel `routing` *connector* (NOT the deprecated routing processor) dispatches per metric to the right sketch family — the canonical wire shape for MVP issue #46. ## Bugfix call-out: routing is a CONNECTOR, not a processor OTel collector v0.106+ removed `routingprocessor` and re-shipped the routing component as a connector under the top-level `connectors:` block. The legacy emit path placed `routing` under `processors:`, which fails confmap validation at agent boot: `error decoding 'processors': unknown type: "routing"`. This PR emits the connector-form layout the asap-otel binary's builder-config registers: receivers: { otlp } processors: { gorillas3?, batch, ddsketchprocessor, kllprocessor, hllprocessor, countsketchprocessor, countminsketchprocessor } connectors: { routing: { default_pipelines: [...], table: [...route() statements...] } } exporters: { otlp/backend, otlphttp/prometheus? } service.pipelines: metrics: (entry — receivers: [otlp], exporters: [routing]) metrics/raw_passthrough (default — receivers: [routing], processors: [gorillas3?, batch], exporters: [otlp/backend]) metrics/ddsketch_path metrics/kll_path metrics/hll_path metrics/countsketch_path (per-family — receivers: [routing], metrics/countminsketch_path processors: [gorillas3?, <family>processor, batch], exporters: [otlp/backend]) `gorillas3` runs FIRST in every per-sketch pipeline (when an archive tier is declared) so the cold-tier write happens BEFORE the family processor mutates / suffix-renames the stream — same invariant the legacy emit path enforces. ## EdgeStageConfig field Adds `metric_to_family: HashMap<String, SketchKind>` to `EdgeStageConfig` (agreed convention with the parallel planner agent). Empty map ⇒ legacy single-pipeline / Mode-3 / warm-passthrough emit shapes are preserved verbatim (backward-compat). When non-empty, the new `emit_edge_yaml_5sketch_routing` helper emits the canonical 6-pipeline layout. Metrics in the map dispatch per-family; metrics absent fall through to `metrics/raw_passthrough` (this is the home for `http_requests_total` and the freshness probes from PR #333's warm-passthrough routing). ## Field plumbing Updates every `EdgeStageConfig` constructor in `controller/src/` to initialise `metric_to_family: HashMap::new()` so existing callers keep producing the legacy shape. Adds `connectors: HashMap<String, Value>` to the structural `CollectorYaml` type with `skip_serializing_if = "is_empty"` so the legacy single-pipeline shape doesn't gain an empty `connectors: {}` block. ## Test coverage 11 new tests pin the new wire shape: * mvp46_emit_loads_all_5_sketch_processors * mvp46_routing_lives_in_connectors_not_processors (the bugfix) * mvp46_emits_all_6_named_pipelines * mvp46_entry_pipeline_routes_to_connector_not_processor * mvp46_per_sketch_pipelines_use_routing_as_receiver * mvp46_per_sketch_pipelines_have_gorillas3_first_when_archive_declared * mvp46_routing_table_dispatches_per_metric_to_correct_family * mvp46_default_pipeline_is_raw_passthrough * mvp46_warm_passthrough_routes_to_raw_passthrough_pipeline * mvp46_empty_metric_to_family_falls_back_to_legacy_emit * mvp46_composes_with_prometheus_archive_mode3 All 99 `config::*` tests pass; the 10 pre-existing failures on `main` (analyzer, intent_algebra, planner cost_model, opamp, api_tests) are untouched by this change. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three targeted fixes after agent #98 (PR #332) left ④ and ⑥ as UNKNOWN. ④'s archive ground-truth side now flows (500 quantile rows return archive_status=ok via Thanos), but the warm engine and the agent-side probe routing have separate pre-existing bugs that block both criteria from flipping to PASS in this run.
freshness_metrics(warm + archive probes) and all workload-registry metrics intoarchive_tier_metricsandwarm_passthrough_metrics. Generates a typed YAML with propergorillas3+ routing pattern. Groundwork for the OpAMP-on-connect port (replan.rs::push_config_to_agent → typed pipeline) that's the durable fix.--sync-block-duration=30s(was upstream default 3 min). Honest about the trade-off: 30s sync keeps demo soak windows seeing fresh blocks; production deployments run with the 3-min default because read-heavy archive workloads tolerate minutes-of-staleness.Verdict matrix vs PR #332
status:errorfor the same 500 queries — separate warm-engine pattern-matcher gapRemaining root causes (separate workstreams)
replay.jsonlshows status=error for 500 of 500 quantile queries and 500 of 1000 sum queries (onlysum by (zone) (rate(...))succeeds). Fixing this needs a backend warm-engine investigation, not a config change.asap-otel-agent-b6-asap-single-sketch.yamlhas only[gorillas3, ddsketch, batch]. DDSketch renameshttp_freshness_probe_warm→_quantile, so the original metric name never reaches the warm tier in queryable form. The durable fix is the OpAMP-on-connect typed-config push (today blocked onreplan.rs::push_config_to_agentstill usinggenerate_agent_config).Test plan
bash deploy/scripts/run_mvp_demo.sh --mode both— both pipelines run synchronously to teardown; report under /tmp/asap-mvp-rerun-fix/MVP_REPORT.md🤖 Generated with Claude Code