feat: DataCollector controller client — dynamic query config by zzylol · Pull Request #2 · ProjectASAP/ASAPQuery-backend

zzylol · 2026-04-04T04:37:49Z

Replaces static inference_config.yaml with dynamic config from DataCollector controller.

ControllerClient fetches plans via GET /api/v1/plan/:metric
Returns sketch type, aggregation params, staged plan
Health check + config YAML fetch endpoints

🤖 Generated with Claude Code

…Store Wire OtlpReceiver to the Store and StreamingConfig so that sketch data received via OTLP gRPC/HTTP is decoded, wrapped in SketchEnvelopeAccumulator, and inserted into SimpleMapStore as PrecomputedOutput entries. This closes the TODO at otel.rs:246. Key changes: - Add SketchEnvelopeAccumulator (precompute_operators) wrapping raw SketchEnvelope protobuf bytes with AggregateCore trait impl - OtlpReceiver now accepts Arc<dyn Store> and Arc<StreamingConfig> - Shared state passed to both gRPC MetricsServiceImpl and HTTP handler - process_otlp_request builds PrecomputedOutput + accumulator pairs from sketch gauge attributes and calls insert_precomputed_output_batch - Metric name resolved to aggregation_id via streaming config reverse lookup, with fallback to stable hash for unconfigured metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

New module drivers/controller_client.rs provides: - ControllerClient: fetches query plans from DataCollector controller API - get_plan(metric) → ControllerPlan (sketch type, aggregation, staged plan) - get_config_yaml(metric) → generated OTel YAML for the metric - health_check() → verify controller is reachable This replaces the static inference_config.yaml pattern matching: instead of defining supported queries upfront, the backend asks the controller "what's the plan for this metric?" at query time. Integration flow: 1. User queries backend with PromQL 2. Backend extracts metric name from query 3. Backend asks controller: GET /api/v1/plan/:metric 4. Controller returns sketch type + aggregation config 5. Backend uses this to dispatch to the right sketch evaluator Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Addresses two sources of drift after rebasing onto main: 1. sketch_envelope_accumulator.rs - crate rename: sketchlib_rust → asap_sketchlib - AggregateCore::get_accumulator_type now returns AggregationType enum (change from #279); use MultipleSubpopulation as the opaque variant - implement newly-required query_statistic trait method (returns Err because opaque envelopes cannot be queried without decoding) 2. controller_client.rs - DataCollector's GET /api/v1/plan/:metric now returns only a slim { metric, sketch_type, valid_until } shape; the rich plan fields (mode, aggregate_by, staged_plan, delta_decision, precompute_jobs) are only produced by POST /api/v1/plan with a QuerySpec body - split into ControllerPlanStatus (GET) and ControllerPlan (POST) - add create_plan(query_spec) that invokes the POST endpoint - get_plan now returns the slim status type Also picks up a rustfmt fold in main.rs around OtlpReceiver::new. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rebuilds the OTLP ingest path on top of the merged PR #1 + PR #2 changes so that OTLP-delivered metrics (and pre-built sketches from the DataCollector OTel collector) flow through the precompute engine's worker pool. The precompute engine performs window-aligned aggregation per StreamingConfig before writing to SimpleMapStore, matching the Prometheus / VictoriaMetrics ingest pattern. Architecture: DataCollector OTel collector → OTLP gRPC/HTTP (OtlpReceiver) → precompute engine ingest router → per-(agg_id, group_key) worker panes → StoreOutputSink → SimpleMapStore → query engine Key changes ----------- - PrecomputeEngine::new() now eagerly builds channels, router, agg_configs and a shared Arc<IngestState>. The state is exposed via a new `ingest_state()` getter so other ingest sources can push into the same worker pool without duplicating setup. - IngestState is promoted from pub(crate) to pub, alongside an `extract_group_key_for(series_key, config)` associated helper that other drivers (OTLP here, Kafka potentially later) reuse for label→group extraction. - New WorkerMessage::AccumulatorInput variant carrying a pre-built Box<dyn AggregateCore> with agg_id, group_key, and timestamp. Routed to workers by the same (agg_id, group_key) hash as GroupSamples. - GroupState gains a `sketch_panes: BTreeMap<i64, Box<dyn AggregateCore>>` alongside the existing `active_panes`, plus a new `process_accumulator_input()` worker method that: * merges incoming accumulators into the covering pane via merge_with, * honors late-data policy (Drop / ForwardToStore), * emits both raw-sample and sketch-pane outputs on window close. `flush_all` is also extended to drain sketch panes for closed windows. - New `merge_sketch_panes_for_window` helper mirrors `merge_panes_for_window`: oldest pane destructively taken, later panes cloned via `clone_boxed_core` for still-open sliding windows. - OtlpReceiver gains `with_ingest_state()` constructor. When wired to the precompute engine, OTLP requests are dispatched as GroupSamples (raw metric points) and AccumulatorInput (sketch payloads) through the engine's router. Label semantics are preserved — each point is formatted into a standard metric{k1="v1",k2="v2"} series key and run through the same extract_group_key pipeline used by the Prometheus path, so StreamingConfig.grouping_labels drives pane keying uniformly. - SketchPayload tuple is promoted to a structured `SketchPoint` that carries name, attr_name, labels, timestamp, and opaque payload bytes. Labels are now preserved through the sketch path (previously lost). - Incoming sketch payloads are wrapped in SketchEnvelopeAccumulator::from_proto_bytes (introduced by PR #2) and sent to the worker as AccumulatorInput. This preserves the full sketch state end-to-end; per-variant concrete decoding (CountMin → CountMinSketchAccumulator, KLL → DatasketchesKLLAccumulator, …) can layer on top later without changing the routing contract. - main.rs constructs the precompute engine BEFORE the OTLP receiver so the receiver can obtain an Arc<IngestState>. Without the precompute engine OTLP falls back to log-only mode. Rebase notes ------------ This commit subsumes the earlier PR #3 commit dd22379 (which wrote directly to SimpleMapStore with SumAccumulator placeholders). The original commit conflicted with PR #2's own OTLP changes after PR #2 landed on main; rather than carry two overlapping commits forward, the earlier placeholder work is replaced in-place by this coherent refactor. Verification ------------ - cargo check --all-targets: clean - cargo clippy --all-targets -- -D warnings: clean - cargo fmt --check: clean - cargo test --lib: 435 passed, 0 failed, 5 ignored Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Task #34 gap #2 of 3. #48's dispatcher activated per-segment execution for combinable stats (Count/Sum/Min/Max) but refused to run for non-combinable stats (Quantile/Topk/Cardinality/ Rate/Increase) — those still fell through to the single-agg path and saw the same data cliff the dispatcher was written to prevent. Reason: there was no way to tell the HTTP caller "this answer is Partial because the query spans a reconfigure boundary and the statistic can't be combined scalarly." Changes: - **`QueryResult::vector_with_warnings` + `warnings()` accessor.** `InstantVector` and `RangeVector` gain a `#[serde(default, skip_serializing_if = "Vec::is_empty")]` `warnings: Vec<String>` field. Default constructors (`QueryResult::vector`, `QueryResult::matrix`) stay wire- compatible — they still serialise without the field when empty, so every existing caller and snapshot test holds. - **Prometheus adapter** adds a top-level `warnings: []` on `PrometheusResponse` (also skip-if-empty), matching upstream's native API. `format_success_response` + `format_range_success_response` thread through any warnings the engine populated; absent → no field. - **Dispatcher** drops the combinable-only early return. The full loop now runs for every statistic whenever the timeline has ≥2 segments. `combine_statistic` still returns `Full(v)` for cleanly-combinable inputs and `Partial { covered, missing }` everywhere else — on Partial we accumulate the `covered` scalar (when present), flag `any_partial`, and build a human-readable warnings list with the metric, range, statistic, dropped-group count, and up to three unresolved segments (agg_id, clipped range, status, coverage). Over-three are summarised; the full set is still inspectable via `GET /api/v1/db/timeline`. 5 new unit tests: 3 in `engines::query_result::tests` covering the default-empty / with-warnings / matrix wire contract; 2 in the Prometheus adapter covering the response-side serialisation contract. 734 lib tests pass (+5), clippy clean, fmt clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

) Addresses TODO.md blocker #2 subitem #3. Queries answered by the sketch DB now carry a theoretical accuracy bound back to the client as two fields on the Prometheus HTTP response: * **A — structured `accuracy` (top-level)**: ```json "accuracy": { "epsilon": 0.008125, "delta": 0.0, "kind": "relative_cardinality", "per_segment": [...] // populated on schema-timeline crossings } ``` * **C — human-readable `infos` (Prometheus 3.0 / Grafana 11+)**: ```json "infos": ["accuracy: ε=0.008125, δ=0, kind=relative_cardinality"] ``` Both are standard Prometheus-tolerant extensions — unknown top-level fields are ignored by the upstream client/Grafana 10. A regression test confirms a stripped-down "standard Prometheus" decoder round-trips through our extended response. ## Wire shape `warnings` stays reserved for partial-result / fallback advisories (PR #49's schema-timeline dispatcher still uses it); accuracy gets its own dedicated field so the semantics don't mix. When a query crosses a schema-timeline boundary, `per_segment` lists each segment's `(agg_id, range_ms, profile)`. The top-level `profile` is the max-ε, max-δ envelope across segments — a conservative upper bound. ## Changes * `stores/sketch_db/accuracy.rs`: * `AccuracyEnvelope { profile, per_segment }` + builders (`single`, `from_segments`) * `PerSegmentAccuracy { agg_id, range_ms, profile }` * `AccuracyProfile::summary()` / `AccuracyEnvelope::summary()` emit the `infos` one-liner. * `engines/query_result.rs`: `InstantVector` / `RangeVector` carry `accuracy: Option<AccuracyEnvelope>`; new `QueryResult::{accuracy(), with_accuracy()}`. * `drivers/query/adapters/prometheus_http.rs`: `PrometheusResponse::{infos, accuracy}` fields + `with_accuracy()` builder. `format_success_response` / `format_range_success_response` thread `result.accuracy()` onto the response. * `engines/simple_engine.rs`: * `SimpleEngine::accuracy_envelope_for(agg_id)` — single- aggregation helper. * `execute_context` attaches single-agg accuracy. * Timeline dispatch builds per-segment accuracy list and attaches the multi-segment envelope. ## Tests 777 → 784 (+5 green): * `prometheus_response_carries_accuracy_top_level_and_infos_mirror` * `prometheus_response_without_accuracy_skips_both_fields` * `prometheus_response_per_segment_contains_all_segments_with_worst_case_top` * `accuracy_coexists_with_warnings_without_interference` * `promql_standard_client_can_decode_response_ignoring_extensions` clippy + fmt clean.

…rity + monotonicity sweeps Addresses TODO.md #2 sub-item #2. Paper-artifact regression guard on the `AccuracyProfile::derive` module: ## 1. Published-constant parity Each sketch family's theoretical ε at a canonical parameter matches the paper-published constant to ≥ 6 decimal places: * HLL(p=14) = 1.04/√(2^14) = 0.008125 — Flajolet 2007 * CMS(w=1000, d=3) = e/w ≈ 2.718e-3 — Cormode-Muthukrishnan 2005 * CountSketch(w=10_000) = 1/√w = 0.01 — Charikar-Chen-Farach-Colton * KLL(k=200) = 2.296/√k — Karnin-Lang-Liberty FOCS 2016 * DDSketch(α) passes through verbatim — Masson-Rim-Lee VLDB 2019 * CMS-with-heap combines CMS + retention — Metwally ICDT 2005 ## 2. Monotonicity sweeps Larger capacity monotonically tightens the bound — non-monotone sweeps indicate a broken formula. Covered: HLL over precision, CMS ε over width, CMS δ over depth, CountSketch ε over width, KLL ε over k, DDSketch ε over α, CMS-with-heap ε over heap_size. ## 3. Relative ordering Cross-family sanity: at w=10_000, CMS's ε (e/w ≈ 2.7e-4) beats CountSketch's (1/√w = 0.01), confirming the two formulas weren't accidentally swapped. ## Why no live sketch runs The backend's `asap_sketchlib` git-dep exposes a different API surface from sketchlib-bench's workspace path-dep (older commit, pre-ErtlMLE/DataInput refactor). Bridging the two inside a unit test couples to a specific dep pinning that churns. Live "measure sketch, compare to bound" sweeps live in sketchlib-bench's `sketchlib bench --metrics accuracy` output and are compared to `AccuracyProfile::derive` via an external script; this test pins the *theoretical* side of that comparison so it doesn't drift. ## Tests 788 → 801 (+13), all non-ignored, clippy + fmt clean.

…2 + #2.4) (#58) * docs(sketchdb): proofs.md — accuracy bounds + timeline / barrier / backfill theorems (ASAPQuery-backend #2.4) * test(sketchdb): accuracy empirical validation — published-constant parity + monotonicity sweeps Addresses TODO.md #2 sub-item #2. Paper-artifact regression guard on the `AccuracyProfile::derive` module: ## 1. Published-constant parity Each sketch family's theoretical ε at a canonical parameter matches the paper-published constant to ≥ 6 decimal places: * HLL(p=14) = 1.04/√(2^14) = 0.008125 — Flajolet 2007 * CMS(w=1000, d=3) = e/w ≈ 2.718e-3 — Cormode-Muthukrishnan 2005 * CountSketch(w=10_000) = 1/√w = 0.01 — Charikar-Chen-Farach-Colton * KLL(k=200) = 2.296/√k — Karnin-Lang-Liberty FOCS 2016 * DDSketch(α) passes through verbatim — Masson-Rim-Lee VLDB 2019 * CMS-with-heap combines CMS + retention — Metwally ICDT 2005 ## 2. Monotonicity sweeps Larger capacity monotonically tightens the bound — non-monotone sweeps indicate a broken formula. Covered: HLL over precision, CMS ε over width, CMS δ over depth, CountSketch ε over width, KLL ε over k, DDSketch ε over α, CMS-with-heap ε over heap_size. ## 3. Relative ordering Cross-family sanity: at w=10_000, CMS's ε (e/w ≈ 2.7e-4) beats CountSketch's (1/√w = 0.01), confirming the two formulas weren't accidentally swapped. ## Why no live sketch runs The backend's `asap_sketchlib` git-dep exposes a different API surface from sketchlib-bench's workspace path-dep (older commit, pre-ErtlMLE/DataInput refactor). Bridging the two inside a unit test couples to a specific dep pinning that churns. Live "measure sketch, compare to bound" sweeps live in sketchlib-bench's `sketchlib bench --metrics accuracy` output and are compared to `AccuracyProfile::derive` via an external script; this test pins the *theoretical* side of that comparison so it doesn't drift. ## Tests 788 → 801 (+13), all non-ignored, clippy + fmt clean.

Adds three regression tests against the `process_accumulator_input → window-close → emit_batch → per_key store` path the OTLP sketch ingest dispatcher relies on: - `test_process_accumulator_input_persists_after_window_close` — a single group emits exactly one persisted output once a sketch in a later window advances the group watermark past the prior window boundary. - `test_sketch_ingest_persists_and_query_returns_non_empty` — a real `SimpleMapStorePerKey` sees non-empty results from a `query_precomputed_output` call right after sketch ingest completes a window. This is the unit-level repro of sweep blocker #2's symptom (`No precomputed outputs found for metric: ..., aggregation_id: 1`). - `test_grouping_labels_roll_up_per_tuple_sketches` — pins the agent-emit-shape vs. backend-grouping-config invariant: backend's `grouping_labels = [zone]` collapses 5 per-`(zone,rack,node,pod)` sketches into 2 per-zone outputs (one per zone). The tests pass against `origin/main`, so all three sweep-blocker hypotheses (A-rollup, B-metric-suffix, C-persistence-regression) are disproven at the unit level. The `process_accumulator_input → per_key store` plumbing is correct given that timestamps advance enough to close windows. Live e2e remediation likely requires diagnosing event-time stagnation or a schema-barrier transition in production. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…dows (#83) When event-time stagnates (e.g. agents stamp every sketch with the same `time_unix_nano`), `flush_all`'s `+1ms` watermark advance is a no-op: `closed_windows(prev_wm, prev_wm+1)` returns empty forever, the 30s window never closes, and warm-tier queries come back empty even though `worker_process_accumulator` keeps logging — exactly the live sweep blocker #2 symptom (8000 sketch arrivals, 0 store entries, query empty). PR #82 pinned the post-window-close persistence path is correct given event-time advances; it explicitly punted this fix as "watermark- semantics design change". This PR lands it. The fix tracks each pane's wall-clock birth time in `GroupState::pane_wall_clock_starts_ms` and, in `flush_all`, force- advances `effective_wm` past `pane_start + window_size_ms` for any pane older than `window_size_ms + grace_period_ms` of WALL-CLOCK time. Event-time-driven closure remains the primary path; wall-clock is fallback. Monotonicity is preserved — the fallback only ever pushes the watermark forward. `PrecomputeEngineConfig::wall_clock_grace_period_ms` (default 5_000ms, matching the existing `allowed_lateness_ms` default) tunes the grace period; set to `<= 0` to opt out and keep strict event-time-only semantics. For testability, `Worker` carries an injectable `now_ms_fn` (default `SystemTime::now`-backed). The new `wall_clock_fallback_closes_idle_window` test injects a fake clock and pins the fix in milliseconds rather than needing `std::thread::sleep(35s)`. The 3 PR #82 pin tests still pass (regression baseline). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ter (#114) `asap-query-engine` ships with `--controller-endpoint` for capability- miss notifications back to the controller (criterion #2 reverse channel). The MVP overlay sets `ASAP_CONTROLLER_URL` in `deploy/docker-compose/base.yml`, but `clap` only read the flag — not the env var — so the backend silently never notified the controller. Added `env = "ASAP_CONTROLLER_URL"` to the arg attribute so the existing compose env var is honored. Also generalized `routing/freshness_probe_cache.rs` so any metric whose name *contains* `freshness_probe_` is captured by the RAM short-circuit (was: literal prefix `http_freshness_probe_`). User-extensible probe families (e.g. `latency_freshness_probe_*`) now hit the cache without a code change. The MVP demo's three canonical probes (`http_freshness_probe_{raw,warm,archive}`) still match — verified by the existing `is_freshness_probe_matches_three_demo_spellings` test. Verification: - `cargo build --manifest-path asap-query-engine/Cargo.toml`: clean - `cargo test --lib freshness_probe_cache`: 8 passed Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ath (Phase 4 + 5 + 6 finish) (#122) Lights up the centralized sid resolver and the new SketchIndex on the live ingest+query path. Companion to refactor #2 (#121) which built the SketchIndex storage substrate. ## Phase 4 finish - OTLP receive (`route_modified_otlp_sketches_to_precompute`): per-DP sid resolution gate. Three cases: (sid=0, attrs present) → resolver mints fresh sid via compute-or-mint cache (sid != 0, attrs present) → trust attrs; backend-resolved value wins. Mismatch → push old sid to `unknown_series_ids` for sender-side eviction. (sid != 0, attrs empty) → resolver.is_known(sid) lookup; miss → push to `unknown_series_ids`, drop DP. - `route_modified_otlp_sketches_to_precompute` now returns `Vec<u64>` of unknown sids; both `MetricsServiceImpl::export` and `handle_otlp_http` stamp it into `ExportMetricsServiceResponse. unknown_series_ids` (the field already existed on the wire from Phase 2, was Vec::new() placeholder). - New `ResolveSeriesIDs` gRPC service in `metrics_service.proto`: bulk pre-resolution endpoint for senders that have an attribute-only batch ready before the next Export. Reuses the existing `SeriesAssignment` shape; idempotent end-to-end. ## Phase 5 finish - `IngestState` now carries `Arc<SeriesIdResolver>` + `Arc<SketchIndex>`. Both allocated once in `main.rs` and threaded through `PrecomputeEngine::new` so the OTLP receiver, the precompute engine, and `SimpleEngine` all share the same instances. - After sid resolution, on first-seen sid the OTLP path materializes `SketchInstanceMetadata` (capability inferred from sketch kind: DDSketch/KLL → QuantileApprox, HLL → CardinalityApprox, CountSketch/CountMin → FrequencyTopk; sketch_config from the parent-container fields lifted in Phase 2; accuracy via `AccuracyBound::from_config`) and registers it. Per-DP: `append_sample(sid, label_values, (start_ms, end_ms), state)`. - `ModifiedOtlpSketchDp` extended with `series_id`, `start_time_unix_nano`, and `container_config: SketchConfig`. The five typed-DP match arms thread these from the OTLP wire shape. - New helpers: `sketch_kind_handle_for(&dp)` and `encoding_to_handle(i32) -> SketchEncoding`. - Legacy `WorkerMessage::AccumulatorInput` write site annotated `// DEPRECATED:` — kept as fallback during validation, removed after warm-tier carries traffic in production. - `SimpleEngine` got an optional `sketch_index: Option<Arc<SketchIndex>>` field + `with_sketch_index` builder. `QueryEngine::execute` now pre-classifies the query's sid set: 1. Parse PromQL → extract metric name + label-matcher KEY set. 2. `index.instances_matching(metric, required_keys)` → candidate sids. 3. For each: `index.classify(sid)`: - all `Hit` → fall through to handle_query (legacy path; the per-Capability sketch reducer over `query_range` is a follow-up — see TODO in the engine). - any `Ghost` or `Unknown` → return `EngineError::CapabilityMiss(SketchWarmTier, ...)`. - empty match → likewise CapabilityMiss. - New `SketchIndex::instances_matching(metric, required_keys)` helper: filters instances where metric_name matches and required_keys is a subset of the instance's `group_by_keys` (over-approximation; archive failover catches the rest). ## Phase 6 finish - The existing `EngineRouter::execute` already retries on `CapabilityMiss` against the next engine in priority order, so the Ghost/Unknown branch falls through to the archive engine (Thanos forward) automatically. No router change needed — `CapabilityMiss(SketchWarmTier, ...)` IS the trigger. ## Deferred (TODOs at call sites) - Per-Capability sketch reducer over `SketchIndex.query_range` for the all-Hit case (today still flows through the legacy `SimpleMapStore` path). Follow-up PR. - Phase 6 hybrid stitch (warm covers `[t0..t1']`, archive covers `[t1'..t1]`, concatenate). Requires `QueryResult` to carry timestamp coverage metadata, which it doesn't today. Deferred. - Legacy `WorkerMessage::AccumulatorInput` aggregation_id-keyed write path: marked `// DEPRECATED:` only. Removal is a separate cleanup PR after warm-tier validation. ## Build + test - `cargo build --release -p query_engine_rust` clean (warnings only). - 7 net-new tests pass: drivers::ingest::otel::sid_resolution_tests::* (3 tests) engines::simple::engine::warm_tier_classify_tests::* (3 tests) plus 1 small expansion in series_resolver tests - 902/902 tests pass for code touched by this PR. The 32 pre-existing failures (datafusion / persistence / schema_timeline_dispatch) were red on main before this branch existed and are unrelated to warm-tier wiring. Co-authored-by: zz_y <zz_y@node0.zz-y-304941.softmeasure-pg0.clemson.cloudlab.us> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Post-M2.3 reorg #2 of 8. Moves `sketch_db/store/persistence/` → `sketch_db/persistence/` so the disk-persistence layer sits as a top-level sibling of `store/`, `data/`, `schema/`, etc. The previous nesting (`store/persistence/`) implied the persistence layer belonged to the store; in fact it's a parallel concern that any sid-keyed store wires into via the `EpochSource` trait. `store/mod.rs` re-exports `pub use sketch_db::persistence` so legacy paths (`store::persistence::*`) keep compiling. Canonical path is now `sketch_db::persistence::*`. 783/783 lib tests pass. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Schema retirement #2 of 5. The `/api/v1/db/timeline` handler now reads from `state.sketch_index` (always attached) via `sketch_db::query::timeline::timeline_for_metric` (PR #183) instead of `state.schemas.timeline_for_metric`. User-visible behavior: - Endpoint always works (no more 503 when schema-registry isn't wired — the sid catalog is always present). - `agg_id` field carries a stable content-derived signature id (xxh64 of `metric + agg_kind + group_by_keys`) instead of the controller-emitted `agg_id` (which is gone after M2.2 / PR #152). - Segments now reflect the sid catalog directly; reconfigure semantics propagate once the next sub-PR (lifecycle reconcile) lands. Test impact: - `test_get_timeline_without_registry_returns_503` rewritten to `..._with_no_sids_returns_empty_200` matching the new semantics. - `test_get_timeline_returns_segments_after_reconfigure` ignored (uses POST /streaming-config → SchemaRegistry::reconcile, which doesn't yet propagate to the sid catalog; re-enabled in the next sub-PR with the lifecycle reconcile). 789/792 lib tests pass (3 ignored — 2 pre-existing + this one). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…llas3 bucket (B1 downstream bundle) (#279) Three follow-ups to ASAPCollector#383 (agent OpAMP-apply) and companion PRs, bundled because they all touch control_plane/src/emit/stage_config.rs. Fix 1 (Issue #2): thread X-Agent-ID into opamp emit The controller's emitted opamp extension block didn't include `headers: X-Agent-ID: <id>` -- so when the agent applies a controller-pushed config and Docker restarts it, the reconnected agent has no agent-id and the controller's OpAMP server can't identify it. Empty `/api/v1/agents` after restart. Thread agent_id from replan / main into the three emit functions; add to each opamp block. Per-agent call sites (`push_config_to_agent`, the per-agent re-emit loop inside `replan_metric`, and the bootstrap GET path when `pinned_agent_id` is set) thread the real agent id. Broadcast call sites that don't have a single agent in scope (handle_plan's typed-stage-split push, handle_rollback, replan fallback) emit the literal `$AGENT_ID` placeholder and rely on the agent container's env to expand it at boot. Fix 2 (Issue #3): ASAP_AGENT_MEMORY_LIMIT_MIB env knob memory_limiter was hardcoded to 1280 MiB. Smoke agent at 1.5 GiB trips the soft limit under the 5-sketch + gorillas3 workload. Read ASAP_AGENT_MEMORY_LIMIT_MIB from the controller's env (default 1280, mirroring the build_gorillas3_yaml env-substitute pattern). `spike_limit_mib` scales as `max(256, limit/5)` so the ratio stays sensible as operators tune the limit. Operators raise both the env var AND the agent container's cgroup limit together. Fix 3 (gorillas3 Bucket Phase 2): drop `bucket:` from emit ASAPCollector#387 made the gorillas3 `Bucket` field a no-op: validation + log + TSDBBucket fallback retired. Controller doesn't need to emit it anymore. Remove the bucket: line from build_gorillas3_yaml. ASAPCollector#387's gorillas3 Config struct still has the `Bucket` field (mapstructure compat) but it's now unread. Test plan: * `cargo test -p control_plane --lib`: 706 (699 baseline + 7 new) * X-Agent-ID assertion: emitted yaml under emit_edge_yaml (both legacy + 5-sketch routing) and emit_gateway_yaml contains `X-Agent-ID:` with the threaded agent_id; broadcast callers preserve the `$AGENT_ID` placeholder verbatim. * memory_limit assertion: emitted yaml's memory_limiter.limit_mib matches ASAP_AGENT_MEMORY_LIMIT_MIB env (or 1280 default), with spike_limit_mib scaling as max(256, limit/5). * bucket: absence: build_gorillas3_yaml output does NOT contain a top-level `bucket:` line (but DOES contain `tsdb_bucket:`). Closes B1-downstream Issues #2 + #3 + gorillas3-Bucket Phase 2. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…-Sum deferred) (#282) Two fixes that all live in control_plane/src/emit/ and adjacent files, bundled to avoid sequential merge churn. The third intended fix (B2 — Sum aggregation alongside sketch) was deferred because the existing workload_store keys by metric name and stores ONE QueryWorkload per metric — the multi-aggregation shape needed for `sum by (zone)` alongside `quantile_over_time` requires either a store-level restructure or a second PhysicalExpr per metric, both substantially bigger than #1+#2. Left as a follow-up; the YAML's existing sum-shaped entries (MVP entries 2/3/4 targeting http_requests_total) still collapse to the last write on a per-metric basis. == Fix #1: B3 population gap == Symptom (smoke test): every emitted `transform/keep_for_<metric>` block had `keep_keys(datapoint.attributes, [])` — strips ALL attrs instead of keeping `grouping_labels`. Sid catalog landed with one sid per metric instead of one per (metric, zone). Root cause: the OpAMP-push path's `metric_to_grouping_labels` WAS being populated from the WorkloadStore (in `main:: emit_bootstrap_typed`, `main::handle_plan`'s typed-stage-split branch, and `replan::Replanner::try_emit_typed_edge_yaml`). But the source `QueryWorkload.group_by_labels` was empty for the canonical MVP query `quantile_over_time(0.99, http_requests_total_latency_ms[30s])` — the PromQL parser only surfaces grouping labels from `by (...)` clauses, and a bare `quantile_over_time` has none. The pre-pop loop in `main.rs` also passed `group_by_labels: vec![]` in the QuerySpec, leaving nothing to merge with the empty parsed value. Fix: add a declarative `grouping_labels: Vec<String>` field to `WorkloadEntry` so the YAML can state the streaming-config grouping contract directly. Thread `entry.grouping_labels` into `QuerySpec.group_by_labels` in the pre-pop loop (main.rs ~276) so `analyzer.analyze()` merges the YAML-declared labels with any PromQL `by` keys into `QueryWorkload.group_by_labels`, which `collect_metric_to_grouping_labels` then drops into `EdgeStageConfig.metric_to_grouping_labels` → the emitter's `keep_keys(datapoint.attributes, [...])` list. Same plumbing in `emit::runtime_tests::populate_store_from_registry` so the existing 5-sketch round-trip test keeps tracking main.rs. Regression coverage (2 new tests in emit/mod.rs): * workload_entry_grouping_labels_round_trip_through_emit_to_keep_keys * workload_entry_grouping_labels_surface_in_emit_keep_keys_list == Fix #2: B4 — controller picks window_duration from query == Symptom: agent's sketch processor's `window_duration` was hardcoded at 300s (5m) for KLL/CMS/etc., 60s for others. The backend's streaming-config `windowSize` likewise drifted from whatever the user wrote in their PromQL `[range]`. Queries with `[30s]` ranges always landed inside an open sketch window and returned NoData. Root cause: the pre-pop QuerySpec hardcoded `time_window: "5m".into()` (main.rs ~278), which the analyzer prefers over the PromQL-parsed value at pipeline.rs:203. The parsed `[30s]` was thrown away. No clamp existed downstream to catch this either, so a `[5m]` workload landed a 300s sketch window that fell outside every sensible replay range. Fix: 1. Pass `time_window: ""` from main.rs and emit/runtime_tests when the entry HAS a `query_string` — lets the analyzer extract the matrix-selector range itself. Falls back to "5m" only when query_string is None (so the analyzer doesn't error at Step 4). 2. Add `clamp_window_secs(Option<u64>) -> Option<u64>` in emit/stage_config.rs with bounds [5, 60]: * Lower 5s — below this the sketch processor mints new windows before it has enough samples for the family's quality bound, and per-flush sid-catalog cardinality explodes. * Upper 60s (= MAX_WINDOW_SECS, the historical default). Above this the user's replay range no longer contains a closed sketch window. 3. Apply the clamp at every emission site so the agent's `window_duration` and the backend's `windowSize` agree exactly (drift de-syncs warm-tier replay): * `build_edge_processor_block` callers in the legacy single-pipeline and 5-sketch routing emit paths (stage_config.rs ~169 and ~1003). * `build_backend_aggregation_json` (stage_config.rs ~1636 — the streaming-config JSON path). * `generate_streaming_config_yaml` (asapquery_backend.rs — the legacy YAML emit path; same clamp so legacy / typed paths agree). * `build_processor_block` in the legacy agent emitter (agent.rs ~138). Regression coverage: * 4 unit tests for `clamp_window_secs` itself (in-range, above-max, below-min, None). * 4 emit-level tests: legacy edge-yaml clamps 300 → 60, legacy edge-yaml preserves 30, 5-sketch routing clamps across all 5 family processors, streaming-config JSON clamps `windowSize`. * 3 legacy `agent.rs` tests covering the same clamp contract (clamps_oversize, clamps_undersize, preserves_inrange). * 1 pre-existing test (`contains_window_duration`) updated to assert the post-clamp 60s value instead of the fixture's pre-clamp 5m. Test plan: * `cargo test -p control_plane --lib`: 719 pass (was 706 pre-change baseline; +13 new tests covering both fixes) * `cargo test -p data_plane --lib`: 712 pass (no data_plane changes — verification only) * `cargo check -p control_plane`: clean Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…agg from disk, and report real memory (#330) PR #329's durable tier passed its unit tests but the first live run (--persistence-seal-window-count=4 --persistence-hot-window-secs=120) left parts/ empty after 13 min of ingest, lost all data on docker restart, dropped [5m]/HLL queries under persistence, and reported ~0 KB sealed bytes. Three root causes: 1. Flush never fired (most severe). The flusher only ever flushes SEALED epochs, and sealing only fires on the count cadence (seal_window_count distinct windows). A slow/stalled series never reaches the cadence, so its aged windows sit un-sealed in current_epoch forever — never made durable. Fix: a time-driven "phase 0" seal — the flusher now rolls every current_epoch window older than the hot window into a sealed epoch each tick (EpochSource::seal_aged_epochs / SidStoreData::seal_aged_windows / MutableEpoch::split_window_ends_before) so it becomes flushable regardless of cadence. Parts now commit during runtime and survive restart. 2. Exact-agg disk read-back missing. query_exact_agg_range and exact_agg_coverage_bounds read only in-memory epochs, so a `sum by (...)` / rate query returned "No result" once its windows were flushed-then-evicted. Fix: both now union the durable tier, reconstructing scalar accumulators (Sum/Increase/MinMax + Multiple*) from disk via reconstruct_exact_agg, keyed by the rebuilt label map. 3. approx_memory_bytes ignored current_epoch, so the MEMORY_DIAG under-reported and the flusher's memory-pressure trigger was blind to the bulk of memory (which under persistence lives un-sealed in current_epoch). Fix: count hot current_epoch + sealed; relabel the diagnostic. Persistence-OFF default path is unchanged (seal_aged is a no-op when persistence_enabled is false; the disk unions are no-ops without a read handle). Reproducing tests fail on origin/main and pass here: live_aged_unsealed_panes_flush_and_survive_restart (#1), live_exact_agg_resolves_from_disk_after_evict (#2), live_total_memory_accounts_for_current_epoch (#3), plus columnar/seal and flusher-level unit tests. Remaining follow-up: MultipleMinMaxAccumulator (needs an out-of-band min/max sub_type) and the sketch-backed accumulator forms still have no generic byte factory, so their evicted-to-disk exact-agg portion is skipped; they remain served from memory. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

zzylol and others added 3 commits April 12, 2026 22:49

zzylol force-pushed the feat/controller-driven-config branch from df644ec to 126d5bd Compare April 13, 2026 02:57

zzylol merged commit a36765f into main Apr 13, 2026
5 of 6 checks passed

zzylol deleted the feat/controller-driven-config branch April 13, 2026 03:13

zzylol mentioned this pull request Apr 20, 2026

feat(http): surface §7 Partial results on Prometheus warnings #49

Merged

4 tasks

zzylol mentioned this pull request Apr 21, 2026

feat(sketchdb): surface accuracy envelope on PromQL response (§6.4, A+C) #56

Merged

zzylol mentioned this pull request Apr 21, 2026

docs + test(sketchdb): proofs.md + accuracy empirical validation (#2.2 + #2.4) #58

Merged

zzylol mentioned this pull request May 5, 2026

fix(precompute-engine): warm-tier persists ingested sketches under queryable key #82

Merged

4 tasks

zzylol mentioned this pull request May 5, 2026

fix(precompute-engine): wall-clock watermark fallback closes idle windows #83

Merged

6 tasks

This was referenced May 10, 2026

refactor: SketchIndex on epoch-columnar storage (refactor #2) #121

Merged

feat: wire SeriesIdResolver + SketchIndex into OTLP receive + query (Phase 4+5+6 finish) #122

Merged

zzylol mentioned this pull request May 13, 2026

refactor(sketch_db): hoist persistence/ to sketch_db level #178

Merged

1 task

This was referenced May 13, 2026

feat(sketch_db): sid-level timeline_for_metric #183

Merged

feat(http): migrate /api/v1/db/timeline to sid-level timeline #184

Merged

zzylol mentioned this pull request May 18, 2026

fix(control_plane): X-Agent-ID header + memory_limit knob + drop gorillas3 bucket (B1 downstream bundle) #279

Merged

6 tasks

zzylol mentioned this pull request May 18, 2026

fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282

Merged

3 tasks

This was referenced May 18, 2026

schema-retirement #5: retire query-path aggregation_id, route directly through sid catalog #272

Closed

perf: reduce backend precompute engine CPU #316

Merged

This was referenced Jul 27, 2026

chore: fold in 5 safe review-findings fixes (dedup/stale-doc/dead-code) #418

Merged

feat: RoutingIndex -- metric-bucketed Tier-2 structural match #435

Merged

milindsrivastava1997 mentioned this pull request Sep 13, 2026

Instant sum/count/avg by and avg_over_time are not accelerated; instant sum fails to compile #702

Open

Navigation Menu

feat: DataCollector controller client — dynamic query config - #2

feat: DataCollector controller client — dynamic query config#2
zzylol merged 3 commits into
mainfrom
feat/controller-driven-config

zzylol commented Apr 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented Apr 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant