MVP demo: agent emits full-attr sketches → no sid match for grouped streaming-config

[zzylol]

Found during the single-node MVP smoke test (/mydata/mvp-smoke-test/REPORT.md on the dev box). Sister to ASAPCollector#381 (agent OpAMP-push doesn't reload) — this is what makes Axis C (end-to-end PromQL) return empty even though Axis D shows 8800 sids registered.

Correction to an earlier version of this issue: the original write-up framed this as an "agg_id hash mismatch" between two xxh64 schemes. That's wrong — compute_sketch_sid was retired in PR #190/#192 and sid is now a counter (per docs/design-sid-lifecycle.md). The actual root cause is sid identity mismatch: the agent emits sketches with all wire attrs, so the backend mints sids over the full attr set, but the streaming-config (and the still-extant aggregation_id query path) expects sids grouped down to grouping_labels only.

What's stored

SeriesIdResolver mints sid = fetch_add(1) keyed by (metric_name, attrs_fingerprint, agg_kind_canonical). The fake-exporter emits http_requests_total_latency_ms with {zone, rack, node, pod, endpoint, service.name, telemetry.sdk.name, telemetry.sdk.language, telemetry.sdk.version} (9 keys); the agent's DDSketch processor doesn't drop or roll up any of them. So the backend mints 51 sids for http_requests_total_latency_ms — one per unique full-attr tuple — each carrying group_by_keys = [endpoint, node, pod, rack, service.name, telemetry.sdk.*, zone].

What the query path looks for

asap_query_engine/engine.rs:1743 logs:

Query execution failed: No precomputed outputs found for metric: http_requests_total_latency_ms,
                        aggregation_id: 16882953407686652000

16882953407686652000 is PolicyFingerprint::as_u64() of the streaming-config entry — xxh64(metric + agg_type + sub_type + params + grouping_labels=[zone] + aggregated_labels=[]). The store has no rows keyed by that agg_id because nothing on the ingest path computed a PolicyFingerprint with grouping_labels=[zone] (the agent never group-reduced before sketching).

Reproduction

bash /mydata/mvp-smoke-test/run_smoke.sh

After 90 s:

• curl http://localhost:19091/api/v1/streaming-config → aggregation_id: 16882953407686652000 registered
• curl http://localhost:19091/api/v1/db/schemas → 51 sids for the metric, each with full-attr group_by_keys
• curl 'http://localhost:19091/api/v1/db/timeline?metric=http_requests_total_latency_ms&start_ms=...&end_ms=...' → 1 segment under agg_id: 6206425945726099596 — note this is the timeline endpoint's content-derived signature id (xxh64 of metric + agg_kind + group_by_keys per the comment in handle_get_timeline), not the PolicyFingerprint the query path looks up. Two different hash schemes, both surfaced as agg_id in the API output — confusing but real.
• curl --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[30s])' http://localhost:19091/api/v1/query → {"error":"No result for query"}

Fix paths

This sits at the intersection of two in-progress migrations:

1. Schema-retirement feat: vendor DataCollector's modified OTLP proto (PR A, Phase 1) #5 in engine.rs:2392 — finish removing the aggregation_id lookup from the query plan and route queries directly through the sid catalog ((metric, agg_kind, attrs_subset) → sids that subsume the asked grouping). When this lands the agg_id path goes away entirely, and the question becomes "find sids whose group_by_keys are a superset of the query's grouping, then merge by the asked subset."
2. Agent-side group-reduction — when ASAPCollector#381 lands and the controller's typed-stage-split YAML actually reaches the agent, the per-metric pipeline needs an attributes/keep (or equivalent) processor upstream of each sketch processor that whitelists grouping_labels. Then the agent emits sketches with attrs already reduced to [zone], the resolver mints the desired N-zone sids, and any query that asks for the streaming-config's grouping finds them directly.

Either fix unblocks Axis C. Both might land independently — (1) is purely backend, (2) needs ASAPCollector#381 first.

Tagged bug, mvp-demo, query-engine, schema-retirement-5.

[zzylol]

Schema-retirement #5 (one of the two fix paths called out in this issue) is now tracked in #272.

[zzylol]

Fix path 2 (backend-side: union instances_matching into the modern execute() sid resolution) landed in #273. End-to-end smoke test Axis C now returns a real DDSketch quantile:

$ curl --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[5m])' \
       http://localhost:19091/api/v1/query
{"accuracy":{"delta":0.0,"epsilon":0.01,"kind":"relative_quantile"},
 "data":{"result":[{"metric":{"zone":""},
                    "value":[1779043268.182,"90.93548893834691"]}],
         "resultType":"vector"},
 "infos":["accuracy: ε=0.01, δ=0, kind=relative_quantile",
          "precompute_window: [1779042780000, 1779042810000) ms (width 30000 ms)",
          "data_source: asap_query"]}

NB: smoke test must use [5m] (or any range \≥ the agent's 60s window_duration); [30s] returns NoData because the range can land entirely inside an open sketch window. Separately, the sum by (zone) (http_requests_total) query still fails — the agent emits DDSketch but the query needs Sum capability, which is fix path 1 territory (controller pushes appropriate sketch type or agent emits ExactAgg precompute). Closing this issue would be premature until fix path 1 is also resolved or scope-narrowed.

The remaining agg_id retirement work (legacy handle_query path, ingest bucketing, precompute worker, backfill) is tracked in #272.

[zzylol]

Status update 2026-05-17 — fix path 2 (controller-side) now lands in stages:

B3 (agent emits sketches grouped by [zone])

• ASAPQuery-backend feat(emit): inject attributes/keep upstream of each sketch processor (B3) #278 — controller injects attributes/keep upstream of each sketch processor (only when grouping_labels is set)
• ASAPQuery-backend fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282 — populates metric_to_grouping_labels from WorkloadEntry so the routing-table builder has data to thread
• ASAPCollector docs(control_plane): phase-by-phase migration plan for BackendPlan cutover #390 — adds grouping_labels: [zone] to all 8 entries in deploy/mvp-singlenode/configs/mvp-workload.yaml

End-to-end verification post-#390:

$ curl --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[30s])' \
       http://localhost:19091/api/v1/query
# returns real DDSketch quantile, ε=0.01, grouped by zone

Still failing — sum by (zone) (http_requests_total): DDSketch can't answer Sum capability. Tracked as B2 (Sum-as-separate-aggregation). The fix in flight restructures WorkloadStore from HashMap<metric, QueryWorkload> to HashMap<(metric, AggRole), QueryWorkload> so the same metric can carry both a quantile AggregationConfig (DDSketch) and a sum AggregationConfig (ExactAgg) simultaneously.

Closing this issue when B2 lands and Sum-capability returns end-to-end.

[zzylol]

Closing as fixed — the wave is end-to-end green on both single-node smoke harness and 4-node multinode deploy.

Path 1 (controller-side, originally tracked here):

• B2 (Sum as separate aggregation): feat(planner): (metric, role)-keyed WorkloadStore + PlanStore (B2 full) #283 + fix(emit): cumulative streaming-config across (metric, role) pairs (B2 follow-up) #287 + refactor(control_plane): unify Replanner through typed cumulative emit; retire generate_streaming_config_yaml (Option B) #290 (Option B unified emit)
• B3 (group-reduction upstream of sketches): feat(emit): inject attributes/keep upstream of each sketch processor (B3) #278 + fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282 + ASAPCollector docs(control_plane): phase-by-phase migration plan for BackendPlan cutover #390
• B4 (window=30s from query): fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282
• B7.5/B7.6/B7.7 (sid-keyed everything): refactor(query): delete legacy handle_query family (B7.5 v2 redo) #280 / refactor(data_plane): rekey ingest bucketing from (agg_id, group_key) to sid (B7.6) #284 / refactor(data_plane): rekey backfill processor + SketchStore from agg_id to sid (B7.7) #286

Path 2 (backend-side, the original "fix path 2"): #273 — landed early in the wave

Last-mile data_plane work:

• feat(query): dispatch sum by (...) PromQL to ExactAgg(Sum) sids (closes Option-B downstream) #291 — dispatch sum by (...) to ExactAgg(Sum) sids
• feat(query): compose rate-over-Sum into ExactAgg(Sum) candidate (multinode topk dispatch) #292 — compose rate-over-Sum into ExactAgg(Sum) candidate (enables topk(K, sum by (...) (rate(...))))

Multinode harness gap (separate but related): ASAPCollector #394 — added controller container to mvp-multinode runner + fixed replay typo + extended queries-e2e.json with wave queries + cleaned stale bucket: yaml key.

Smoke verification (singlenode):

$ curl --data-urlencode 'query=sum by (zone) (http_requests_total)' \
       http://localhost:19091/api/v1/query
# 4 zones, data_source: asap_query

$ curl --data-urlencode 'query=topk(5, sum by (zone) (rate(http_requests_total[5m])))' \
       http://localhost:19091/api/v1/query
# top-K, data_source: asap_query

Multinode verification (4-node, against node2:9091):

• sum by (zone) (http_requests_total) → z0..z3 ~981M each
• sum by (zone) (rate(http_requests_total[5m])) → z0..z3 ~4.8M
• topk(5, sum by (zone) (rate(...))) → ranked 4 zones
• All three: data_source: asap_query
• Replay JSONL: 384 success lines (vs 0 pre-wave)