schema-retirement #5: retire query-path aggregation_id, route directly through sid catalog

[zzylol]

Tracking issue for the final step of the agg_id retirement migration. Steps #1-#4 already landed (sid-level eviction, sid-level reconcile from streaming-config, sid-level §6.3 ingest barrier, big-bang SchemaRegistry deletion — see PRs #183-#187 + #189). Step #5 is the remaining surface: the query path still resolves precomputed outputs via StreamingConfig.aggregation_id, and ingest still buckets per-(aggregation_id, group_key) even though storage downstream is sid-keyed.

This is the issue that #271 (MVP demo: agent emits full-attr sketches → no sid match) calls out as one of two fix paths for Axis C of the MVP smoke test.

Concrete sites still holding agg_id

data_plane/src/query_engines/asap_query_engine/engine.rs
  :2392   "Until schema-retirement #5 ports the per-segment dispatch
           to sid-level evaluation directly …"
  :945, :990, :969, :1042, :1127, :2066, :2099, :2197, :2305
          full per-segment dispatch + error paths still keyed on agg_id

data_plane/src/drivers/ingest/otel.rs
  :561    by_group.entry((config.aggregation_id(), group_key))
  :640    WorkerMessage::AccumulatorInput { agg_id: config.aggregation_id(), … }
  :1203   // DEPRECATED: aggregation_id-keyed write — remove
  :520, :723   "Schema retirement #5 — agg_id-keyed `SchemaRegistry` is gone"
          (already done; reference comments)

data_plane/src/precompute_engine/output_sink.rs
  :206, :232   output sink still tags writes with cfg.aggregation_id()

data_plane/src/storage_engines/sketch_db/backfill/processor.rs
  :343, :350, :418, :440, :596, :646   backfill processor maps still keyed on agg_id

data_plane/src/storage_engines/types/precomputed_output.rs
  :40, :162   `// with aggregation_id kept for legacy compat; PR 5 retired …`

data_plane/src/main.rs
  :491, :562, :739, :784   `// Schema retirement #5 — …` reference comments

What "done" looks like

1. Query path's build_query_execution_context_promql_for_agg_id (engine.rs:2197) folded into a sid-keyed equivalent that takes (metric, agg_kind, attrs_subset) and finds candidate sids whose group_by_keys ⊇ attrs_subset, then merges across them.
2. Ingest bucketing in otel.rs:561 rekeyed from (aggregation_id, group_key) → (sid, …) or (metric, agg_kind, group_key) → sids.
3. output_sink.rs and backfill/processor.rs rekeyed from agg_id → sid.
4. PolicyFingerprint::as_u64() survives as the streaming-config identity for the OUTSIDE world (controller → backend POST, persistence), but the internal config.aggregation_id() accessor disappears from hot paths.
5. Re-enable the two ignored cross-reconfigure dispatch tests in tests/schema_timeline_dispatch_tests.rs (gated by sid-level per-segment dispatch — fixture needs reworking too because the two test configs collapse to one sid under content-addressing; see asapquery-backend-phase5-m2-3-progress notes).
6. Delete pub fn aggregation_id() on AggregationConfig (or downgrade to test-only) once all consumers are gone.

Why this unblocks the MVP demo

With schema-retirement #5 done, the query path resolves directly from the metric name + asked grouping to a set of sids in the catalog — no PolicyFingerprint detour. The full-attr sketches the agent currently emits (because of ASAPCollector#381 — agent doesn't apply OpAMP runtime config) become queryable: their group_by_keys is a superset of [zone], so a sum by (zone) query merges across them at execution time.

(Until #5 lands, the alternate unblock is to have the agent group-reduce upstream of the sketch — needs ASAPCollector#381 first.)

Done already (don't redo)

• agg_id-keyed SchemaRegistry deleted (PR feat(sketch_db): retire schema/ module — sid-only lifecycle #189)
• sid-level eviction sweep (PRs feat(sketch_db): sid-level timeline_for_metric #183-feat(sketch_db): sid-level §6.3 ingest barrier; drop schemas.is_writable #187)
• sid-level reconcile from streaming-config (PR feat(sketch_db): sid-level reconcile_from_streaming_config #186)
• §6.3 ingest barrier moved to sid (PR feat(sketch_db): sid-level §6.3 ingest barrier; drop schemas.is_writable #187)
• HTTP /api/v1/db/schemas?:agg_id endpoints rewired to :sid

See asapquery-backend-phase5-m2-3-progress memory + the post-M2.3 sketch_db reorg notes for the broader context.

Tagged mvp-demo, query-engine, schema-retirement-5.

[zzylol]

Step 1 (query path: union instances_matching into ASAP-tier sid resolution at the two modern execute() sites) merged in PR #273. End-to-end smoke test now returns a real DDSketch quantile via the modern execute() path. See PR for the before/after.

Still TODO (each its own PR):

1. Legacy handle_query / execute_context path (engine.rs:945–1127) still emits the No precomputed outputs found for metric: X, aggregation_id: Y WARN for every query. The path is already "superseded" — process_via_simple_engine falls over to modern execute() when legacy returns None, and post-fix(query): union instances_matching into ASAP-tier sid resolution (schema-retirement #5 step 1) #273 the fallback succeeds. Options: (a) downgrade the WARN to DEBUG since it's no longer fatal, (b) retire the legacy path entirely.
2. Ingest bucketing rekey (otel.rs:561,640 → WorkerMessage::AccumulatorInput { agg_id, group_key }). Touches the precompute worker's internal hashmap shape.
3. Output sink + backfill processor — 8 config.aggregation_id() sites in output_sink.rs + backfill/processor.rs.
4. Re-enable the two ignored cross-reconfigure dispatch tests in tests/schema_timeline_dispatch_tests.rs.
5. Delete pub fn aggregation_id() on AggregationConfig once all consumers are gone.

[zzylol]

Step 2 (port resolve_sketch_metric_alias to modern execute() and execute_range_promql_modern) merged in PR #274. Modern can now do the _quantile / _hll rewrite that legacy did.

Next sub-steps, in approximate dependency order:

Sequential (each unblocks the next):
3. Spawn capability-miss notify from modern execute() — prerequisite for the call-order flip
4. Reorder process_via_simple_engine to call modern first; investigate the engine.rs:792 time=0 underflow that surfaces when legacy is reached via the modern-Err fallback
5. Migrate try_handle_query_promql_via_timeline to sid-level per-segment dispatch (the last legacy-only feature)
6. Delete the legacy handle_query / handle_query_promql / execute_context family

Independent (can be parallelized with 3–6):

• Rekey ingest bucketing in otel.rs:561,640 from (aggregation_id, group_key) → sid. Cascades through WorkerMessage::AccumulatorInput, precompute_engine::{worker,output_sink}. Likely needs 2 PRs (worker shape change is one; output_sink + downstream is another).
• Rekey backfill processor (sketch_db/backfill/processor.rs — 6 sites).

End-state cleanup (once everything above lands):

• Re-enable the two ignored cross-reconfigure dispatch tests
• Delete pub fn aggregation_id() on AggregationConfig (and PolicyFingerprint::as_u64() if it has no other consumers)

[zzylol]

Status update 2026-05-17 — progress on the 5 remaining sub-PRs:

#	Item	Status
1	Legacy handle_query family retirement	✅ landed in #280 (B7.5 v2)
2	Ingest bucketing rekey (otel.rs + WorkerMessage::AccumulatorInput + worker bucketing)	🚧 in flight as B7.6
3	Output sink + backfill rekey (8 aggregation_id() sites)	⏭️ B7.7 — queued after B7.6
4	Re-enable ignored cross-reconfigure dispatch tests	⏭️ end-state cleanup
5	Delete pub fn aggregation_id() on AggregationConfig	⏭️ after #2 + #3 land

Also-relevant landed this wave (not in the original step list but they cleared adjacent debt):

• revert+retire(query): delete dead resolve_sketch_metric_alias rewrite #275 reverted + retired the dead resolve_sketch_metric_alias (sketch processors don't rename metric names since the 2026-05 refactor)
• feat: autonomous capability-match allocation — (ε+queries) → {sketches, p, δ}, applied live #382 (ASAPCollector) retired the MetricSuffix field from sketch processors for the same reason
• fix(emit): deterministic YAML via BTreeMap (kills agent apply-loop) #281 fixed an apply-loop in emit/stage_config.rs (BTreeMap for deterministic yaml)
• docs(control_plane): design doc for BackendPlan wire format #389 (ASAPCollector) added no-op RemoteConfig short-circuit + ReportsRemoteConfig capability, paired with fix(emit): deterministic YAML via BTreeMap (kills agent apply-loop) #281 to fully break the apply-loop

Next: B7.6 PR coming; B7.7 follows.

[zzylol]

Status update 2026-05-18 — remaining checklist:

#	Item	Status
1	Legacy handle_query family retirement	✅ #280 (B7.5 v2)
2	Ingest bucketing rekey (sid)	✅ #284 (B7.6)
3	Output sink + backfill rekey (sid)	✅ #286 (B7.7)
4	Re-enable ignored cross-reconfigure dispatch tests	✅ moot — #288 documented; the file was deleted in #280, the two panic!("ignored: …") stubs were placeholders awaiting a port that became unnecessary when the legacy dispatcher was retired
5	Delete pub fn aggregation_id() on AggregationConfig	⏭️ remaining

Step 5 awaits a final sweep of remaining policy_fp.as_u64() users (these are legitimate policy-registry uses of the same numeric — the accessor name is misleading but the call sites are correct). PR #286 enumerated the surface (~13 sites in StreamingConfig map-key construction).

Also along the way, surfaced + closed:

• format_series_key/parse_labels_from_series_key quote-roundtrip bug (fix(data_plane): make format_series_key + parse_labels_from_series_key roundtrip cleanly #285) — found by refactor(data_plane): rekey ingest bucketing from (agg_id, group_key) to sid (B7.6) #284 agent during B7.6
• controller_plan_to_query_full_roundtrip_{ddsketch,kll} pre-existing test failures (fix(tests): repair controller_plan_to_query_full_roundtrip post-#280 legacy retirement #289) — start_time_unix_nano: 0 building OTLP fixtures
• Cumulative streaming-config emit across (metric, role) pairs (fix(emit): cumulative streaming-config across (metric, role) pairs (B2 follow-up) #287 + refactor(control_plane): unify Replanner through typed cumulative emit; retire generate_streaming_config_yaml (Option B) #290)
• Sum-dispatch for sum by (...) (feat(query): dispatch sum by (...) PromQL to ExactAgg(Sum) sids (closes Option-B downstream) #291) and rate-over-Sum compose (feat(query): compose rate-over-Sum into ExactAgg(Sum) candidate (multinode topk dispatch) #292) — the data_plane consumers of the post-B2 multi-role world

Sister issue #271 (the original MVP demo bug) is now closed — both single-node and multinode validate end-to-end.

Leaving #272 open until step 5 (aggregation_id() accessor retirement) lands.