feat: wire SeriesIdResolver + SketchIndex into OTLP receive + query (Phase 4+5+6 finish)

[zzylol]

Summary

Lights up the centralized sid resolver and the new SketchIndex on the live ingest+query path. Previous PRs in this chain:

• refactor: relocate controller into asap-query-backend + delete asap-common + scaffolding for centralized series_id #120 — controller relocation + Phase 4 / 5 scaffolding
• refactor: SketchIndex on epoch-columnar storage (refactor #2) #121 — SketchIndex on epoch-columnar storage

This PR is the wire-in. Together with #120 + #121 + ProjectASAP/ASAPCollector#372, it completes the centralized-sid-namespace refactor laid out in docs/design-controller-into-backend.md.

What landed

Phase 4 finish

• OTLP receive sid-resolution gate in route_modified_otlp_sketches_to_precompute. For each sketch DataPoint:
  • (sid=0, attrs present) → resolver mints fresh sid via compute-or-mint cache.
  • (sid != 0, attrs present) → trust attrs; backend-resolved value wins. Mismatch → push old sid to unknown_series_ids for sender-side eviction.
  • (sid != 0, attrs empty) → resolver.is_known(sid) lookup; miss → push to unknown_series_ids, drop DP (sender will resend with attrs on next push).
• route_modified_otlp_sketches_to_precompute now returns Vec<u64> of unknown sids; both MetricsServiceImpl::export and handle_otlp_http stamp it into ExportMetricsServiceResponse.unknown_series_ids (the field already existed on the wire from Phase 2 as a Vec::new() placeholder).
• New ResolveSeriesIDs gRPC service in metrics_service.proto: bulk pre-resolution endpoint for senders that have an attribute-only batch ready before the next Export. Reuses the existing SeriesAssignment shape; idempotent end-to-end.

Phase 5 finish

• IngestState carries Arc<SeriesIdResolver> + Arc<SketchIndex>. Both allocated once in main.rs and threaded through PrecomputeEngine::new so the OTLP receiver, the precompute engine, and SimpleEngine all share the same instances.
• OTLP ingest writes: after sid resolution, on first-seen sid the receiver materializes SketchInstanceMetadata (capability inferred from sketch kind: DDSketch/KLL → QuantileApprox, HLL → CardinalityApprox, CountSketch/CountMin → FrequencyTopk; sketch_config from the parent-container fields lifted in Phase 2; accuracy via AccuracyBound::from_config) and registers it. Per-DP: append_sample(sid, label_values, (start_ms, end_ms), SketchSampleState { bytes, encoding }).
• ModifiedOtlpSketchDp extended with series_id, start_time_unix_nano, and container_config: SketchConfig. The five typed-DP match arms thread these from the OTLP wire shape.
• Helpers: sketch_kind_handle_for(&dp) and encoding_to_handle(i32) -> SketchEncoding.
• Query path warm-tier read in SimpleEngine::execute: parses PromQL → extracts metric name + label-matcher KEY set → calls SketchIndex::instances_matching → classifies each candidate sid:
  • All Hit → falls through to handle_query (the per-Capability sketch reducer over query_range is a follow-up; today the all-Hit case still flows the legacy SimpleMapStore path).
  • Any Ghost or Unknown → returns EngineError::CapabilityMiss(SketchWarmTier, …) to trigger archive failover.
  • Empty match → likewise CapabilityMiss.
• New SketchIndex::instances_matching(metric, required_keys) helper (~15 lines): filters instances where metric_name matches and required_keys is a subset of the instance's group_by_keys (over-approximation; archive failover catches false positives).
• Legacy WorkerMessage::AccumulatorInput (aggregation_id-keyed) write site annotated // DEPRECATED: — kept as fallback during validation, removed after warm-tier carries traffic.

Phase 6 finish

• EngineRouter::execute already retries on CapabilityMiss against the next engine in priority order, so Ghost/Unknown automatically fall through to the archive engine (ThanosForward is already wired into the router today). No router change needed — CapabilityMiss(SketchWarmTier, ...) IS the failover trigger. Verified by reading routing/engine_router.rs::execute (lines 218–249).

Deferred (TODOs at call sites)

• Per-Capability sketch reducer over SketchIndex.query_range for the all-Hit case. Today still flows through the legacy SimpleMapStore path. Follow-up PR will swap that for a real warm-tier evaluator (DDSketch quantile fold, HLL cardinality estimate, CMS / CS frequency lookup).
• Phase 6 hybrid stitch (warm covers [t0..t1'], archive covers [t1'..t1], concatenate). Requires QueryResult to carry timestamp coverage metadata, which it doesn't today. Deferred — flagged as a stretch goal in the original plan.
• Legacy WorkerMessage::AccumulatorInput aggregation_id-keyed write path: marked // DEPRECATED: only. Removal is a separate cleanup PR after warm-tier validation in production.

Diff stats

asap-query-engine/src/drivers/ingest/otel.rs              | 569 +++++++++++++++---
asap-query-engine/src/engines/simple/engine.rs            | 277 +++++++++
asap-query-engine/src/main.rs                             |  32 +-
asap-query-engine/src/precompute_engine/engine.rs         |   4 +
asap-query-engine/src/precompute_engine/ingest_handler.rs |  16 +
asap-query-engine/src/stores/sketch_index.rs              |  25 +
crates/.../metrics_service.proto                          |  28 +
7 files changed, 884 insertions(+), 67 deletions(-)

Test plan

• cargo build --release -p query_engine_rust clean (warnings only)
• 7 net-new tests pass:
  • drivers::ingest::otel::sid_resolution_tests::* (3 tests: fresh-mint, unknown-sid-empty-attrs, sid-attrs-mismatch)
  • engines::simple::engine::warm_tier_classify_tests::* (3 tests: Ghost → CapabilityMiss, Unknown → CapabilityMiss, no-instance → CapabilityMiss)
  • 1 expansion in series_resolver tests
• Baseline-checked: 32 pre-existing failures (datafusion::*, persistence_integration_tests::*, schema_timeline_dispatch_tests::*) are red on main independently of this branch — verified via git stash + run on main. This PR's changes are net +7 passing, -1 failing.
• Follow-up: per-Capability sketch reducer for the all-Hit case
• Follow-up: hybrid stitch with QueryResult coverage metadata
• End-to-end smoke after Refactor feat: DataCollector controller client — dynamic query config #2 + this PR + companion ASAPCollector#372: tcpdump shows series_id traffic on subsequent Exports, unknown_series_ids repopulates after backend restart without persistence

🤖 Generated with Claude Code