fix(data_plane): make recovered disk-resident warm-sketch series queryable after restart

[zzylol]

Problem

PRs #329 (durable tier) + #330 (flush/time-seal/recovery-load/exact-agg disk read) landed, and a live test confirmed flush + recovery load parts (persistence recovery: live=234). But after a data-plane restart, asap_query returned "No result" cluster-wide and the in-memory SketchStore collapsed from ~9000 to ~4 instances.

Root cause

Recovery (persistence/recovery.rs::recover + SketchStore::start_persistence) loads the parts manifest + part-cache but does not register the disk-resident series as sids. Registration into the instances map only ever happens on the live ingest path (drivers/ingest/otel.rs) when a fresh DataPoint arrives; the SeriesIdResolver WAL recovers (metric, attrs_fp, agg_kind) -> sid but does not push identities into the SketchStore.

So after a true restart the instances registry is empty for the recovered metrics:

• instances_matching (index/mod.rs) keys off instances -> enumerates nothing -> engine returns "No result" before reading any window.
• Even if a sid were enumerated, union_disk_parts_into / union_disk_exact_agg_into early-return on sid_group_by_keys(sid) (also instances-backed).

The on-disk part format (part.rs) carries only label VALUES + sketch_type_name -- not the metric_name, group-by KEYS, or structured AggKind the query path needs to re-register a queryable sid.

Fix

Add a compact per-sid metadata sidecar (sid_metadata.json, new persistence/metadata.rs):

• The flusher upserts each flushed sid's record (metric name, group-by keys, structured AggKind) right before it makes a part durable, via a new EpochSource::instance_metadata_for_persist hook.
• On recovery, start_persistence replays the sidecar and re-registers each disk-resident sid as a queryable instance (register_recovered_disk_series). capability/accuracy are re-derived from the persisted agg_kind exactly as the ingest path derives them (so no serde needed on the control-plane Capability/SketchKindHandle enums).

Properties:

• Idempotent -- a sid a live DataPoint already re-registered is not clobbered (in-memory metadata is authoritative).
• Persistence-OFF unchanged -- the sidecar only exists under the flusher; persistence_disabled_keeps_327_retention_behavior still passes.
• Forward/backward tolerant -- a missing/unparsable sidecar (parts written before this feature, or a fresh dir) degrades to the prior behavior: those sids re-register on the next live DataPoint.

Why #330's tests missed it

restart_recovery_makes_flushed_data_queryable and live_aged_unsealed_panes_flush_and_survive_restart both call idx2.register(...) on the fresh store before querying (the test even comments "here we re-register to model that"). That re-population masks the disk-only path. The two new tests do a genuine fresh reopen with no register().

Reproducing tests (fail on origin/main, pass here)

• recovered_sketch_series_queryable_after_fresh_reopen_without_register -- KLL quantile: ingest -> seal -> flush -> evict (disk-only), fresh reopen, assert instances_matching finds the sid and query_range returns the evicted window.
• recovered_exact_agg_series_queryable_after_fresh_reopen_without_register -- Sum exact-agg by (zone): same shape via query_exact_agg_range + exact_agg_coverage_bounds.

Both assert at the instances_matching step (registry=0 on origin/main).

Test plan

• cargo test -p data_plane -- 763 lib tests pass (+7 new: 2 integration + 5 sidecar unit), all 38 persistence tests pass, fix(data_plane): canonicalize sketch metric name at OTLP ingest #323-fix(data_plane): make durable sketch tier flush, read exact-agg from disk, report real memory #330 regressions green.
• cargo clippy -p data_plane -- no new warnings in touched files (pre-existing approx_constant at forward.rs:850 ignored).
• New tests confirmed to FAIL with the recovery re-register disabled (simulating origin/main: registry=0, "No result").

🤖 Generated with Claude Code