feat(data_plane): durable disk-backed tier for warm SketchStore

[zzylol]

Summary

Turns the warm SketchStore's half-wired persistence layer into a real durable tiered store: hot data in memory, older sealed data flushed to disk and queryable from disk, surviving restarts. Memory is now bounded by flush-then-evict under --persistence-enabled, instead of #327's age-based drop (which still bounds the in-memory-only default). Builds on #327 + #323–#326 — does not revert any of them.

Tiered design

HOT current_epoch (mutable, in-mem)
  -> SEALED epochs (in-mem, pending flush)
     -> DISK parts (durable, via PartCache)

query_range unions all three across the requested range.

• Sealing fires under persistence. SidStoreData gains a seal_window_count cadence (default 20 distinct windows ≈ 10 min of 30s panes — large enough to amortize per-part header/index overhead, small enough that data behind the flusher's hot_window is actually sealed and thus flushable). current_epoch rotates into sealed_epochs on cadence so the flusher has sealed epochs to persist. The max_epochs rotation-drop is disabled under persistence (the flusher owns sealed-epoch lifecycle: seal → durable part → evict). With persistence OFF, nothing seals → fix(data_plane): bound warm SketchStore memory with retention horizon #327 retention bounds memory (unchanged).
• Query reads disk. union_disk_parts_into consults Manifest::live_parts_overlapping + PartCache for the evicted portion of the range. The fix(data_plane): canonicalize sketch metric name at OTLP ingest #323–fix(data_plane): overlap-scan sketch reads for short/instant windows #326 read contract is preserved across the in-mem/on-disk boundary: same half-open overlap (end_ts > start && start_ts < end), and the delta-stitching carry-in fetches the most-recent Full base from disk when it has aged out of memory. The full label key→value map is rebuilt from the sid's group_by_keys (sorted) zipped against the stored values vector — no part-format change for keys. SketchEncoding (Full vs Delta) is round-tripped, via a repurposed v1 part-entry pad byte (legacy 0 decodes as Full, the safe carry-in default).
• Retention vs flush coexist. When persistence is ON, enforce_retention is a no-op so retention never drops un-flushed sealed/current_epoch data out from under the flusher; the disk-tier TTL (delete_older_than_ms) bounds the durable copy. When OFF, fix(data_plane): bound warm SketchStore memory with retention horizon #327 retention stays the bound. Composition is explicit and tested both ways.
• Restart durability. start_persistence runs recovery::recover, installs a PersistenceReadHandle (recovered manifest + cache) and the seal cadence, then starts the flusher — a reopened store on the same dir immediately serves recovered data.

Deploy flags (operator)

--persistence-enabled
--persistence-dir /var/lib/asap/sketchstore        # needs a PERSISTENT VOLUME mount
--persistence-memory-limit-mb 2048                  # sealed-epoch high-water; flusher evicts oldest-first above this
--persistence-hot-window-secs 3600                  # seal epochs older than 1h get flushed regardless of pressure
--persistence-delete-older-than-secs 604800         # disk-tier TTL (7d) — the durable retention horizon
--persistence-flush-interval-ms 1000
--persistence-part-cache-mb 256                     # Tier-2 mmap cache (default min(10% mem, 512))
--persistence-seal-window-count 20                  # NEW: seal cadence in distinct windows

Volume: mount a persistent volume at --persistence-dir (parts + manifest live under <dir>/sketch_index/; the series-resolver WAL under <dir>/series_resolver.wal).

Files changed

• index/epoch_columnar.rs — SidStoreData::{seal_window_count, persistence_enabled}; maybe_rotate_epoch (cadence + no-drop under persistence); enforce_retention (no-op under persistence).
• index/mod.rs — PersistenceReadHandle; SketchStore::{persistence_read, seal_window_count, fresh_sid_store, enable_persistence_mode, union_disk_parts_into, sid_group_by_keys, rebuild_label_map, list_sealed_epochs_len}; query_range disk union; encoding_to_tag/tag_to_encoding; snapshot_sealed_epoch records encoding.
• persistence/{config,part,source}.rs — seal_window_count config; per-entry encoding_tag round-trip; encoding_tag constants.
• main.rs — --persistence-seal-window-count plumbed into the config.

Test plan

• cargo test -p data_plane --lib — 748 passed, 0 failed
• sealing fires under persistence; sealed > 0
• flush+evict bounds memory (sealed bytes → 0, parts appear)
• query over an evicted range resolves from disk (label map + samples)
• delta-only query window gets a disk-resident Full carry-in base
• restart recovery: reopen on same dir → flushed data queryable
• persistence-disabled non-regression (fix(data_plane): bound warm SketchStore memory with retention horizon #327 retention + fix(data_plane): canonicalize sketch metric name at OTLP ingest #323–fix(data_plane): overlap-scan sketch reads for short/instant windows #326 shapes)
• cargo clippy -p data_plane --lib — no new errors (pre-existing approx_constant at thanos_query_engine/forward.rs:850 unchanged)

Follow-ups (not in this PR)

• Disk read-back is wired for the sketch path (query_range). The exact-agg / precompute disk read (query_exact_agg_range, query_precomputes_by_agg) is left in-memory-only — those payloads aren't part of the marquee sketch queries; extend with the same union_disk_parts_into pattern when needed.
• The PR keeps the surgical unindented if let block in query_range to minimize diff (repo is not rustfmt-clean on main); a follow-up reindent is cosmetic.

🤖 Generated with Claude Code