data-plane: honest SketchStore memory diagnostic + idle-sid eviction

[zzylol]

Problem

Driving the Google-cluster trace through the agent → data-plane stack surfaced two SketchStore memory issues:

• The memory diagnostic lies at idle. SketchStore::approx_memory_bytes is the flusher's evictable-payload gauge (current_epoch + sealed payloads). Once payloads flush to disk it reads ~0 — but the per-sid registry (instances metadata + per-series InternTable label caches) stays resident. The 30s MEMORY_DIAG line read 0.00 KB while process RSS sat at hundreds of MB (e.g. 36,484 idle sids → ~600 MB RSS reported as 0.00 KB).
• Stale sketch sids are never reclaimed. Schema reconciliation skips sketch sids and eviction only removes Expired sids (config-/retention-driven), so a series that goes quiet pins its in-memory state indefinitely. Under cardinality churn this grows unbounded (observed: a long-running stack at 210k sids / 3.6 GiB / 496 GB disk).

Changes (both scoped to the SketchStore)

1. Honest memory diagnostic

• InternTable::approx_heap_bytes + SketchStore::approx_resident_bytes — registry metadata + intern caches + live payload.
• process_resident_bytes() reads /proc/self/statm for ground-truth RSS.
• MEMORY_DIAG now logs payload (evictable) / registry+intern (resident, not flushable) / process RSS.
• The flusher's payload-pressure trigger is deliberately left on the payload gauge — feeding non-flushable registry memory into it would livelock the flush loop (it can only evict payload).

2. Idle-sid eviction — opt-in --idle-sid-evict-secs / ASAP_IDLE_SID_EVICT_SECS (default 0 = off)

• Periodic sweep drops the in-memory SidStoreData (epoch columns + intern cache + series slot) for sketch sids that are persistence-backed, write-idle past the threshold, and fully durable on disk (sealed_epochs + current_epoch empty) — while keeping the queryable SketchInstanceMetadata.
• Correctness: union_disk_parts_into reads the durable tier via the retained metadata only (never self.series), so an evicted series stays answerable from disk; the append path's entry().or_insert_with rehydrates on the next write. This is the same disk-read path the normal flush-evict loop and restart-recovery already rely on.
• Durability guard ensures un-flushed data is never dropped. Effective idle horizon is max(threshold, persistence_hot_window).
• Adds SidStoreData::last_write_unix_ms, stamped under the append write lock already held (no extra hot-path cost).

Tests

5 new unit tests: resident accounting (registry counted when payload is 0; grows with interned label cardinality), eviction predicate, rehydration-on-write, and the durability guard (recent + pending-data sids spared). cargo test -p data_plane --lib green.

Live validation (Google-cluster trace, multinode)

• MEMORY_DIAG now reports e.g. payload=4207 KB (evictable) / registry+intern≈37 MB (resident) / process RSS=131 MB instead of 0.00 KB.
• With eviction enabled: 11 sweeps, 31,527 stale sids reclaimed, in-memory sid state held to a few thousand instead of pinning all 36,484.

Notes

• Eviction is opt-in; existing deployments are unchanged (default 0).
• Stacked on Sum AggregationType ingest + CMS per-item estimate(key) #355 (feat/sum-aggregation-type); retarget to main once that merges.
• Out of scope / separate follow-up: warm quantile queries for the Google-cluster dataset return empty/kind=exact because the agent ships degenerate ε=0 DDSketch frames (the controller policy correctly declares alpha: 0.02, but registered sids carry relative_accuracy: 0.0) — an ASAPCollector emit/processor gap, not addressed here.

🤖 Generated with Claude Code