fix(data_plane): make durable sketch tier flush, read exact-agg from disk, report real memory

[zzylol]

Summary

PR #329's durable tier passed unit tests but the first live run
(--persistence-seal-window-count=4 --persistence-hot-window-secs=120)
exposed three bugs the tests missed. On node2, after 13 min of ingest the
sketch_index/parts/ dir was empty, parts_manifest.log was 0 bytes and
the snapshot was a freshly-initialized empty manifest — only
series_resolver.wal grew (1.5M). So data flowed but nothing was ever made
durable, a docker restart recovered live=0, and post-restart queries
returned "No result".

Bug 1 — flush never fired (most severe). file:persistence/flusher.rs

The flusher only flushes sealed epochs, and sealing only fires on the
count cadence (seal_window_count distinct windows in current_epoch,
epoch_columnar.rs::maybe_rotate_epoch). A slow/stalled series never reaches
the cadence, so its aged windows sit un-sealed in current_epoch forever
and are never flushable. Fix: a time-driven "phase 0" seal — each tick the
flusher rolls every current_epoch window older than the hot window into a
sealed epoch (new EpochSource::seal_aged_epochs →
SidStoreData::seal_aged_windows → MutableEpoch::split_window_ends_before),
so it becomes flushable regardless of cadence. Parts now commit during runtime
and survive restart.

Bug 2 — exact-agg disk read-back missing. file:index/mod.rs

query_exact_agg_range / exact_agg_coverage_bounds read only in-memory
epochs, so a sum by (...) / rate query returned "No result" once its
windows were flushed-then-evicted. Fix: both now union the durable tier,
reconstructing scalar accumulators (Sum/Increase/MinMax + Multiple*) from disk
via reconstruct_exact_agg, keyed by the rebuilt label map.

Bug 3 — approx_memory_bytes ignored current_epoch. file:index/mod.rs + main.rs

It summed sealed epochs only, so the MEMORY_DIAG under-reported ("0.00–0.12
KB approx sealed bytes") and the flusher's memory-pressure trigger was
blind to the bulk of memory (which under persistence lives un-sealed in
current_epoch). Fix: count hot current_epoch + sealed; relabel the
diagnostic.

The persistence-OFF default path (#327) is unchanged: seal_aged_epochs is a
no-op when persistence_enabled is false, and the disk unions are no-ops
without a read handle.

Reproducing tests (fail on origin/main, pass here)

• live_aged_unsealed_panes_flush_and_survive_restart — bug 1: aged
  below-cadence panes flush to disk and survive a reopen of the same dir.
• live_exact_agg_resolves_from_disk_after_evict — bug 2: sum by (zone)
  shape + coverage bounds resolve from disk after flush+evict.
• live_total_memory_accounts_for_current_epoch — bug 3: memory accounting
  includes un-sealed current_epoch.
• Plus split_window_ends_before_*, seal_aged_windows_*,
  hot_window_time_seals_unsealed_aged_epochs_then_flushes.

Test plan

• cargo test -p data_plane — 756 lib + integration pass, 0 fail
• cargo clippy -p data_plane — exit 0, no new warnings in changed files
  (pre-existing approx_constant at forward.rs:850 unchanged)
• Live re-validation (parent): rebuild+redeploy whole stack at this branch,
  ingest, then docker restart the data-plane and re-run the marquee
  queries.

Recommended live re-validation (forced-flush) settings

--persistence-enabled --persistence-dir=... --persistence-seal-window-count=4 --persistence-hot-window-secs=30 --persistence-flush-interval-ms=1000. A
30s hot window (below the 120s used originally) makes the time-driven seal
fire promptly so flushing is observable within ~1 min; expect non-empty
parts/ + a growing parts_manifest.log, and MEMORY_DIAG should now show
real in-memory bytes. Then docker restart and confirm
persistence recovery: live=N>0 and the quantile_over_time(...[30m]) query
returns data.

Remaining follow-up

MultipleMinMaxAccumulator (needs an out-of-band min/max sub_type not stored
in the part) and the sketch-backed accumulator forms have no generic byte
factory, so their evicted-to-disk exact-agg portion is skipped in
reconstruct_exact_agg — they remain served from memory. Sketch query_range
read-back (KLL/HLL/DDSketch as opaque bytes) is already disk-aware and
unaffected.

🤖 Generated with Claude Code