fix(precompute-engine): wall-clock watermark fallback closes idle windows

[zzylol]

Summary

Land the actual fix for sweep blocker #2 that PR #82 (just merged) explicitly punted as "watermark-semantics design change". PR #82 pinned that the post-window-close persistence pathway is correct given timestamps advance; the live e2e symptom (8000 sketch arrivals, 0 store entries, query empty) means timestamps don't advance enough to close 30s windows.

When the agent stamps every sketch with the same time_unix_nano (e.g. window-start instead of flush-time), flush_all's +1ms watermark advance is a no-op: closed_windows(prev_wm, prev_wm+1) returns empty forever, the window never closes, and warm-tier queries come back empty even though worker_process_accumulator keeps logging.

Fix shape

• GroupState now tracks each open pane's wall-clock birth time in a new pane_wall_clock_starts_ms: BTreeMap<i64, i64>. The first sample/sketch routed into a pane stamps it with the current wall-clock time.
• flush_all force-advances each group's effective_wm past pane_start + window_size_ms for any pane whose wall-clock age exceeds window_size_ms + wall_clock_grace_period_ms. The existing event-time path (closed_windows + merge_*_for_window + emit_batch) consumes the bumped watermark exactly as before — no duplicated close logic.
• Event-time-driven closure stays the primary path; wall-clock is only a fallback for stuck event-time.
• Monotonicity preserved: the fallback only ever pushes effective_wm forward, never backward.
• PrecomputeEngineConfig::wall_clock_grace_period_ms (default 5_000 ms — matching the existing allowed_lateness_ms default) tunes the grace. Setting it to <= 0 opts out and reverts to strict event-time-only semantics (pin test included).
• Worker carries an injectable now_ms_fn (default SystemTime::now-backed) so tests can drive the fallback in milliseconds rather than needing real-time sleep(35s).

Diff size

• Source LOC (precompute_engine + bin compile-fixes): ~62 actual code lines in worker.rs + ~7 in config.rs + 1 in engine.rs + 7 across bins/tests for the new PrecomputeEngineConfig field. Net actual code: ~70 LOC, well under the 100-LOC autopilot threshold. Heavy on doc comments because the failure mode is subtle.
• Test LOC: ~179 lines (2 new tests + a small make_worker_with_grace helper).

New tests

• wall_clock_fallback_closes_idle_window — ingests 10 DDSketches stamped with frozen event-time t=0, then advances a fake clock by window_size + grace = 35s and asserts flush_all closes + emits + persists the [0, 30_000) window with a DDSketchAccumulator carrying all 10 sketches. Re-asserts idempotency: a second flush_all with no new data does not re-emit.
• wall_clock_fallback_disabled_preserves_event_time_only_semantics — pins that wall_clock_grace_period_ms = 0 opts out of the fallback (event-time-only behaviour, matching pre-fix semantics). Even after a 24h fake-clock advance, no emit happens.

Subtleties

• Grace period choice: 5s default mirrors the existing allowed_lateness_ms = 5_000 default. Configurable via the YAML / CLI like every other PrecomputeEngineConfig field. Setting <= 0 is the explicit opt-out.
• Monotonicity: both the event-time boundary (propagated_wm + 1) and the wall-clock fallback only push effective_wm forward. Once a window closes via the fallback, its pane is drained from sketch_panes (and active_panes) and from pane_wall_clock_starts_ms via prune_pane_wall_clock_starts, so subsequent flush_alls have no candidate for re-closure. The "close at most once" invariant is preserved.
• Sliding windows: closed_windows(prev, force_to) naturally yields all windows that closed in the bumped jump, including overlapping ones for sliding-window configs. No special-case logic needed.
• Clock testability: chose a closure-based injection (Box<dyn Fn() -> i64 + Send + Sync>) over a trait so we don't introduce a new test pattern for a single use site. The set_now_ms_fn setter is #[cfg(test)]-gated so production code never touches it.
• Wall-clock-start timing: stamped at first pane touch, not at window-start. A pane that opens late (e.g. agent boot) gets now_ms matched to its actual creation time, not its event-time window-start. The fallback fires after window_size + grace of real elapsed time — which is the desired "the agent stopped progressing event-time" semantics.

Open question (separate hardening pass)

PR #82 pinned hypothesis (1) — frozen time_unix_nano — as the most likely production root cause. Hypothesis (3) flagged a related case: if the agent's time_unix_nano is 0, (0 / 1_000_000) as i64 = 0 becomes the timestamp, all sketches land in pane_start_for(0) = 0, and the wall-clock fallback now correctly fires at now - pane_birth >= 35s. Worth a separate look at decode_modified_otlp_sketch_bytes's timestamp handling — clock-skew between agent and backend could still produce panes whose pane_start is wildly different from now_ms, and the fallback's "wall-clock age" semantics implicitly assume the backend's clock is the authoritative one. For agents shipping batched sketches with their own clock embedded, that assumption is fine. For pathological clock skew, a max(event-time-wall-clock-equivalent, ingest-wall-clock) heuristic might be more robust.

Test plan

• cargo test --release -p query_engine_rust --lib precompute_engine::worker — 29 passed (all 3 PR fix(precompute-engine): warm-tier persists ingested sketches under queryable key #82 pin tests + both new tests + 24 prior).
• cargo test --release -p query_engine_rust --lib precompute_engine — 68 passed (was 66 on main; +2 new).
• cargo test --release -p query_engine_rust --lib — 796 passed / 34 failed (matches main's 794 / 34 with +2 new tests, zero regressions).
• cargo clippy --release -p query_engine_rust --lib --tests — 5 pre-existing warnings, zero new warnings from this PR's code.
• cargo build --release -p query_engine_rust — all bins build clean (added the new field at every PrecomputeEngineConfig literal).
• No submodule pointer drift.

🤖 Generated with Claude Code