fix(precompute-engine): warm-tier persists ingested sketches under queryable key

[zzylol]

Summary

Sweep blocker #2 diagnostic claimed the warm-tier query path returned empty
even when sketches were arriving at the backend worker
(worker_process_accumulator log lines fire ~8000 times but
Metric ... not found in store). Three hypotheses were proposed:
(A) streaming.yaml grouping/rollup mismatch, (B) _quantile metric-name
suffix mismatch, (C) worker → per_key store persistence regression.

Verification verdict — at the unit level all three hypotheses are
disproven, and three new pinning tests prove the existing
process_accumulator_input → window-close → emit_batch → per_key store
path is correct given that timestamps advance enough to close windows.

What was checked

• (B) per_key store is keyed by aggregation_id only
  (per_key.rs:817 store_key = output.aggregation_id). The metric field
  is diagnostic-only. So _quantile suffix in the agent emit cannot
  cause a query miss as long as the agent's metric matches the
  streaming.yaml's metric for agg_id=1. Not the bug.

• (A) backend's extract_group_key reads only grouping_labels —
  rollup_labels is parsed by aggregation_config.rs but never read in
  precompute_engine/ (verified via grep). 1000 per-tuple agent
  sketches with grouping: [zone] collapse into ~zones-many groups,
  each merged via AggregateCore::merge_with. The new
  test_grouping_labels_roll_up_per_tuple_sketches pins this. Not the bug.

• (C) the new test_process_accumulator_input_persists_after_window_close

  • test_sketch_ingest_persists_and_query_returns_non_empty exercise the
    full ingest→worker→SimpleMapStorePerKey→query_precomputed_output
    round-trip with real DDSketchAccumulator values. They pass on
    origin/main, so the persistence pathway itself is not regressed.

What this PR ships

A pure test-only addition (no production-code change) that pins the
warm-tier sketch persistence contract so any future regression to:

• WorkerMessage::AccumulatorInput handling in worker.rs,
• the sketch_panes → merge_sketch_panes_for_window → emit_batch chain,
• or the per_key store's agg_id-keyed insert/query path

will trip a unit test instead of being silently masked by the existing
process_group_samples test coverage (which never exercised the sketch path).

The diff is +302 / -0 lines, contained entirely to the existing
worker.rs::tests module — no new files, no API surface changes.

Hypothesis verdict (per task spec)

• (A) rollup mismatch: NO — rollup_labels is unused in
  precompute_engine; backend rollup happens implicitly via
  grouping_labels. Test pins the contract.
• (B) _quantile suffix: NO — store is keyed by agg_id, not metric.
• (C) persistence regression: NO — full round-trip works in unit test.

The actual root cause is therefore NEITHER (A), (B), NOR (C). Per the
task spec's escalation rule, I'm pinging the user.

Most likely live root cause (ping the user)

Given that the unit-level path is provably correct, the live e2e blocker
is most likely one of these production-only conditions, which a unit test
cannot reproduce:

1. Event-time watermark stagnation. process_accumulator_input only
   closes windows when closed_windows(prev_wm, current_wm) returns a
   non-empty set, which requires current_wm to advance past
   pane_start + window_size_ms. If the agent stamps every sketch with
   the same time_unix_nano (e.g. window-start instead of flush-time),
   the wm stalls. flush_all only adds +1ms to the wm, which is
   insufficient to close a 30s window. A wall-clock-based fallback in
   flush_all would defend against this. That is a design change so
   I have not made it; flagging here per task escalation rule.

2. Schema barrier dropping writes silently. A reconcile transition
   could move agg_id=1 to non-Active, after which is_writable(1) is
   false and route_modified_otlp_sketches_to_precompute (otel.rs:750)
   buckets the sketch into barrier_drops instead of forwarding to the
   worker. But this contradicts the diagnostic's claim that
   worker_process_accumulator{agg_id=1} fires 8000 times. So unlikely.

3. OTLP decode failure path. If the agent's time_unix_nano is 0
   or unset, (0 / 1_000_000) as i64 = 0 becomes the timestamp, all
   sketches land in the pane at pane_start_for(0), and unless a later
   batch advances the wm past 0 + 30s = 30000, no window closes.
   This is a real edge case; the proposed wall-clock fallback (1) covers
   it too.

Open follow-ups (separate from this PR)

• rollup_labels is parsed from streaming.yaml but unused in
  precompute_engine. This is a UX trap: operators who write
  rollup: [rack, node, pod] expecting it to drive backend rollup
  semantics get the right answer for the wrong reason (it's
  grouping_labels doing the work). Worth either deleting the field
  from the YAML schema or wiring it into capability matching properly.

• flush_all's +1ms watermark advance is a no-op for stuck-event-time
  scenarios. Adding a wall-clock fallback would close idle windows
  even when event-time freezes — but this is a watermark-semantics
  design change, escalated above.

Test plan

• cargo test --release -p query_engine_rust --lib precompute_engine — 66 passed (was 63 on main, +3 new)
• cargo test --release -p query_engine_rust --lib — 794 passed / 34 failed (matches main's 791 passed / 34 failed; +3 new tests, zero regressions)
• No production-code change → no risk of breaking an existing caller
• Sweep agent will re-run live e2e on retry to validate

🤖 Generated with Claude Code