eval(fig7): cold-fallback latency arm — cold/archive-answered PromQL (was blocked)

[zzylol]

What

Adds the Fig 7 cold-fallback latency arm — the p50/p95/p99 latency of cold/archive-answered PromQL — which was previously BLOCKED ("archive/ship fragility, MinIO stayed empty"). Measured live end-to-end through the gorilla cold tier. The warm numbers are kept intact.

Result (real gct, 15 QPS, guard-verified)

600/600 timed queries served by data_source=thanos_query (the cold/archive engine), 0 empties, 0 errors:

arm	p50	p95	p99	n
warm (unchanged)	18.25	19.45	20.03 ms	599
cold-fallback (all)	22.28	47.17	67.07 ms	600
cold quantile_over_time (1000-series, Thanos over raw samples)	24.27	48.59	68.41 ms	400
cold sum (lossless archive)	15.48	33.33	42.74 ms	200

Overall p50 ≈ 1.2× warm, p99 ≈ 3.3× warm — the cold path stays in the tens of ms (no order-of-magnitude blowup) with a heavier p99 tail (data-plane → thanos-query → gorilla-merger StoreAPI + store-gateway, re-evaluating PromQL over raw Gorilla-XOR samples). Single-node loopback (server-side latency only, no network RTT).

Why it was blocked, and the fix

The original cold arm's agent set only cold:{enabled:true} with no ship_endpoint — the asapedge processor treats an empty endpoint as drain-only, no shipping (config.go: "Empty => drain-only (no shipping)"). So the cold encoder accumulated samples but never POSTed them → MinIO/merger stayed empty.

With a complete cold block (ship_endpoint → gorilla-merger, block_duration, non-empty external_labels), the edge ships 1000-series ASAPFRG1 fragments to the merger (cold ship is decoupled from the control channel — PR #500). Verified: per-shard cold drain/shipBatch shipped without spool/fail, then thanos served count(...)=1000.

How the cold path is forced & verified

• backend-storage-routing-coldon.yaml pins google_cluster_2019_cpu_rate → gorilla_object_store, so its instant queries dispatch to the ThanosQueryEngine (data_source=thanos_query).
• cold_latency_replay.py pins the PromQL eval timestamp to the cold-window anchor (the cold window sits at a fixed past instant once the ~one-window ship lands), so every cold query deterministically hits the archived window.
• The control-plane is stopped during timing — it periodically re-POSTs a storage-routing table that overrides the file cold table back to warm.
• compute_latency.py hard-fails if any cold query was not thanos_query (or any warm query not asap_query), so a warm shortcut can't masquerade as cold.

Files

New: stack-coldon.sh, agent-cold-ship.yaml, backend-storage-routing-coldon.yaml, cold_latency_replay.py, queries-latency-cold.json, per_query_latency_cold.json. Updated: compute_latency.py (both arms + per-arm data_source guard), latency_RESULTS.md (new Cold-fallback arm section + reproduce), latency_cdf.png + latency_summary.json (warm vs cold), docs/evaluation-plan-figures.md Fig 7 (was "cold-fallback arm blocked").

🤖 Generated with Claude Code