fix: archive tier covers all 5 sketched metrics (HLL/CS/CMS/KLL/DDSketch all in Thanos)

[zzylol]

Summary

Fixes the post-#345 demo's archive_miss=1370 failure mode for the four counter-sketch metrics (unique_users_per_min, top_endpoint_qps, endpoint_request_freq, request_size_bytes).

Root cause (a): the backend's POST /api/v1/storage_routing handler is an atomic per-tenant SWAP — every push replaces the whole tenant's routing table. handle_plan emitted a single-element metrics:[…] document per call, so when the demo POSTed /api/v1/plan for each of the 5 sketched contract metrics in sequence, only the last metric's entry survived in the backend. The other 4 metrics defaulted to sketch_warm_tier (which has no warm-tier sketch state for archive-shape queries: count, topk, rate_post_hoc, histogram_quantile, delta, deriv, absent) → backend returned empty / 404 → accuracy_reduce.py logged archive_miss even though gorillas3 did write their TSDB blocks to MinIO and Thanos had them indexed.

Fix: thread per-metric BackendStageConfig through a new AppState.backend_routing_cache: Arc<Mutex<HashMap<String, BackendStageConfig>>>. Each plan-emit cycle updates the cache for the metric being planned and re-emits the storage-routing JSON from the union of all known plans. The next swap then preserves every previously seen metric's routing entry.

Scope: controller-only (controller/src/main.rs). Doesn't touch gorillas3, deploy/configs, or backend.

Evidence

Before (live demo stack with all 5 producers emitting):

$ curl -s 'http://localhost:19092/api/v1/label/__name__/values' | jq .data
[
  "http_freshness_probe_archive",
  "http_freshness_probe_raw",
  "http_freshness_probe_warm",
  "http_requests_total",
  "http_requests_total_latency_ms"
]

$ curl -s 'http://localhost:19091/api/v1/storage_routing' | jq .metrics_count
4   # only bootstrap YAML; controller's per-metric pushes erase each other

After (same stack, fix applied, controller rebuilt):

$ for m in http_requests_total_latency_ms request_size_bytes unique_users_per_min top_endpoint_qps endpoint_request_freq; do
    curl -s -X POST -H 'Content-Type: application/json' \
      -d "{\"metric_name\":\"$m\",...}" 'http://localhost:18080/api/v1/plan'
  done

$ curl -s 'http://localhost:19091/api/v1/storage_routing' | jq '{entries: .metrics_count, hash: .table_hash}'
{ "entries": 5, "hash": "a54a679c395b57f1" }

$ docker logs backend | grep "swap completed"
... entries=1 table_hash=...   # 1st metric posted
... entries=2 table_hash=...   # 2nd, cumulative
... entries=3 table_hash=...
... entries=4 table_hash=...
... entries=5 table_hash=a54a679c395b57f1   # 5th — all metrics preserved

$ curl -s 'http://localhost:19092/api/v1/label/__name__/values' | jq .data
[
  "endpoint_request_freq",         # ← CMS metric, archive-tier visible
  "http_freshness_probe_archive",
  "http_freshness_probe_raw",
  "http_freshness_probe_warm",
  "http_requests_total",
  "http_requests_total_latency_ms",
  "request_size_bytes",            # ← KLL metric
  "top_endpoint_qps",              # ← CountSketch metric
  "unique_users_per_min"           # ← HLL metric
]

Regression test

api_tests::storage_routing_cumulative_push_covers_all_5_sketched_metrics (in controller/src/main.rs):

• Spins up an axum mock backend that captures every POST /api/v1/storage_routing body.
• Posts /api/v1/plan 5 times for the 5 sketched contract metrics.
• Asserts the last body's metrics:[…] array contains all 5 metrics, each carrying a thanos_archive target.
• Asserts the cumulative count grows monotonically across pushes (1 → 2 → 3 → 4 → 5).

$ cargo test --bin controller storage_routing
running 16 tests
... all passing, including new test
test result: ok. 16 passed

What this PR does NOT fix

Out-of-scope failures observed during diagnosis but not within this PR's blast radius:

• OpAMP RemoteConfig push isn't applied by the agent (effective-config preview shows the static bootstrap YAML, not the controller-pushed routing-connector YAML). Agent stays on [gorillas3, ddsketch, batch] single-pipeline. This is upstream of the routing fix and orthogonal — the bootstrap config has gorillas3 first so all metrics still flow through it.
• Producer ResourceExhausted on default cardinality (grpc: received message after decompression larger than max 4194304): with EXPORTER_FIVE_SKETCH_USER_POOL=1000 × EXPORTER_CARDINALITY=500 the OTLP delta export overflows the agent's 4 MiB max_recv_msg_size. Tuning belongs in deploy/configs/asap-otel-agent-*.yaml (parallel agent's scope).

Test plan

• cargo test --bin controller storage_routing (all 16 pass, including new regression)
• Live stack: 5 sequential POST /api/v1/plan → backend storage_routing shows metrics_count=5 (was 1 pre-fix)
• docker exec ... mc ls -r local/asap-gorilla-tsdb/ shows TSDB blocks (gorillas3 unchanged, still writes for all metrics)
• curl thanos /api/v1/label/__name__/values shows all 5 sketched metrics plus freshness probes
• docker logs backend | grep "swap completed" shows entries grow monotonically across cumulative posts

Refs #46.

🤖 Generated with Claude Code