eval(E0): wire workloads + controller-backend link + plan_transition fix

[zzylol]

Summary

Lands four fixes in one bundle so the E0 single-cell smoke (P5–P9 against a live N=1 b3-delta stack) produces non-empty answers and exercises the full P1–P9 wiring end-to-end. This is the upstream gate for the 60-cell sweep (E3) and the headline Pareto figure (E5).

Fixes

1. deploy/scripts/plan_transition.py — rename self._stop → self._stop_event. Python 3.12's threading.Thread has a private _stop() method that Thread._wait_for_tstate_lock calls during join(). Shadowing it with an Event instance raised TypeError: 'Event' object is not callable on every join.
2. deploy/configs/workloads.yaml (new) — declarative registry with 4 entries matching queries-e2e.json's query families against the metrics deploy/fake-exporter actually emits (http_requests_total, http_requests_total_latency_ms). Without this the controller logged workloads file not found; using empty registry and never published a plan, so before_plan / after_plan were always null in transition.jsonl.
3. controller/src/metrics_exposer.rs — exposes asap_active_plan_id{metric, plan_id="<hash>"} per-plan at scrape time, hashed off the plan's content (sketch_type / mode / delta_transmission / window / backend merge / valid_until). plan_transition.py's PlanIdTracker already polls for this exact metric name; with this metric present it picks up before_plan and detects re-plans by hash flip.
4. deploy/docker-compose/e2e-overlay.yml — mounts the workloads YAML into the controller at /etc/asap/workloads.yaml, sets CONTROLLER_WORKLOADS to point at it, and sets CONTROLLER_BACKEND_ENDPOINT=http://backend:9091/api/v1/streaming-config so the replanner pushes StreamingConfig YAML to the backend on every plan change. The default-path lookup workloads.yaml resolves to /workloads.yaml in the debian-slim runtime image (no WORKDIR), so the env-override is required.

Knock-on fix

deploy/scripts/e2e_plots.py::fig_transition crashed with TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' when a cell's t_plan_ready / t_first_hit / t_steady were null (matplotlib bar geometry can't position a None edge). Single-cell smokes against an idle controller hit this every time. Now skips that figure cleanly when timing data is missing and continues to fig_latency_cdf so query_latency_cdf.png lands.

Smoke results (cell cell-smoke-e0, b3-delta, N=1, card=1000, freq=10 Hz, 60 s soak)

Exit criterion	Status	Detail
(1) replay non-error, 300 rows, non-null result per family	FAIL	185 rows (slow queries dominated 60 s wall); 34/185 http_error on sum(http_requests_total) from cold-fallback torn-line bug; quantile / topk / count_unique succeed but return [] because backend's warm-tier inference patterns don't match these queries (see backend-inference.yaml: only quantile_over_time(...[1m]) patterns are wired). Real blocker — see notes below.
(2) transition.jsonl non-null before_plan / after_plan	PASS	before_plan=pe2f92ae12434e5c9, after_plan=pe2f92ae12434e5c9. Same hash since the transition query doesn't trigger a re-plan in the current controller (no workload-driven re-plan path for capability misses) — but the observability is now wired and a real plan flip would be visible.
(3) sample.jsonl populated 1 Hz CPU/mem/net	PASS	408 rows × 51 ticks across 8 containers (fake-exporter, agent, gateway, backend, controller, prometheus, grafana, minio).
(4) cold-truth/ hour-bucketed JSONL	PASS	5.9M rows across http_requests_total + http_requests_total_latency_ms under cold-truth/<metric>/2026/05/05/18/part-fake-1.jsonl.
(5) accuracy.csv rows for all 5 sketch types with non-NaN error	PARTIAL (b3-delta scope)	185 rows present with truth values + plan_id; error / recall columns empty because the backend returned [] for warm-tier (criterion 1). Per the task's scope note, full 5-sketch coverage requires run_e2e_sweep.sh (E3); b3-delta only exercises DDSketch + HLL.
(6) plots/query_latency_cdf.png + ideally pareto_acc_vs_thru.png	PASS	Both produced (plus bandwidth_vs_n.png). transition_timeline.png skipped cleanly with the e2e_plots.py fix above.

Smoke artifacts (per-run state, not committed): /tmp/cell-smoke-e0/{replay.jsonl,transition.jsonl,sample.jsonl,cold-truth/,accuracy.csv,plots/}. Reproduce with the protocol now in deploy/README.md "E0: single-cell smoke".

Real blockers surfaced (criterion 1)

• Backend warm-tier inference config doesn't match queries-e2e.json. histogram_quantile(0.99, sum by (le) (...)), topk(...), count(count by (zone) (...)), and sum(http_requests_total) all fall through simple_engine's pattern matcher. Closing this means either expanding backend-inference.yaml patterns (out-of-scope wiring change) or — better — fixing the engine's pattern matcher to handle sum by (le) and bare aggregations. Tracked in PROGRESS follow-up KLL Aggregator Implementation #4.
• Cold fallback's JSONL parser is intolerant of torn last lines. Already a known issue (PROGRESS follow-up KLL in otel-client #5). Causes the 10 s timeouts on sum(http_requests_total) because that query routes to cold and dies on a partial last-line write from the producer.
• No re-plan trigger for capability misses. The controller publishes the workload-registry plan at startup but never updates it; a plan-transition-aware test needs the Replanner to wake up on capability_miss POSTs from the backend (PROGRESS follow-up: see replan.rs::handle_violation). For E0 we observe before_plan == after_plan, which is technically PASS for (2) but the actual transition path is untested.

Stack state at end

UP at compose project docker-compose (default), with overlays:
base.yml + agents-N1.yml + baseline-b3-delta.yml + e2e-overlay.yml,
AGENT_CONFIG=sketchcol-agent-b3-delta.yaml,
EXPORTER_CARDINALITY=1000,
EXPORTER_FREQ_HZ=10.

(The b3-delta overlay's default 100k × 100 Hz saturates the OTLP exporter's 64 MiB max-message-size on the first DDSketch full-state window. Pinned to 1k × 10 Hz for E0 — documented in the README protocol.)

Constraints honored

• No edits to deploy/scripts/measure-baseline.py, deploy/scripts/run_e2e_sweep.sh, processor/*sketchprocessor/, deploy/configs/prometheus.yml (E1's domain).
• No edits to opentelemetry-go-patch/sdk/metric/, deploy/fake-exporter/ (E2's domain).
• No edits to asap-precompute-rs/.
• Smoke output JSONL/PNGs not committed.

Test plan

• python3 -c "import deploy.scripts.plan_transition" succeeds (verified via worktree-relative import).
• cargo build --release --bin controller clean (37 s, 0 errors).
• controller binary exposes asap_active_plan_id at /metrics after startup with CONTROLLER_WORKLOADS set.
• replay.jsonl non-empty with non-null plan_id on every row.
• transition.jsonl non-null before_plan and after_plan.
• cold-truth/ populated.
• accuracy.csv has rows for quantile / topk / count_unique / sum.
• plots/query_latency_cdf.png lands.
• Stack remains UP at end (verified via docker ps).

🤖 Generated with Claude Code