eval: fill nan columns in sweep CSV (paper blocker #3)

[zzylol]

Summary

• Fills the three nan columns in deploy/scripts/measure-baseline.py output that have been blocking paper claim KLL in telegraf aggreagtor #1 (bandwidth) and KLL in otel-client #5 (query latency) since Phase 2.11B: agent_in/out_kib_per_s on raw/Gorilla/Serf baselines, gateway_* + backend_samples_per_s for drop_original=true baselines, and backend_query_p99_ms whenever the soak doesn't drive queries.
• Three fixes layered into the existing measurement script (no new infrastructure, no rebuild of asap/sketchcol:dev):
  1. docker-stats agent net rx/tx fallback for agent_in/out_kib_per_s (captures on-the-wire bytes, which is what claim KLL in telegraf aggreagtor #1 actually wants — the patched processor counter measures in-process protobuf bytes; see the new docs/eval-instrumentation-notes.md for the caveat).
  2. or vector(0) on gateway PromQL + the gateway-egress fallback so B1/B5 (drop_original: true → no traffic) report 0 instead of NaN, while keeping NaN reserved for "Prometheus is gone".
  3. --replay-jsonl PATH flag computes backend_query_p99_ms client-side from promql_replay.py's JSONL output. Wired through run_e2e_sweep.sh (always-on) and run-baseline-sweep.sh (DRIVE_QUERIES=1 opt-in). Survives the docker compose down -v between cells, where the Prom histogram doesn't.

Verification

deploy/eval-results/sweep-smoke-postfix-20260505.csv is one fresh cell per baseline family (b0a, b0b, b1, b3-delta, b4, b5) at N=1, rate=1000, cardinality=1000. Zero NaN cells across the 17 columns × 6 baselines.

Coverage table per baseline (post-fix):

Baseline	agent_*_kib_per_s	gateway_*	backend_samples_per_s	backend_query_p99_ms
b0a-raw-stream	docker-stats	direct	gateway-egress fallback	client-side from replay JSONL
b0b-raw-batched	docker-stats	direct	gateway-egress fallback	client-side from replay JSONL
b1-serf	docker-stats	0 (drop_original)	0 (drop_original)	client-side from replay JSONL
b3-delta	direct (sketch)	direct	direct or fallback	direct or client-side
b4-tunable	direct (sketch)	direct	direct or fallback	direct or client-side
b5-gorilla	docker-stats	0 (drop_original)	0 (drop_original)	client-side from replay JSONL

Test plan

• All six baseline families produce 17 non-NaN columns in a single sweep cell. CSV checked into deploy/eval-results/sweep-smoke-postfix-20260505.csv.
• python3 -m py_compile deploy/scripts/measure-baseline.py clean; bash -n run-baseline-sweep.sh and bash -n run_e2e_sweep.sh clean.
• No regression in existing CSVs — the docker-stats fallback only fires when the patched-processor counter is NaN (sketch baselines unchanged), and the or vector(0) only applies when the underlying counter is missing.
• No asap-precompute-rs/src/otap/ or asap-precompute-rs/Cargo.toml touched (parallel agent's OTAP-codec work).

Open questions / surprises

• The patched-processor *_bytes_total family measures in-process protobuf bytes (via pmetric.ProtoMarshaler.MetricsSize) — not bytes-on-the-wire. They differ for raw paths because the wire bytes include gRPC framing + HTTP/2 headers that the in-process counter doesn't. Sketch payloads happen to overlap closely (~5%) because the typed proto envelope IS the wire payload. The doc spells out which one to use for which paper figure.
• asap_ingest_samples_total is NOT exposed by the current asap/query-backend:dev image — only query-side counters are. The fallback uses otelcol_exporter_sent_metric_points_total{job="gateway"} (gateway egress = backend ingest, modulo dropped batches). Proper fix is a backend-side counter; tracked as follow-up in ASAPQuery-backend.
• b5-gorilla and b1-serf show producer_cpu_cores ≈ 2 even with EXPORTER_SDK_AGG=default — much higher than b3-delta's 0.27. The producer measure is the same SDK so it shouldn't depend on the baseline; might be a docker-stats sample-window artifact at 5s on a busy host. Out of scope for this PR; flagged in the doc.

What's deliberately NOT fixed

• Backend-side asap_ingest_samples_total counter. Fallback works; proper fix is in the ASAPQuery-backend repo and a separate paper-blocker.
• Grafana dashboards for the five evaluation axes (mentioned alongside the NaN gap in PROGRESS.md blocker DDSketch in opentelemetry-client #3). Separate work; the column-population fixes here are the prerequisite.
• Producer CPU anomaly on raw baselines (~2 cores even at default agg) — measurement is correct, the value just looks high; flagged in the doc rather than tuned.

🤖 Generated with Claude Code