eval: Prometheus client profiling sweep + findings

[GnaneshGnani]

First pass of producer-side Prometheus client_golang profiling, stacked on #195.

This PR answers one narrow question:

If the app exposes metrics with the Prometheus Go client, how much CPU/memory is spent before anything reaches the collector?

The measured path is only:

synthetic app -> Prometheus Go client -> /metrics scrape

The collector, gateway, backend, remote-write path, and sketch processors are intentionally outside the measurement. That makes this comparable to #195 in scope: both PRs measure source-side cost before downstream systems get involved.

Headline Result

At high cardinality and high update rate, the main cost is not scraping. It is per-event dynamic label lookup through WithLabelValues.

Scenario	Cached metric handles	Dynamic WithLabelValues
update-only, C=10000, F=1000	3.944 CPU cores	31.688 CPU cores
combined, C=10000, F=1000, S=1s	4.600 CPU cores	31.723 CPU cores
combined, same cell, scrape p95	67 ms	172 ms
scrape-only, C=10000, S=1s, scrape p95	55 ms	58 ms

Interpretation: cached Prometheus metric handles look viable for high-rate producers. Calling WithLabelValues on every event becomes the dominant producer-side cost at high cardinality.

What This Adds

Code and harness:

File	What changed
deploy/fake-exporter/	Adds EXPORTER_CLIENT=prometheus, Prometheus instruments, /metrics, and /debug/pprof/*
deploy/scripts/run-prom-client-profile-cell.sh	Runs one profiling cell
deploy/scripts/run-prom-client-profiling-eval.sh	Runs the full profiling matrix
deploy/scripts/prom-client-cell-summary.py	Summarizes docker stats, scrape timings, and pprof artifact paths into CSV

Checked-in results under deploy/eval-results/prom-client/ mirror the artifact style in #195: findings, summary CSV, and top-level run log.

File	What it holds
FINDINGS-20260427.md	Short interpretation, key results, profile notes, caveats
prom-client-profiling-20260427-151128.csv	60 profiling cells
prom-client-profiling-20260427-151128.log	Full run log from the tmux-backed run
.gitignore	Keeps per-cell logs and pprof artifacts local-only

Generated locally but not checked in:

Local path	What it holds
profiles/*.cpu.pb, profiles/*.heap.pb	Raw pprof captures for each cell
profiles/*.top.txt	go tool pprof -top summaries
logs/*	Per-cell container logs, scrape timings, and docker stats snapshots

How To Read The Experiment

The run has three phases because Prometheus client cost has two different sources: producer updates and scrape exposition.

Phase	What is running	Why it exists
update-only	App updates Prometheus metrics; no scrape loop during the CPU profile	Isolates the cost of metric update calls
scrape-only	Metrics are preloaded; EXPORTER_FREQ_HZ=0; scrape loop runs	Isolates /metrics collection and text exposition
combined	App updates metrics while /metrics is scraped	Measures the realistic interaction between updates and scraping

The two update modes model the most important Prometheus client usage choice:

Mode	What it does	Why we measure it
cached	Pre-creates per-series metric handles and updates those handles directly	Best-practice path for known/high-cardinality series; avoids label lookup per event
dynamic	Calls WithLabelValues(...) on every event	Common and convenient code path; expected to be more expensive at high rate

The matrix was:

Axis	Values	Reason
Cardinality C	1000, 10000	Same scale as the #195 SDK-cost runs; shows low vs high series count
Per-series frequency F	10, 100, 1000 Hz	Tests moderate through stress update rates
Scrape interval S	1s, 15s, 60s	Covers aggressive scraping, common default-ish scraping, and low-frequency scraping
Instruments	counter,gauge	Matches the fake exporter signals used by #195: event count and latency value
Per-cell timing	SOAK_S=60, PROFILE_SECONDS=30	Lets the process warm up before collecting a profile without making the first pass overnight

histogram support is implemented in the exporter but not included in this first run. The first pass keeps counter,gauge so the workload matches the #195 source-side signals; histogram profiling can be a follow-up if we want bucket update cost specifically.

Result Tables

Update-only isolates producer metric updates:

Mode	C	F	CPU cores	RSS MiB
cached	1000	10	0.050	14.190
cached	1000	100	0.219	13.070
cached	1000	1000	1.558	12.750
cached	10000	10	0.249	61.470
cached	10000	100	1.388	59.630
cached	10000	1000	3.944	61.540
dynamic	1000	10	0.063	13.910
dynamic	1000	100	0.356	13.720
dynamic	1000	1000	2.250	13.300
dynamic	10000	10	0.409	60.080
dynamic	10000	100	2.586	61.290
dynamic	10000	1000	31.688	62.660

Combined with 1-second scraping shows the realistic stress case:

Mode	C	F	CPU cores	RSS MiB	Scrape p95 ms
cached	1000	10	0.042	17.840	19
cached	1000	100	0.298	17.560	21
cached	1000	1000	1.220	17.680	18
cached	10000	10	0.251	90.710	59
cached	10000	100	1.300	92.730	62
cached	10000	1000	4.600	96.140	67
dynamic	1000	10	0.062	17.190	21
dynamic	1000	100	0.308	18.950	20
dynamic	1000	1000	1.965	18.140	15
dynamic	10000	10	0.369	91.040	64
dynamic	10000	100	2.514	94.050	63
dynamic	10000	1000	31.723	98.120	172

Scrape-only with 1-second scraping isolates /metrics exposition:

Mode	C	Scrapes	Bytes/s	Avg ms	P95 ms	RSS MiB
cached	1000	29/29	0.144 MB/s	17.6	20	12.410
cached	10000	28/28	1.391 MB/s	50.3	55	43.080
dynamic	1000	30/30	0.149 MB/s	16.1	20	12.830
dynamic	10000	28/28	1.391 MB/s	51.1	58	43.440

Profile Findings

The local pprof summaries explain the CPU gap:

• Dynamic high-rate cells are dominated by Prometheus label lookup and synchronization: MetricVec.GetMetricWithLabelValues, MetricVec.hashLabelValues, metricMap.getMetricWithHashAndLabelValues, and sync/atomic.(*Int32).Add.
• Cached high-rate cells avoid per-event label lookup. Their profiles are mostly runtime scheduling/timer work plus direct counter/gauge updates.
• Scrape-only profiles are much smaller and mostly show metric collection, sorting, text exposition, and GC.

Validation

• go test ./... in deploy/fake-exporter
• bash -n deploy/scripts/run-prom-client-profile-cell.sh deploy/scripts/run-prom-client-profiling-eval.sh
• python3 -m py_compile deploy/scripts/prom-client-cell-summary.py
• git diff --cached --check before commit
• Full PR-quality profiling run completed all 60 cells in tmux

Caveats

• This is N=1, so it is enough for a first-pass shape, not variance bars.
• docker stats --no-stream CPU values are point-in-time samples. The pprof files are the stronger source for hot functions, but they are kept local-only to avoid bloating the PR.
• Some high-cardinality dynamic cells logged transient connection resets during readiness checks. All 60 cells completed and wrote CSV rows.
• This does not measure collector sketch construction or merge cost. Those remain covered by the existing sketchcollector path and the eval: SDK cost — first sweep pass + findings #195 SDK profiling work.

Next

Use this pass as the Prometheus-client counterpart to #195. If we need paper-quality confidence intervals, rerun the same matrix multiple times. If we need Prometheus histogram-specific cost, rerun with EXPORTER_PROM_INSTRUMENTS=counter,gauge,histogram as a separate follow-up so the bucket-update cost is not mixed into this first comparison.