Skip to content

feat(measure-baseline): producer-side CPU / RSS / bytes-out columns - #191

Merged
zzylol merged 1 commit into
mainfrom
feat/measure-baseline-producer-columns
Apr 23, 2026
Merged

feat(measure-baseline): producer-side CPU / RSS / bytes-out columns#191
zzylol merged 1 commit into
mainfrom
feat/measure-baseline-producer-columns

Conversation

@zzylol

@zzylol zzylol commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Measurement infra for the three-axis SDK sweep (see docs/sdk-aggregation-three-axis-design.md, paper §6.2c). Without producer-side columns the sweep can tell you what the agent / gateway / backend see but not what the SDK decision actually costs the instrumented process — which is the whole point of the encoding ablation.

Adds three columns (placed right after the workload labels so they show up first in spreadsheets):

Column Source
producer_cpu_cores docker stats CPUPerc / 100 on the fake-exporter container
producer_rss_mib docker stats MemUsage on same
producer_bytes_out_per_s docker stats NetIO tx field, sampled twice separated by --bytes-sample-window (default 5s)

Why docker stats and not Prometheus

cAdvisor isn't wired into deploy/configs/prometheus.yml (only OTel targets are scraped), so container_network_transmit_bytes_total doesn't exist. Pulling from docker stats directly is the pragmatic source until someone wants to add a cadvisor service to the compose stack (a larger change I'm deferring).

Implementation notes

  • docker_stats() now parses the NetIO field — cumulative tx/rx per container.
  • docker_stats_with_bytes_rate(window_s) does the two-sample dance in one call, reused for backend CPU/RSS so the sweep pays one --bytes-sample-window pause per baseline instead of three separate docker stats trips.
  • --bytes-sample-window default 5s. Too short (≤1s) and the tx counter barely moves; too long and every sweep iteration gets expensive.
  • --producer-container CLI flag so renamed compose projects can override the default.

Test plan

Local smoke on a live N=1 b0a-raw-stream stack:

$ python3 deploy/scripts/measure-baseline.py \
    --baseline b0a-raw-stream --scale N1 --rate 1000 --cardinality 1000 \
    --bytes-sample-window 5

baseline,scale,rate,cardinality,producer_cpu_cores,producer_rss_mib,producer_bytes_out_per_s,agent_cpu_cores,...
b0a-raw-stream,N1,1000,1000,0.286,55.530,2506.056,nan,...

All three producer columns populate with sensible values.

Follow-up

§6.2a/b/c sub-sweeps at N=1 now have the measurement infra they need. That's the next PR.

🤖 Generated with Claude Code

@zzylol @claude
feat(measure-baseline): producer-side CPU / RSS / bytes-out columns
612a3d2 
The three-axis SDK sweeps (see docs/sdk-aggregation-three-axis-design.md,
paper §6.2c) need to measure what the SDK decision point costs at
the producer, independent of what the agent and backend do. The
current script only pulls agent / gateway / backend stats, so the
producer side — CPU spent building sketches, RSS held by the
raw-buffer, bytes actually leaving the fake-exporter container —
has been invisible.

Adds three columns, placed right after the workload labels so they
show up first in spreadsheets:

  producer_cpu_cores        docker stats CPUPerc / 100 on the
                            fake-exporter container.
  producer_rss_mib          docker stats MemUsage on same.
  producer_bytes_out_per_s  docker stats NetIO tx field, sampled
                            twice separated by --bytes-sample-window
                            (default 5s), divided by wall delta.

Implementation:
- docker_stats() now also parses the NetIO field, returning
  cumulative tx/rx per container alongside CPU% and mem MiB.
- New docker_stats_with_bytes_rate(window_s) does the two-sample
  dance in one call — reused for backend CPU/RSS so the sweep
  only pays one 5s window per baseline iteration instead of
  three trips through `docker stats`.
- Size-parser helpers split out for readability (kB / MB / GB /
  KiB / MiB / GiB).
- --producer-container CLI flag so sweeps with renamed compose
  projects can override the default
  "docker-compose-fake-exporter-1".

Local smoke test on N=1 b0a-raw-stream with the three-axis
fake-exporter binary running at defaults:

  producer_cpu_cores       = 0.286
  producer_rss_mib         = 55.5
  producer_bytes_out_per_s = 2506

(cAdvisor isn't in the compose stack — see
deploy/configs/prometheus.yml — so we can't pull these from
Prometheus directly. docker stats is the pragmatic source until
we add a cadvisor service, which is a larger compose change.)

Follow-up: §6.2a/b/c sub-sweeps at N=1 now have the measurement
infra they need.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Apr 23, 2026
Closed
@zzylol
zzylol merged commit 46d7505 into main Apr 23, 2026
@zzylol
zzylol deleted the feat/measure-baseline-producer-columns branch April 23, 2026 17:50
zzylol added a commit that referenced this pull request Apr 23, 2026
@zzylol @claude
feat(deploy): run-three-axis-sweep.sh — (W, L, agg) grid driver
641f789 
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.

Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:

  6.2a time:     WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
  6.2b label:    WINDOWS="60s"  PROJECTIONS=": zone,rack zone -"
                 AGGS="dd-full"
  6.2c encoding: WINDOWS="60s"  PROJECTIONS="zone,rack"
                 AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"

Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
  labels" (the natural empty-string would be awkward to pass
  through a space-separated list) and "-" for "drop all".
  decode_projection() maps those back to what
  fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
  (OTLP → batch(1s) → OTLP), which keeps the agent from running
  a sketch pipeline on top and conflating the SDK-side
  measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
  is trivially splittable back into the grid for plotting.

Smoke-tested locally against the PR #189/#190/#191 stack:

  $ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
    CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
    ./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv

  baseline,scale,rate,cardinality,producer_cpu_cores,...
  w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...

Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.

Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 23, 2026
@zzylol @claude
feat(deploy): run-three-axis-sweep.sh — (W, L, agg) grid driver (#193)
6f0976d 
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.

Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:

  6.2a time:     WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
  6.2b label:    WINDOWS="60s"  PROJECTIONS=": zone,rack zone -"
                 AGGS="dd-full"
  6.2c encoding: WINDOWS="60s"  PROJECTIONS="zone,rack"
                 AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"

Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
  labels" (the natural empty-string would be awkward to pass
  through a space-separated list) and "-" for "drop all".
  decode_projection() maps those back to what
  fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
  (OTLP → batch(1s) → OTLP), which keeps the agent from running
  a sketch pipeline on top and conflating the SDK-side
  measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
  is trivially splittable back into the grid for plotting.

Smoke-tested locally against the PR #189/#190/#191 stack:

  $ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
    CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
    ./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv

  baseline,scale,rate,cardinality,producer_cpu_cores,...
  w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...

Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.

Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Apr 23, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol