eval: SDK cost — first sweep pass + findings - #195
Merged
Merged
Conversation
Runs the §6.2 a/b/c sub-sweeps defined in
docs/sdk-aggregation-three-axis-design.md at CARDINALITY=1000,
FREQ_HZ=10, N=1, SOAK_S=180, BYTES_WIN=20. 17 cells total,
~65 min wall clock.
Artifacts:
- deploy/scripts/run-paper-6.2-sweeps.sh — wrapper that runs
all three sub-sweeps at named env points and drops CSVs.
- deploy/eval-results/three-axis/6.2a-time-20260423.csv
- deploy/eval-results/three-axis/6.2b-label-20260423.csv
- deploy/eval-results/three-axis/6.2c-encoding-20260423.csv
plus .log files for each sub-sweep stderr.
- deploy/eval-results/three-axis/FINDINGS-20260423.md — results
interpretation with three sections:
- Clean findings (encoding-axis ratios, label-axis RSS drop)
- Methodology caveat (BYTES_WIN < W under-counts bytes)
- Surprises worth re-checking (§6.2b CPU-up-with-projection,
cms-delta memory overhead)
Key clean findings:
- §6.2c encoding: raw-buffer holds 108 MiB RSS vs 33-39 MiB for
sketch encodings (~3× memory cost for per-event storage) and
emits ~7× the wire bytes of any sketch. cms-delta costs 2×
the RSS of cms-full because it holds prev-tick snapshot for
diff.
- §6.2b label: projecting to zone,rack cuts producer RSS from
66.8 → 33.7 MiB as attribute-set folding reduces sketch
instance count.
Known caveat: BYTES_WIN=20s < W=60s in most cells, so
producer_bytes_out_per_s is under-reported by factor ~W/BYTES_WIN.
Ratios within a sub-sweep (all at same W) are unaffected. V2
rerun with BYTES_WIN ≥ 2W will fix absolute numbers — doc'd in
FINDINGS as the top-priority follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops paper-section references from the findings doc so it reads
as experimental results standing on their own, not as "§6.2
sub-sweep output". Structure is now:
1. Experiment design — what the (W, L, agg_type) triple means,
what each sub-experiment varies vs holds fixed, how each
cell is measured.
2. Methodology caveat (unchanged — BYTES_WIN < W under-reports).
3. Results per sub-experiment (encoding axis, label axis,
time axis) with the numbers and what to make of them.
4. Non-obvious findings worth a second look.
5. Next actions.
No numeric content changed; CSVs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 'What W actually is (and push vs pull)' subsection right after the (W, L, agg_type) definition so the doc doesn't assume the reader knows OTel's Reader/PeriodicReader model. Compares OTLP push and Prometheus pull aggregation-equivalence and calls out the two places they actually differ for this project (NAT-friendliness + controller-reconfig channel). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md all still had AggregationRawBuffer flagged ❌. It shipped in #189 back in April — updating the status rows + cross-referencing the contract test (deploy/fake-exporter/sdk_emit_test.go) + the implementation path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrites the AggregationRawBuffer subsection in the design doc to
explain *why* raw-buffer is a first-class encoding slot (not just
what it does). Key points the previous version didn't make:
- Every other upstream Aggregation is lossy by design; raw-buffer
is the only SDK-native option that preserves the full
observation stream. Without it there's no reference point
for sketch-vs-raw claims.
- It's also the encoding that can serve queries no summary
can: exact events for cold-fallback replay, per-sample audit,
downstream sketch computation the SDK policy didn't
anticipate.
- Memory cost scales as O(W × event_rate × cardinality) —
the tradeoff the encoding axis measures.
Contract + overflow + temporality specifics kept from the
earlier version and folded into dedicated subsections.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename `docs/sdk-aggregation-three-axis-design.md` → `docs/sdk-aggregation-cost.md`. Title updated to 'SDK aggregation cost'. Old name was a framework description; new name says directly what the doc is measuring. - Update every cross-reference in docs, PROGRESS.md, TODO.md, deploy/ (compose, Dockerfile, go.mod, main.go, scripts, FINDINGS) to point at the new filename. - Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status tables and TODO checkboxes. Merged to main = ✅; the dates and PR numbers are git-log territory, not running-doc territory. No substantive content change; only the framing/file name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
File: docs/sdk-aggregation-cost.md → docs/sdk-cost-evaluation.md Title: 'SDK cost evaluation' All cross-references in docs/, PROGRESS.md, TODO.md, deploy/ updated. No content change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Delete docs/n10-bottleneck-rca.md — the finding (PR #185's 'N=10 throughput collapse' is SDK pre-aggregation, not a bottleneck) has propagated into the docs that still need it (TODO §1 and the SDK cost evaluation doc summarise the conclusion inline). The long write-up was one-shot debug context, not a reference. - Drop the dangling link from docs/sdk-cost-evaluation.md's header block (and the 'Authoritative for §6.2…' framing — redundant with the doc content). - Fix the other references that pointed at the deleted file: TODO.md §1, PROGRESS.md outstanding-aggregators note, fake-exporter main.go comment and EXPORTER_RATE deprecation log, docker-compose/base.yml comment. Each now explains the point inline. - Rework the kll-delta status in both docs/sdk-cost-evaluation.md and PROGRESS.md from '❌ not yet' → '**no need**'. kll-full covers the encoding-axis comparison; implementing a KLL-specific delta would need a different strategy than the byte-diff the other four sketches use, and the payoff doesn't justify it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…w-up list
Across-the-board cleanup so evaluation docs stand on their own and
don't cite paper section numbers. Evaluations describe design +
methodology + results directly; paper framing belongs only in
docs/paper-outline.md.
- Rename deploy/scripts/run-paper-6.2-sweeps.sh →
run-sdk-cost-sweeps.sh. Filename drops the §6.2 embedding.
- Delete docs/design-stateful-protocol.md's "(Paper §4.5)" title
suffix; title is now just the protocol name.
- docs/sdk-cost-evaluation.md:
- remove the obsolete "Implementation order (follow-up PRs)"
section — most items are merged, and the remaining one
lives in TODO.md anyway
- remove the "Non-goals of this doc" section — pointer list
was stale
- drop the paper-§6 framing in the opening; doc now reads
as standalone evaluation design
- rewrite the ablation table: instead of "6.2a Time axis /
6.2b Label axis / …" with "Claim" column, plain "Time /
Label / Encoding / Combined" with "What the sweep
measures" column
- strip the few remaining §6.2 / §6.5 inline refs
- Strip paper-§N from every config / compose / script / TODO /
PROGRESS comment where it appeared. Specifically:
- PROGRESS.md outstanding-aggregators section
- TODO.md §1 header (now "SDK cost evaluation") + all inline
§6.2 references
- deploy/TODO.md instrumentation section headings + inline
- deploy/README.md "Paper §6 mapping" → "Evaluation → metric
mapping"; drop "paper's §6 eval" framing
- deploy/configs/*.yaml — comment prefixes like "Paper §6.2
baseline:" → plain "Baseline:"
- deploy/configs/prometheus.yml, grafana-datasources.yml —
same
- deploy/docker-compose/*.yml — "Paper §6" → "Scale dials"
etc.
- deploy/helm/asap/{values,Chart}.yaml — same
- deploy/fake-exporter/main.go — "Paper's W axis" / "L axis"
/ "encoding axis" → "Time axis" / "Label axis" / "Encoding
axis"
- deploy/fake-exporter/sdk_emit_test.go — §6.2c → encoding-axis
- Left in place: internal design-doc cross-references like
"delta-transmission-design.md §16.2" where the §N refers to a
section of a design doc rather than the paper. Those are legit
technical writing, not paper citations.
No substantive content or behaviour change anywhere; pure
framing/naming cleanup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- deploy/eval-results/three-axis/ → deploy/eval-results/sdk-cost/
to match the design doc (docs/sdk-cost-evaluation.md).
- CSV + log files lose the paper-section prefix:
6.2a-time-20260423 → time-axis-20260423
6.2b-label-20260423 → label-axis-20260423
6.2c-encoding-20260423 → encoding-axis-20260423
- Update every reference in
docs/sdk-cost-evaluation.md, TODO.md,
deploy/scripts/run-sdk-cost-sweeps.sh, and the FINDINGS doc.
Data unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
changed the title
The previous 'Why this baseline exists' blurb framed raw-buffer as a generic reference point but didn't pin down (a) that OTel's native aggregators are lossy *specifically over the emit period W*, or (b) that the experiment's value comes from running raw-buffer at the *same* W as the sketch encodings so the comparison is apples-to-apples. Rewrite leads with those two observations, then closes on the 'without it, the ablation collapses into sketch vs nothing' line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous 'Adding Aggregation<X>Delta (×5)' block read like a
PR checklist — semantics / encoding / LOC estimate. This is a
design doc, not a task tracker.
Rewrite leads with *what* delta encoding is (full state vs. sparse
diff of changed cells; aggregator keeps prev-tick state; full
re-emit on L / param change) and *why* it deserves a distinct
slot in the encoding-axis evaluation:
1. The bandwidth claim 'sketch < raw' has two independent
factors (sketch payload vs raw, and delta vs full sketch).
Measuring *-full and *-delta as separate rows on the
encoding axis lets each factor be attributed directly.
2. Delta is a memory-for-bandwidth trade: the aggregator
holds prev-tick state alongside current, roughly doubling
RSS. That tradeoff belongs next to the bandwidth savings
in the encoding row, not buried as an implementation note.
Also notes that KLL is intentionally absent from the delta column
and points at the existing note above for the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AggregationRawBuffer design → AggregationRawBuffer Aggregation<X>Delta — what it is and why it's a separate slot → Aggregation<X>Delta Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Last paper-shaped artefact in the cost-evaluation tooling. The inner grid driver is now 'run-sdk-cost-sweep.sh' (singular — runs one sweep over the grid the caller specifies), and the outer wrapper stays 'run-sdk-cost-sweeps.sh' (plural — runs the three canonical sub-sweeps). Updates every reference in the evaluation doc, findings, TODO, and the wrapper script itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- run-sdk-cost-sweep.sh → run-sdk-cost-grid.sh
Runs a W × L × agg grid the caller specifies via env.
- run-sdk-cost-sweeps.sh → run-sdk-cost-eval.sh
Runs the three canonical sub-sweeps (time / label / encoding)
that make up the SDK cost evaluation; calls run-sdk-cost-grid
three times with different fixed/swept axes.
The singular-vs-plural distinction was too subtle; 'grid' vs
'eval' spells out the role difference directly.
Updates references in docs/sdk-cost-evaluation.md, FINDINGS,
TODO.md, and the eval script's own call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two top-level docs were covering the same ground — PROGRESS
drifted into historical session notes, TODO duplicated the
'what's landed' summary and added 'what's left'. Consolidated
into a single PROGRESS.md with three sections:
- Implemented — ground-truthed against current code
- Outstanding — paper blockers + non-blocker SDK work
- Future work — post-paper
Top-level TODO.md deleted. Cross-references in deploy/TODO.md
and deploy/README.md updated to point at PROGRESS.md instead
(the local 'See TODO.md in this directory' ref in README refers
to deploy/TODO.md, kept as-is).
Sketch-processor and aggregator status tables verified against
the patch trees:
opentelemetry-go-patch/sdk/metric/aggregation.go lists all six
Aggregation types we document (DDSketch/KLL/CS/CMS/HLL +
RawBuffer), matching the table.
opentelemetry-collector-contrib-patch/processor/ has the five
per-sketch processors + two new *mergeprocessor variants
(countsketchmerge, countminsketchmerge) surfaced in the
table.
deploy/helm/asap/ has Chart.yaml + values.yaml, no templates/
dir — reflected honestly.
No semantic change; pure consolidation + refresh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
deploy/TODO.md was ~90% duplicated with PROGRESS.md's 'Outstanding' section — same instrumentation, query-side, fault-injection, and Helm items, rephrased. Consolidate. Genuinely unique content folded into PROGRESS.md: - Helm template landing order (file-by-file sequence, from _helpers.tpl through prometheus.yaml/grafana.yaml) — now part of the 'Helm chart templates' future-work bullet. - Fault-injection script names (controller-kill.sh, agent-kill.sh, network-partition.sh) — now bullet-listed under paper blocker #8. - Compose polish (per-agent AGENT_ID label, CI check, deploy/k8s/ plain-manifest alternative) — new bullet under Future work; didn't previously live anywhere else. Fixed three dangling refs to deploy/TODO.md: - deploy/README.md — the Helm 'see TODO.md in this directory' line now points at ../PROGRESS.md. - deploy/docker-compose/base.yml header comment — 'tracked in deploy/TODO.md' → 'tracked in PROGRESS.md'. - PROGRESS.md's own sibling-docs list drops the now-dead deploy/TODO.md entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to previous commit which tried to update this line but lost it to a read-modify-write race. Pure comment fix.
Answers the follow-up: at the same ground-truth raw-sample rate (10 Hz per series), does it cost more or less to emit each sample per tick (W=100ms) vs buffer them and ship in a batch (W=10s)? Knob changes: - deploy/fake-exporter/main.go — new EXPORTER_PPROF_ADDR env var. When set, opens net/http/pprof on the given address so external tools can sample CPU / heap / goroutines live. No-op when unset; production runs stay unchanged. Findings (full detail in deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md, 4 configs × 15s pprof captures against the live gateway): - At matching cardinality, per-tick emit (W=100ms) uses ~2× the CPU of batched (W=10s). The delta isn't data volume — it's the per-flush overhead (gRPC ClientConn.Invoke + flate writer init) that W=100ms pays 100× more often. - Per-tick emit keeps RSS flat. Batched W=10s with card=100 spikes +39 MiB at flush — the protobuf transform intermediates (transform.Value / KeyValue / DataPoints) dominate heap. - Config A (per-tick) CPU hotspots: PeriodicReader.collectAndExport → otlpmetricgrpc.Export → grpc.ClientConn.Invoke (54 % cum). - Config B (batched) CPU hotspots at card=100: runtime.gcBgMarkWorker (30 %) + runtime.gcDrain (30 %) — GC chases the flush burst. - Neither scales linearly where you'd expect: A's CPU scales with 1/W (flush frequency), B's memory scales with cardinality × samples_per_window. - Design corner: small W + modest cardinality is cheap; large W + large cardinality is dangerous (memory bursts that container cgroups will clip). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… events/s
Extends the emit-cadence profile to fine-grained scale:
cardinality=1000 × freq_hz ∈ {100, 1000}, so 100 k and 1 M
events/s. Runs the same A(W=100ms) vs B(W=10s) comparison
as the first pass.
Three findings not visible in the small-scale pass:
1. A-vs-B CPU gap collapses. At tiny scale A was ~2× B
(per-flush fixed cost dominated). At fine scale per-flush
overhead amortises over 10k–100k dp batches and the gap
disappears (A:2.8c vs B:3.0c at 100k/s; A:8.1c vs B:9.8c
at 1M/s — B slightly worse).
2. Hot path shifts from gRPC export to aggregator mutex
contention. rawBufferValues.measure takes a single
valuesMu across the whole attribute map. With 1000
goroutines firing 1000 Adds/s = 1M Lock acquisitions/s,
lockSlow dominates (~26% of CPU). This is a real design
issue — a sharded buffer or per-series lock is needed
before raw-buffer is usable at ≥1M events/s.
3. Heap is dominated by OTLP transform intermediates, not
the raw-buffer. At 1M events/s ~1 GiB of KeyValue/Value/
DataPoints objects are allocated per flush in both
configs. Pool reuse in the upstream transform would help.
Results table + hot-path diff + heap breakdown appended to
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First pass of the SDK cost evaluation — 17 cells at
CARDINALITY=1000, FREQ_HZ=10, SOAK_S=180, N=1, ~65 min wall.Architecture:
docs/controller-optimization-problem.md— full data-plane lifecycle (App+SDK → Agent → Backend → ASAPQuery).Evaluation design + methodology:
docs/sdk-cost-evaluation.md— the(W, L, agg_type)triple, what each knob means, how each axis is swept.Files
Under
deploy/eval-results/sdk-cost/:FINDINGS-20260423.mdtime-axis-20260423.csvW ∈ {1s, 15s, 60s, 300s}; holdsL = keep-all,agg = dd-fulllabel-axis-20260423.csvL ∈ {keep-all, zone+rack+node, zone+rack, zone, drop-all}; holdsW = 60s,agg = dd-fullencoding-axis-20260423.csvagg ∈ {raw-buffer, dd-full, dd-delta, kll, cms-full, cms-delta, hll-full, hll-delta}; holdsW = 60s,L = zone,rack*-axis-20260423.logdeploy/scripts/run-sdk-cost-sweeps.shFindings (detail in
FINDINGS-20260423.md)raw-bufferuses ~3× RSS and ~7× wire bytes vs any sketch.cms-delta= 2×cms-fullRSS — delta keeps the prev-tick snapshot for diff; it's a memory/bw tradeoff, not a free win.keep-all → zone,rackcuts producer RSS ~50%. CPU increases with more projection —AttributeFilteron the hot path, needs a profile before it goes into any figure.W=1stoW=15s+.Caveat
BYTES_WIN=20 < W=60son most cells ⇒producer_bytes_out_per_sunder-reported by ~W/BYTES_WIN. Within-sub-experiment ratios (all rows share the sameW) are still valid.Next
Auto-floor
BYTES_WIN ≥ 2×Win the driver, then v2 rerun (~90 min) if absolute bytes numbers are needed. Profile the label-axisAttributeFilterCPU climb.🤖 Generated with Claude Code