feat(sdk/metric): AggregationRawBuffer — raw-sample encoding slot - #189
Merged
Merged
Conversation
PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Formalizes the SDK-side decision point as a (W, L, agg_type) triple: time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric. - docs/sdk-aggregation-three-axis-design.md (new) — authoritative design doc for the framework. Maps to existing Mode 1/2/3 vocabulary in delta-transmission-design.md. Enumerates the outstanding SDK-side aggregators (raw-buffer + 5 delta variants) and the SDK runtime hook (hot-reload AttributeFilter) needed to exercise the full planner loop. - docs/paper-outline.md §6 — split §6.2 into four sub-sweeps (a/b/c/d) along the three axes, introduce §6.5 planner-quality as an independent experiment, update claims table to express bandwidth reduction as a three-factor product. - PROGRESS.md — bump date to 2026-04-23; point at new design doc; enumerate outstanding aggregators (~150 LOC raw-buffer + ~500 LOC five delta variants) + fake-exporter knobs + measure-baseline producer-side columns. - docs/n10-bottleneck-rca.md — prepend postscript noting the Options A–D recommendation section is superseded by the three-axis design. Diagnosis content unchanged. - TODO.md (top-level) — retire "N=10 throughput collapse" as P0 blocker (the RCA closed it); replace with the concrete implementation punch list for the three-axis framework. - deploy/TODO.md — add producer-side measurement requirement (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the instrumentation P1 list; the three-axis sweeps read these from fake-exporter container directly rather than inferring from gateway counters. Includes the n10-bottleneck-rca.md content from the earlier diag/n10-bottleneck-rca branch (superseded by this PR — close that one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows that DeltaTransmission is already a field on DDSketch / CountSketch / CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so there's no need to ship five new AggregationDelta<X> types. Only KLL lacks delta support, and its multi-level sample-buffer structure doesn't admit a naive byte-diff — separate design problem, not a §6.2 blocker. Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer (~150 LOC). Proceeding to implement that next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds AggregationRawBuffer to the SDK as the encoding-axis baseline
for the three-axis framework (see docs/sdk-aggregation-three-axis-design.md):
where every Counter.Add / Gauge.Record call becomes its own
NumberDataPoint on the wire instead of being reduced to a Sum or
LastValue per attribute set per tick.
Implementation:
- sdk/metric/aggregation.go — new AggregationRawBuffer{MaxEventsPerSeries}
public type, wired into the Aggregation interface and err()
validation alongside the existing sketch aggregators.
- sdk/metric/internal/aggregate/rawbuffer.go — aggregator impl. Per
attribute.Distinct key, appends (ts, value) to a bounded slice in
measure(); in collect() emits each buffered sample as its own
metricdata.DataPoint[N] inside a Gauge[N], then clears the buffer.
Overflow on a single series is silently dropped with a per-series
drop counter (exposing that via a side-channel metric is a
follow-up tracked in PROGRESS.md).
- sdk/metric/internal/aggregate/aggregate.go — Builder.RawBuffer
method, mirroring the KLLSketch / HLLSketch shape.
- sdk/metric/pipeline.go — dispatch into Builder.RawBuffer +
isAggregatorCompatible.
Tests: four unit tests covering the core contract (every sample
emitted, second collect is empty i.e. no cumulative semantics
retained, per-series cap honoured, many-attribute-set isolation).
Design notes:
- Both delta and cumulative paths call the same collect(). Raw-buffer
has no meaningful cumulative semantics — re-emitting all history
every tick would be useless — so the buffer always resets after
collect regardless of requested temporality.
- We cap per-series, not globally, because the L (label-projection)
axis sweep intentionally drives cardinality down; a global cap
would couple the two axes.
Pre-existing hllsketch build break fixed in passing:
- sketchlib-go renamed HyperLogLog.Insert → InsertValue (float64
arg) and Estimate → EstimateCardinality. Two-line patch in
hllsketch.go to restore a green build. The sketchlib-go
call surface is unchanged at semver zero, so this is
mechanical drift, not a behaviour change.
Stacked on docs/sdk-three-axis-framework — review that first for
the design context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.
Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:
6.2a time: WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
6.2b label: WINDOWS="60s" PROJECTIONS=": zone,rack zone -"
AGGS="dd-full"
6.2c encoding: WINDOWS="60s" PROJECTIONS="zone,rack"
AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"
Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
labels" (the natural empty-string would be awkward to pass
through a space-separated list) and "-" for "drop all".
decode_projection() maps those back to what
fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
(OTLP → batch(1s) → OTLP), which keeps the agent from running
a sketch pipeline on top and conflating the SDK-side
measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
is trivially splittable back into the grid for plotting.
Smoke-tested locally against the PR #189/#190/#191 stack:
$ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv
baseline,scale,rate,cardinality,producer_cpu_cores,...
w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...
Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.
Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.
Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:
6.2a time: WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
6.2b label: WINDOWS="60s" PROJECTIONS=": zone,rack zone -"
AGGS="dd-full"
6.2c encoding: WINDOWS="60s" PROJECTIONS="zone,rack"
AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"
Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
labels" (the natural empty-string would be awkward to pass
through a space-separated list) and "-" for "drop all".
decode_projection() maps those back to what
fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
(OTLP → batch(1s) → OTLP), which keeps the agent from running
a sketch pipeline on top and conflating the SDK-side
measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
is trivially splittable back into the grid for plotting.
Smoke-tested locally against the PR #189/#190/#191 stack:
$ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv
baseline,scale,rate,cardinality,producer_cpu_cores,...
w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...
Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.
Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md all still had AggregationRawBuffer flagged ❌. It shipped in #189 back in April — updating the status rows + cross-referencing the contract test (deploy/fake-exporter/sdk_emit_test.go) + the implementation path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
- Rename `docs/sdk-aggregation-three-axis-design.md` → `docs/sdk-aggregation-cost.md`. Title updated to 'SDK aggregation cost'. Old name was a framework description; new name says directly what the doc is measuring. - Update every cross-reference in docs, PROGRESS.md, TODO.md, deploy/ (compose, Dockerfile, go.mod, main.go, scripts, FINDINGS) to point at the new filename. - Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status tables and TODO checkboxes. Merged to main = ✅; the dates and PR numbers are git-log territory, not running-doc territory. No substantive content change; only the framing/file name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 30, 2026
* eval(§6.2): first three-axis sweep pass + findings write-up
Runs the §6.2 a/b/c sub-sweeps defined in
docs/sdk-aggregation-three-axis-design.md at CARDINALITY=1000,
FREQ_HZ=10, N=1, SOAK_S=180, BYTES_WIN=20. 17 cells total,
~65 min wall clock.
Artifacts:
- deploy/scripts/run-paper-6.2-sweeps.sh — wrapper that runs
all three sub-sweeps at named env points and drops CSVs.
- deploy/eval-results/three-axis/6.2a-time-20260423.csv
- deploy/eval-results/three-axis/6.2b-label-20260423.csv
- deploy/eval-results/three-axis/6.2c-encoding-20260423.csv
plus .log files for each sub-sweep stderr.
- deploy/eval-results/three-axis/FINDINGS-20260423.md — results
interpretation with three sections:
- Clean findings (encoding-axis ratios, label-axis RSS drop)
- Methodology caveat (BYTES_WIN < W under-counts bytes)
- Surprises worth re-checking (§6.2b CPU-up-with-projection,
cms-delta memory overhead)
Key clean findings:
- §6.2c encoding: raw-buffer holds 108 MiB RSS vs 33-39 MiB for
sketch encodings (~3× memory cost for per-event storage) and
emits ~7× the wire bytes of any sketch. cms-delta costs 2×
the RSS of cms-full because it holds prev-tick snapshot for
diff.
- §6.2b label: projecting to zone,rack cuts producer RSS from
66.8 → 33.7 MiB as attribute-set folding reduces sketch
instance count.
Known caveat: BYTES_WIN=20s < W=60s in most cells, so
producer_bytes_out_per_s is under-reported by factor ~W/BYTES_WIN.
Ratios within a sub-sweep (all at same W) are unaffected. V2
rerun with BYTES_WIN ≥ 2W will fix absolute numbers — doc'd in
FINDINGS as the top-priority follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rewrite FINDINGS to describe the experiment self-contained
Drops paper-section references from the findings doc so it reads
as experimental results standing on their own, not as "§6.2
sub-sweep output". Structure is now:
1. Experiment design — what the (W, L, agg_type) triple means,
what each sub-experiment varies vs holds fixed, how each
cell is measured.
2. Methodology caveat (unchanged — BYTES_WIN < W under-reports).
3. Results per sub-experiment (encoding axis, label axis,
time axis) with the numbers and what to make of them.
4. Non-obvious findings worth a second look.
5. Next actions.
No numeric content changed; CSVs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rename findings title to 'SDK aggregation experiments'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain W + push vs pull in FINDINGS
Adds a 'What W actually is (and push vs pull)' subsection right
after the (W, L, agg_type) definition so the doc doesn't assume
the reader knows OTel's Reader/PeriodicReader model. Compares
OTLP push and Prometheus pull aggregation-equivalence and
calls out the two places they actually differ for this project
(NAT-friendliness + controller-reconfig channel).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: mark AggregationRawBuffer as landed (#189)
docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md
all still had AggregationRawBuffer flagged ❌. It shipped in
#189 back in April — updating the status rows + cross-referencing
the contract test (deploy/fake-exporter/sdk_emit_test.go) +
the implementation path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: expand AggregationRawBuffer design — why + what
Rewrites the AggregationRawBuffer subsection in the design doc to
explain *why* raw-buffer is a first-class encoding slot (not just
what it does). Key points the previous version didn't make:
- Every other upstream Aggregation is lossy by design; raw-buffer
is the only SDK-native option that preserves the full
observation stream. Without it there's no reference point
for sketch-vs-raw claims.
- It's also the encoding that can serve queries no summary
can: exact events for cold-fallback replay, per-sample audit,
downstream sketch computation the SDK policy didn't
anticipate.
- Memory cost scales as O(W × event_rate × cardinality) —
the tradeoff the encoding axis measures.
Contract + overflow + temporality specifics kept from the
earlier version and folded into dedicated subsections.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rename to 'SDK aggregation cost' + strip landed-date stamps
- Rename `docs/sdk-aggregation-three-axis-design.md` →
`docs/sdk-aggregation-cost.md`. Title updated to 'SDK
aggregation cost'. Old name was a framework description;
new name says directly what the doc is measuring.
- Update every cross-reference in docs, PROGRESS.md, TODO.md,
deploy/ (compose, Dockerfile, go.mod, main.go, scripts,
FINDINGS) to point at the new filename.
- Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status
tables and TODO checkboxes. Merged to main = ✅; the dates
and PR numbers are git-log territory, not running-doc
territory.
No substantive content change; only the framing/file name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove stray __pycache__ + ignore it
* docs: rename to 'SDK cost evaluation' to signal this is the eval doc
File: docs/sdk-aggregation-cost.md → docs/sdk-cost-evaluation.md
Title: 'SDK cost evaluation'
All cross-references in docs/, PROGRESS.md, TODO.md, deploy/
updated. No content change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: delete n10-bottleneck-rca.md; rework kll-delta status to 'no need'
- Delete docs/n10-bottleneck-rca.md — the finding (PR #185's
'N=10 throughput collapse' is SDK pre-aggregation, not a
bottleneck) has propagated into the docs that still need it
(TODO §1 and the SDK cost evaluation doc summarise the
conclusion inline). The long write-up was one-shot debug
context, not a reference.
- Drop the dangling link from docs/sdk-cost-evaluation.md's
header block (and the 'Authoritative for §6.2…' framing —
redundant with the doc content).
- Fix the other references that pointed at the deleted file:
TODO.md §1, PROGRESS.md outstanding-aggregators note,
fake-exporter main.go comment and EXPORTER_RATE deprecation
log, docker-compose/base.yml comment. Each now explains the
point inline.
- Rework the kll-delta status in both docs/sdk-cost-evaluation.md
and PROGRESS.md from '❌ not yet' → '**no need**'. kll-full
covers the encoding-axis comparison; implementing a KLL-specific
delta would need a different strategy than the byte-diff the
other four sketches use, and the payoff doesn't justify it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: strip paper-section references; rename sweep driver; drop follow-up list
Across-the-board cleanup so evaluation docs stand on their own and
don't cite paper section numbers. Evaluations describe design +
methodology + results directly; paper framing belongs only in
docs/paper-outline.md.
- Rename deploy/scripts/run-paper-6.2-sweeps.sh →
run-sdk-cost-sweeps.sh. Filename drops the §6.2 embedding.
- Delete docs/design-stateful-protocol.md's "(Paper §4.5)" title
suffix; title is now just the protocol name.
- docs/sdk-cost-evaluation.md:
- remove the obsolete "Implementation order (follow-up PRs)"
section — most items are merged, and the remaining one
lives in TODO.md anyway
- remove the "Non-goals of this doc" section — pointer list
was stale
- drop the paper-§6 framing in the opening; doc now reads
as standalone evaluation design
- rewrite the ablation table: instead of "6.2a Time axis /
6.2b Label axis / …" with "Claim" column, plain "Time /
Label / Encoding / Combined" with "What the sweep
measures" column
- strip the few remaining §6.2 / §6.5 inline refs
- Strip paper-§N from every config / compose / script / TODO /
PROGRESS comment where it appeared. Specifically:
- PROGRESS.md outstanding-aggregators section
- TODO.md §1 header (now "SDK cost evaluation") + all inline
§6.2 references
- deploy/TODO.md instrumentation section headings + inline
- deploy/README.md "Paper §6 mapping" → "Evaluation → metric
mapping"; drop "paper's §6 eval" framing
- deploy/configs/*.yaml — comment prefixes like "Paper §6.2
baseline:" → plain "Baseline:"
- deploy/configs/prometheus.yml, grafana-datasources.yml —
same
- deploy/docker-compose/*.yml — "Paper §6" → "Scale dials"
etc.
- deploy/helm/asap/{values,Chart}.yaml — same
- deploy/fake-exporter/main.go — "Paper's W axis" / "L axis"
/ "encoding axis" → "Time axis" / "Label axis" / "Encoding
axis"
- deploy/fake-exporter/sdk_emit_test.go — §6.2c → encoding-axis
- Left in place: internal design-doc cross-references like
"delta-transmission-design.md §16.2" where the §N refers to a
section of a design doc rather than the paper. Those are legit
technical writing, not paper citations.
No substantive content or behaviour change anywhere; pure
framing/naming cleanup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* eval: rename artifacts to match SDK cost evaluation naming
- deploy/eval-results/three-axis/ → deploy/eval-results/sdk-cost/
to match the design doc (docs/sdk-cost-evaluation.md).
- CSV + log files lose the paper-section prefix:
6.2a-time-20260423 → time-axis-20260423
6.2b-label-20260423 → label-axis-20260423
6.2c-encoding-20260423 → encoding-axis-20260423
- Update every reference in
docs/sdk-cost-evaluation.md, TODO.md,
deploy/scripts/run-sdk-cost-sweeps.sh, and the FINDINGS doc.
Data unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: sharpen raw-buffer rationale in SDK cost eval
The previous 'Why this baseline exists' blurb framed raw-buffer as
a generic reference point but didn't pin down (a) that OTel's
native aggregators are lossy *specifically over the emit period
W*, or (b) that the experiment's value comes from running
raw-buffer at the *same* W as the sketch encodings so the
comparison is apples-to-apples.
Rewrite leads with those two observations, then closes on the
'without it, the ablation collapses into sketch vs nothing' line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain Aggregation<X>Delta as a design decision, not a TODO
The previous 'Adding Aggregation<X>Delta (×5)' block read like a
PR checklist — semantics / encoding / LOC estimate. This is a
design doc, not a task tracker.
Rewrite leads with *what* delta encoding is (full state vs. sparse
diff of changed cells; aggregator keeps prev-tick state; full
re-emit on L / param change) and *why* it deserves a distinct
slot in the encoding-axis evaluation:
1. The bandwidth claim 'sketch < raw' has two independent
factors (sketch payload vs raw, and delta vs full sketch).
Measuring *-full and *-delta as separate rows on the
encoding axis lets each factor be attributed directly.
2. Delta is a memory-for-bandwidth trade: the aggregator
holds prev-tick state alongside current, roughly doubling
RSS. That tradeoff belongs next to the bandwidth savings
in the encoding row, not buried as an implementation note.
Also notes that KLL is intentionally absent from the delta column
and points at the existing note above for the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: shorten section headings
AggregationRawBuffer design → AggregationRawBuffer
Aggregation<X>Delta — what it is and why it's a separate slot →
Aggregation<X>Delta
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deploy: rename run-three-axis-sweep.sh → run-sdk-cost-sweep.sh
Last paper-shaped artefact in the cost-evaluation tooling. The
inner grid driver is now 'run-sdk-cost-sweep.sh' (singular —
runs one sweep over the grid the caller specifies), and the
outer wrapper stays 'run-sdk-cost-sweeps.sh' (plural — runs the
three canonical sub-sweeps).
Updates every reference in the evaluation doc, findings, TODO,
and the wrapper script itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deploy: rename sweep scripts by role (grid vs eval)
- run-sdk-cost-sweep.sh → run-sdk-cost-grid.sh
Runs a W × L × agg grid the caller specifies via env.
- run-sdk-cost-sweeps.sh → run-sdk-cost-eval.sh
Runs the three canonical sub-sweeps (time / label / encoding)
that make up the SDK cost evaluation; calls run-sdk-cost-grid
three times with different fixed/swept axes.
The singular-vs-plural distinction was too subtle; 'grid' vs
'eval' spells out the role difference directly.
Updates references in docs/sdk-cost-evaluation.md, FINDINGS,
TODO.md, and the eval script's own call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: merge TODO.md into PROGRESS.md
Two top-level docs were covering the same ground — PROGRESS
drifted into historical session notes, TODO duplicated the
'what's landed' summary and added 'what's left'. Consolidated
into a single PROGRESS.md with three sections:
- Implemented — ground-truthed against current code
- Outstanding — paper blockers + non-blocker SDK work
- Future work — post-paper
Top-level TODO.md deleted. Cross-references in deploy/TODO.md
and deploy/README.md updated to point at PROGRESS.md instead
(the local 'See TODO.md in this directory' ref in README refers
to deploy/TODO.md, kept as-is).
Sketch-processor and aggregator status tables verified against
the patch trees:
opentelemetry-go-patch/sdk/metric/aggregation.go lists all six
Aggregation types we document (DDSketch/KLL/CS/CMS/HLL +
RawBuffer), matching the table.
opentelemetry-collector-contrib-patch/processor/ has the five
per-sketch processors + two new *mergeprocessor variants
(countsketchmerge, countminsketchmerge) surfaced in the
table.
deploy/helm/asap/ has Chart.yaml + values.yaml, no templates/
dir — reflected honestly.
No semantic change; pure consolidation + refresh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: merge deploy/TODO.md into PROGRESS.md
deploy/TODO.md was ~90% duplicated with PROGRESS.md's
'Outstanding' section — same instrumentation, query-side,
fault-injection, and Helm items, rephrased. Consolidate.
Genuinely unique content folded into PROGRESS.md:
- Helm template landing order (file-by-file sequence, from
_helpers.tpl through prometheus.yaml/grafana.yaml) — now
part of the 'Helm chart templates' future-work bullet.
- Fault-injection script names (controller-kill.sh,
agent-kill.sh, network-partition.sh) — now bullet-listed
under paper blocker #8.
- Compose polish (per-agent AGENT_ID label, CI check,
deploy/k8s/ plain-manifest alternative) — new bullet under
Future work; didn't previously live anywhere else.
Fixed three dangling refs to deploy/TODO.md:
- deploy/README.md — the Helm 'see TODO.md in this directory'
line now points at ../PROGRESS.md.
- deploy/docker-compose/base.yml header comment — 'tracked
in deploy/TODO.md' → 'tracked in PROGRESS.md'.
- PROGRESS.md's own sibling-docs list drops the now-dead
deploy/TODO.md entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: base.yml comment ref to deploy/TODO.md → PROGRESS.md
Follow-up to previous commit which tried to update this line but
lost it to a read-modify-write race. Pure comment fix.
* eval(sdk-cost): profile per-sample vs batched raw-buffer emit
Answers the follow-up: at the same ground-truth raw-sample
rate (10 Hz per series), does it cost more or less to emit
each sample per tick (W=100ms) vs buffer them and ship in a
batch (W=10s)?
Knob changes:
- deploy/fake-exporter/main.go — new EXPORTER_PPROF_ADDR env
var. When set, opens net/http/pprof on the given address so
external tools can sample CPU / heap / goroutines live.
No-op when unset; production runs stay unchanged.
Findings (full detail in
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md,
4 configs × 15s pprof captures against the live gateway):
- At matching cardinality, per-tick emit (W=100ms) uses ~2×
the CPU of batched (W=10s). The delta isn't data volume —
it's the per-flush overhead (gRPC ClientConn.Invoke + flate
writer init) that W=100ms pays 100× more often.
- Per-tick emit keeps RSS flat. Batched W=10s with card=100
spikes +39 MiB at flush — the protobuf transform
intermediates (transform.Value / KeyValue / DataPoints)
dominate heap.
- Config A (per-tick) CPU hotspots: PeriodicReader.collectAndExport
→ otlpmetricgrpc.Export → grpc.ClientConn.Invoke (54 % cum).
- Config B (batched) CPU hotspots at card=100:
runtime.gcBgMarkWorker (30 %) + runtime.gcDrain (30 %) —
GC chases the flush burst.
- Neither scales linearly where you'd expect: A's CPU scales
with 1/W (flush frequency), B's memory scales with
cardinality × samples_per_window.
- Design corner: small W + modest cardinality is cheap; large
W + large cardinality is dangerous (memory bursts that
container cgroups will clip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* eval(sdk-cost): fine-grained pprof — mutex contention dominates at 1M events/s
Extends the emit-cadence profile to fine-grained scale:
cardinality=1000 × freq_hz ∈ {100, 1000}, so 100 k and 1 M
events/s. Runs the same A(W=100ms) vs B(W=10s) comparison
as the first pass.
Three findings not visible in the small-scale pass:
1. A-vs-B CPU gap collapses. At tiny scale A was ~2× B
(per-flush fixed cost dominated). At fine scale per-flush
overhead amortises over 10k–100k dp batches and the gap
disappears (A:2.8c vs B:3.0c at 100k/s; A:8.1c vs B:9.8c
at 1M/s — B slightly worse).
2. Hot path shifts from gRPC export to aggregator mutex
contention. rawBufferValues.measure takes a single
valuesMu across the whole attribute map. With 1000
goroutines firing 1000 Adds/s = 1M Lock acquisitions/s,
lockSlow dominates (~26% of CPU). This is a real design
issue — a sharded buffer or per-series lock is needed
before raw-buffer is usable at ≥1M events/s.
3. Heap is dominated by OTLP transform intermediates, not
the raw-buffer. At 1M events/s ~1 GiB of KeyValue/Value/
DataPoints objects are allocated per flush in both
configs. Pool reuse in the upstream transform would help.
Results table + hot-path diff + heap breakdown appended to
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First code PR for the three-axis framework (design doc: #N — stacked on docs/sdk-three-axis-framework). Adds
AggregationRawBuffer— the SDK-side encoding where everyCounter.Add/Gauge.Recordcall becomes its ownNumberDataPointon the wire instead of being reduced to a Sum / LastValue per attribute set per tick. This is the paper's "raw" baseline on §6.2c's encoding axis.Changes
sdk/metric/aggregation.go— new publicAggregationRawBuffer{MaxEventsPerSeries}type, wired into theAggregationinterface +err()validation.sdk/metric/internal/aggregate/rawbuffer.go— aggregator impl (~140 LOC). Buffers(ts, value)perattribute.Distinctkey onmeasure(); emits each buffered sample as its ownmetricdata.DataPoint[N]inside aGauge[N]oncollect(), then clears the buffer.sdk/metric/internal/aggregate/aggregate.go—Builder.RawBuffermethod, mirroringKLLSketch/HLLSketch.sdk/metric/pipeline.go— dispatch intoBuilder.RawBuffer+isAggregatorCompatible.Design notes
collect(). Raw-buffer has no meaningful cumulative semantics — re-emitting all history every tick would be useless — so the buffer always resets after collect regardless of requested temporality. (Documented in thecollect()godoc.)PROGRESS.md.Drive-by fix
hllsketch.gowas broken against currentsketchlib-go:HyperLogLog.Insert(float64)was renamed toInsertValue(float64)Estimate()(no args) was renamed toEstimateCardinality() intTwo mechanical one-liners, no behaviour change. Needed to pass
go buildlocally.Test plan
Verified locally (Ubuntu 22.04, Go 1.22 w/
GOTOOLCHAIN=autopicking 1.24 per go.mod):```
$ cd opentelemetry-go/sdk/metric && go test -run RawBuffer -count=1 -v ./internal/aggregate
=== RUN TestRawBufferEmitsEverySample --- PASS
=== RUN TestRawBufferSecondCollectIsEmpty --- PASS
=== RUN TestRawBufferRespectsMaxEventsPerSeries --- PASS
=== RUN TestRawBufferHandlesManyAttributeSets --- PASS
$ go build ./... # sdk/metric full package — clean
$ go vet ./... # clean
```
Four unit tests cover:
Add/Recordcall in an interval produces oneDataPoint— no SDK pre-aggregationcollect()with no new observations emits zero data points (cumulative doesn't accumulate)Still to do in follow-ups:
fake-exporterwiring (EXPORTER_SDK_AGG=raw-bufferpath)measure-baseline.py🤖 Generated with Claude Code