Skip to content

feat(sdk/metric): AggregationRawBuffer — raw-sample encoding slot - #189

Merged
zzylol merged 4 commits into
mainfrom
feat/aggregation-raw-buffer
Apr 23, 2026
Merged

zzylol merged 4 commits into
mainfrom
feat/aggregation-raw-buffer

Conversation

@zzylol

@zzylol zzylol commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Summary

First code PR for the three-axis framework (design doc: #N — stacked on docs/sdk-three-axis-framework). Adds AggregationRawBuffer — the SDK-side encoding where every Counter.Add / Gauge.Record call becomes its own NumberDataPoint on the wire instead of being reduced to a Sum / LastValue per attribute set per tick. This is the paper's "raw" baseline on §6.2c's encoding axis.

Changes

  • sdk/metric/aggregation.go — new public AggregationRawBuffer{MaxEventsPerSeries} type, wired into the Aggregation interface + err() validation.
  • sdk/metric/internal/aggregate/rawbuffer.go — aggregator impl (~140 LOC). Buffers (ts, value) per attribute.Distinct key on measure(); emits each buffered sample as its own metricdata.DataPoint[N] inside a Gauge[N] on collect(), then clears the buffer.
  • sdk/metric/internal/aggregate/aggregate.goBuilder.RawBuffer method, mirroring KLLSketch / HLLSketch.
  • sdk/metric/pipeline.go — dispatch into Builder.RawBuffer + isAggregatorCompatible.

Design notes

  • Delta and cumulative call the same collect(). Raw-buffer has no meaningful cumulative semantics — re-emitting all history every tick would be useless — so the buffer always resets after collect regardless of requested temporality. (Documented in the collect() godoc.)
  • Overflow is silent drop with a per-series drop counter. Not backpressure, because backpressure would conflate "SDK overload" with "app slow path" in experiment numbers. Exposing the drop count as a side-channel metric is a follow-up tracked in PROGRESS.md.
  • Per-series cap, not global cap. The L-axis (label projection) sweep intentionally drives cardinality down; a global cap would couple the two axes.

Drive-by fix

hllsketch.go was broken against current sketchlib-go:

  • HyperLogLog.Insert(float64) was renamed to InsertValue(float64)
  • Estimate() (no args) was renamed to EstimateCardinality() int

Two mechanical one-liners, no behaviour change. Needed to pass go build locally.

Test plan

Verified locally (Ubuntu 22.04, Go 1.22 w/ GOTOOLCHAIN=auto picking 1.24 per go.mod):

```
$ cd opentelemetry-go/sdk/metric && go test -run RawBuffer -count=1 -v ./internal/aggregate
=== RUN TestRawBufferEmitsEverySample --- PASS
=== RUN TestRawBufferSecondCollectIsEmpty --- PASS
=== RUN TestRawBufferRespectsMaxEventsPerSeries --- PASS
=== RUN TestRawBufferHandlesManyAttributeSets --- PASS

$ go build ./... # sdk/metric full package — clean
$ go vet ./... # clean
```

Four unit tests cover:

  1. Every Add/Record call in an interval produces one DataPoint — no SDK pre-aggregation
  2. Second collect() with no new observations emits zero data points (cumulative doesn't accumulate)
  3. Per-series cap honoured — extra measurements on a full buffer are dropped
  4. Many distinct attribute sets each keep independent buffers

Still to do in follow-ups:

  • fake-exporter wiring (EXPORTER_SDK_AGG=raw-buffer path)
  • Producer-side columns in measure-baseline.py

🤖 Generated with Claude Code

zzylol and others added 4 commits April 23, 2026 11:04
PR #185 flagged a universal ~2k pts/s floor at N=10 across all
six baselines and attributed it to a coordinated throttle
(OTLP SDK / Docker userland-proxy / kernel socket buffers).

Rate-invariance test disproves the bottleneck reading. Holding
cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000
(10× change) on the same b0a-raw-stream N=1 stack: gateway
rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s.
If any of the candidate bottlenecks were the cause, 10× input
would produce observable throughput delta — it doesn't.

Root cause: PR #182 (`feat(workload): fake-exporter trace-replay
mode`) changed the OTel MeterProvider's PeriodicReader interval
from `time.Second / time.Duration(rate)` to `time.Second` fixed.
With a 1 s interval the SDK pre-aggregates Counter.Add and
Gauge.Record calls per attribute set within each tick, so the
export rate becomes `cardinality × #instruments × (1/interval)`
= 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10
is just 10 concurrent producers each correctly emitting 2k.

Bigger implication: the paper §6.2 "raw vs sketch bandwidth"
story is more fragile than it looks. With the current
fake-exporter, the "raw" baselines (B0a / B0b / B1) are already
SDK pre-aggregated at the producer — they are not emitting
per-sample traffic. The bandwidth delta vs sketch baselines
measures payload shape, not "raw samples vs summary per window".

Lays out four paper-story options (A revert interval / B dual-
interval / C bypass SDK aggregation / D reframe §6.2 as bytes-
per-window). Recommends B + D. Does not land a fix; that's a
follow-up PR that needs a paper-framing decision.

Artifacts:
- docs/n10-bottleneck-rca.md — full write-up
- deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv —
  the two-row evidence

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Formalizes the SDK-side decision point as a (W, L, agg_type)
triple: time window × label projection × encoding. All three
axes are independent and correspond one-to-one to what the
controller's planner emits per metric.

- docs/sdk-aggregation-three-axis-design.md (new) — authoritative
  design doc for the framework. Maps to existing Mode 1/2/3
  vocabulary in delta-transmission-design.md. Enumerates the
  outstanding SDK-side aggregators (raw-buffer + 5 delta
  variants) and the SDK runtime hook (hot-reload AttributeFilter)
  needed to exercise the full planner loop.

- docs/paper-outline.md §6 — split §6.2 into four sub-sweeps
  (a/b/c/d) along the three axes, introduce §6.5 planner-quality
  as an independent experiment, update claims table to express
  bandwidth reduction as a three-factor product.

- PROGRESS.md — bump date to 2026-04-23; point at new design
  doc; enumerate outstanding aggregators (~150 LOC raw-buffer +
  ~500 LOC five delta variants) + fake-exporter knobs +
  measure-baseline producer-side columns.

- docs/n10-bottleneck-rca.md — prepend postscript noting the
  Options A–D recommendation section is superseded by the
  three-axis design. Diagnosis content unchanged.

- TODO.md (top-level) — retire "N=10 throughput collapse" as
  P0 blocker (the RCA closed it); replace with the concrete
  implementation punch list for the three-axis framework.

- deploy/TODO.md — add producer-side measurement requirement
  (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the
  instrumentation P1 list; the three-axis sweeps read these
  from fake-exporter container directly rather than inferring
  from gateway counters.

Includes the n10-bottleneck-rca.md content from the earlier
diag/n10-bottleneck-rca branch (superseded by this PR — close
that one).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows
that DeltaTransmission is already a field on DDSketch / CountSketch
/ CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so
there's no need to ship five new AggregationDelta<X> types.

Only KLL lacks delta support, and its multi-level sample-buffer
structure doesn't admit a naive byte-diff — separate design
problem, not a §6.2 blocker.

Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md,
and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer
(~150 LOC). Proceeding to implement that next.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds AggregationRawBuffer to the SDK as the encoding-axis baseline
for the three-axis framework (see docs/sdk-aggregation-three-axis-design.md):
where every Counter.Add / Gauge.Record call becomes its own
NumberDataPoint on the wire instead of being reduced to a Sum or
LastValue per attribute set per tick.

Implementation:
- sdk/metric/aggregation.go — new AggregationRawBuffer{MaxEventsPerSeries}
  public type, wired into the Aggregation interface and err()
  validation alongside the existing sketch aggregators.
- sdk/metric/internal/aggregate/rawbuffer.go — aggregator impl. Per
  attribute.Distinct key, appends (ts, value) to a bounded slice in
  measure(); in collect() emits each buffered sample as its own
  metricdata.DataPoint[N] inside a Gauge[N], then clears the buffer.
  Overflow on a single series is silently dropped with a per-series
  drop counter (exposing that via a side-channel metric is a
  follow-up tracked in PROGRESS.md).
- sdk/metric/internal/aggregate/aggregate.go — Builder.RawBuffer
  method, mirroring the KLLSketch / HLLSketch shape.
- sdk/metric/pipeline.go — dispatch into Builder.RawBuffer +
  isAggregatorCompatible.

Tests: four unit tests covering the core contract (every sample
emitted, second collect is empty i.e. no cumulative semantics
retained, per-series cap honoured, many-attribute-set isolation).

Design notes:
- Both delta and cumulative paths call the same collect(). Raw-buffer
  has no meaningful cumulative semantics — re-emitting all history
  every tick would be useless — so the buffer always resets after
  collect regardless of requested temporality.
- We cap per-series, not globally, because the L (label-projection)
  axis sweep intentionally drives cardinality down; a global cap
  would couple the two axes.

Pre-existing hllsketch build break fixed in passing:
- sketchlib-go renamed HyperLogLog.Insert → InsertValue (float64
  arg) and Estimate → EstimateCardinality. Two-line patch in
  hllsketch.go to restore a green build. The sketchlib-go
  call surface is unchanged at semver zero, so this is
  mechanical drift, not a behaviour change.

Stacked on docs/sdk-three-axis-framework — review that first for
the design context.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Base automatically changed from docs/sdk-three-axis-framework to main April 23, 2026 17:33
@zzylol
zzylol merged commit 3a2f477 into main Apr 23, 2026
@zzylol
zzylol deleted the feat/aggregation-raw-buffer branch April 23, 2026 17:34
zzylol added a commit that referenced this pull request Apr 23, 2026
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.

Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:

  6.2a time:     WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
  6.2b label:    WINDOWS="60s"  PROJECTIONS=": zone,rack zone -"
                 AGGS="dd-full"
  6.2c encoding: WINDOWS="60s"  PROJECTIONS="zone,rack"
                 AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"

Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
  labels" (the natural empty-string would be awkward to pass
  through a space-separated list) and "-" for "drop all".
  decode_projection() maps those back to what
  fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
  (OTLP → batch(1s) → OTLP), which keeps the agent from running
  a sketch pipeline on top and conflating the SDK-side
  measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
  is trivially splittable back into the grid for plotting.

Smoke-tested locally against the PR #189/#190/#191 stack:

  $ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
    CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
    ./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv

  baseline,scale,rate,cardinality,producer_cpu_cores,...
  w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...

Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.

Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 23, 2026
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.

Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:

  6.2a time:     WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
  6.2b label:    WINDOWS="60s"  PROJECTIONS=": zone,rack zone -"
                 AGGS="dd-full"
  6.2c encoding: WINDOWS="60s"  PROJECTIONS="zone,rack"
                 AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"

Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
  labels" (the natural empty-string would be awkward to pass
  through a space-separated list) and "-" for "drop all".
  decode_projection() maps those back to what
  fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
  (OTLP → batch(1s) → OTLP), which keeps the agent from running
  a sketch pipeline on top and conflating the SDK-side
  measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
  is trivially splittable back into the grid for plotting.

Smoke-tested locally against the PR #189/#190/#191 stack:

  $ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
    CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
    ./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv

  baseline,scale,rate,cardinality,producer_cpu_cores,...
  w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...

Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.

Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 23, 2026
docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md
all still had AggregationRawBuffer flagged ❌. It shipped in
#189 back in April — updating the status rows + cross-referencing
the contract test (deploy/fake-exporter/sdk_emit_test.go) +
the implementation path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 23, 2026
- Rename `docs/sdk-aggregation-three-axis-design.md` →
  `docs/sdk-aggregation-cost.md`. Title updated to 'SDK
  aggregation cost'. Old name was a framework description;
  new name says directly what the doc is measuring.
- Update every cross-reference in docs, PROGRESS.md, TODO.md,
  deploy/ (compose, Dockerfile, go.mod, main.go, scripts,
  FINDINGS) to point at the new filename.
- Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status
  tables and TODO checkboxes. Merged to main = ✅; the dates
  and PR numbers are git-log territory, not running-doc
  territory.

No substantive content change; only the framing/file name.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 30, 2026
* eval(§6.2): first three-axis sweep pass + findings write-up

Runs the §6.2 a/b/c sub-sweeps defined in
docs/sdk-aggregation-three-axis-design.md at CARDINALITY=1000,
FREQ_HZ=10, N=1, SOAK_S=180, BYTES_WIN=20. 17 cells total,
~65 min wall clock.

Artifacts:
- deploy/scripts/run-paper-6.2-sweeps.sh — wrapper that runs
  all three sub-sweeps at named env points and drops CSVs.
- deploy/eval-results/three-axis/6.2a-time-20260423.csv
- deploy/eval-results/three-axis/6.2b-label-20260423.csv
- deploy/eval-results/three-axis/6.2c-encoding-20260423.csv
  plus .log files for each sub-sweep stderr.
- deploy/eval-results/three-axis/FINDINGS-20260423.md — results
  interpretation with three sections:
  - Clean findings (encoding-axis ratios, label-axis RSS drop)
  - Methodology caveat (BYTES_WIN < W under-counts bytes)
  - Surprises worth re-checking (§6.2b CPU-up-with-projection,
    cms-delta memory overhead)

Key clean findings:
- §6.2c encoding: raw-buffer holds 108 MiB RSS vs 33-39 MiB for
  sketch encodings (~3× memory cost for per-event storage) and
  emits ~7× the wire bytes of any sketch. cms-delta costs 2×
  the RSS of cms-full because it holds prev-tick snapshot for
  diff.
- §6.2b label: projecting to zone,rack cuts producer RSS from
  66.8 → 33.7 MiB as attribute-set folding reduces sketch
  instance count.

Known caveat: BYTES_WIN=20s < W=60s in most cells, so
producer_bytes_out_per_s is under-reported by factor ~W/BYTES_WIN.
Ratios within a sub-sweep (all at same W) are unaffected. V2
rerun with BYTES_WIN ≥ 2W will fix absolute numbers — doc'd in
FINDINGS as the top-priority follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: rewrite FINDINGS to describe the experiment self-contained

Drops paper-section references from the findings doc so it reads
as experimental results standing on their own, not as "§6.2
sub-sweep output". Structure is now:

  1. Experiment design — what the (W, L, agg_type) triple means,
     what each sub-experiment varies vs holds fixed, how each
     cell is measured.
  2. Methodology caveat (unchanged — BYTES_WIN < W under-reports).
  3. Results per sub-experiment (encoding axis, label axis,
     time axis) with the numbers and what to make of them.
  4. Non-obvious findings worth a second look.
  5. Next actions.

No numeric content changed; CSVs untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: rename findings title to 'SDK aggregation experiments'

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: explain W + push vs pull in FINDINGS

Adds a 'What W actually is (and push vs pull)' subsection right
after the (W, L, agg_type) definition so the doc doesn't assume
the reader knows OTel's Reader/PeriodicReader model. Compares
OTLP push and Prometheus pull aggregation-equivalence and
calls out the two places they actually differ for this project
(NAT-friendliness + controller-reconfig channel).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: mark AggregationRawBuffer as landed (#189)

docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md
all still had AggregationRawBuffer flagged ❌. It shipped in
#189 back in April — updating the status rows + cross-referencing
the contract test (deploy/fake-exporter/sdk_emit_test.go) +
the implementation path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: expand AggregationRawBuffer design — why + what

Rewrites the AggregationRawBuffer subsection in the design doc to
explain *why* raw-buffer is a first-class encoding slot (not just
what it does). Key points the previous version didn't make:

  - Every other upstream Aggregation is lossy by design; raw-buffer
    is the only SDK-native option that preserves the full
    observation stream. Without it there's no reference point
    for sketch-vs-raw claims.
  - It's also the encoding that can serve queries no summary
    can: exact events for cold-fallback replay, per-sample audit,
    downstream sketch computation the SDK policy didn't
    anticipate.
  - Memory cost scales as O(W × event_rate × cardinality) —
    the tradeoff the encoding axis measures.

Contract + overflow + temporality specifics kept from the
earlier version and folded into dedicated subsections.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: rename to 'SDK aggregation cost' + strip landed-date stamps

- Rename `docs/sdk-aggregation-three-axis-design.md` →
  `docs/sdk-aggregation-cost.md`. Title updated to 'SDK
  aggregation cost'. Old name was a framework description;
  new name says directly what the doc is measuring.
- Update every cross-reference in docs, PROGRESS.md, TODO.md,
  deploy/ (compose, Dockerfile, go.mod, main.go, scripts,
  FINDINGS) to point at the new filename.
- Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status
  tables and TODO checkboxes. Merged to main = ✅; the dates
  and PR numbers are git-log territory, not running-doc
  territory.

No substantive content change; only the framing/file name.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: remove stray __pycache__ + ignore it

* docs: rename to 'SDK cost evaluation' to signal this is the eval doc

File: docs/sdk-aggregation-cost.md → docs/sdk-cost-evaluation.md
Title: 'SDK cost evaluation'
All cross-references in docs/, PROGRESS.md, TODO.md, deploy/
updated. No content change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: delete n10-bottleneck-rca.md; rework kll-delta status to 'no need'

- Delete docs/n10-bottleneck-rca.md — the finding (PR #185's
  'N=10 throughput collapse' is SDK pre-aggregation, not a
  bottleneck) has propagated into the docs that still need it
  (TODO §1 and the SDK cost evaluation doc summarise the
  conclusion inline). The long write-up was one-shot debug
  context, not a reference.
- Drop the dangling link from docs/sdk-cost-evaluation.md's
  header block (and the 'Authoritative for §6.2…' framing —
  redundant with the doc content).
- Fix the other references that pointed at the deleted file:
  TODO.md §1, PROGRESS.md outstanding-aggregators note,
  fake-exporter main.go comment and EXPORTER_RATE deprecation
  log, docker-compose/base.yml comment. Each now explains the
  point inline.
- Rework the kll-delta status in both docs/sdk-cost-evaluation.md
  and PROGRESS.md from '❌ not yet' → '**no need**'. kll-full
  covers the encoding-axis comparison; implementing a KLL-specific
  delta would need a different strategy than the byte-diff the
  other four sketches use, and the payoff doesn't justify it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: strip paper-section references; rename sweep driver; drop follow-up list

Across-the-board cleanup so evaluation docs stand on their own and
don't cite paper section numbers. Evaluations describe design +
methodology + results directly; paper framing belongs only in
docs/paper-outline.md.

- Rename deploy/scripts/run-paper-6.2-sweeps.sh →
  run-sdk-cost-sweeps.sh. Filename drops the §6.2 embedding.
- Delete docs/design-stateful-protocol.md's "(Paper §4.5)" title
  suffix; title is now just the protocol name.
- docs/sdk-cost-evaluation.md:
    - remove the obsolete "Implementation order (follow-up PRs)"
      section — most items are merged, and the remaining one
      lives in TODO.md anyway
    - remove the "Non-goals of this doc" section — pointer list
      was stale
    - drop the paper-§6 framing in the opening; doc now reads
      as standalone evaluation design
    - rewrite the ablation table: instead of "6.2a Time axis /
      6.2b Label axis / …" with "Claim" column, plain "Time /
      Label / Encoding / Combined" with "What the sweep
      measures" column
    - strip the few remaining §6.2 / §6.5 inline refs
- Strip paper-§N from every config / compose / script / TODO /
  PROGRESS comment where it appeared. Specifically:
    - PROGRESS.md outstanding-aggregators section
    - TODO.md §1 header (now "SDK cost evaluation") + all inline
      §6.2 references
    - deploy/TODO.md instrumentation section headings + inline
    - deploy/README.md "Paper §6 mapping" → "Evaluation → metric
      mapping"; drop "paper's §6 eval" framing
    - deploy/configs/*.yaml — comment prefixes like "Paper §6.2
      baseline:" → plain "Baseline:"
    - deploy/configs/prometheus.yml, grafana-datasources.yml —
      same
    - deploy/docker-compose/*.yml — "Paper §6" → "Scale dials"
      etc.
    - deploy/helm/asap/{values,Chart}.yaml — same
    - deploy/fake-exporter/main.go — "Paper's W axis" / "L axis"
      / "encoding axis" → "Time axis" / "Label axis" / "Encoding
      axis"
    - deploy/fake-exporter/sdk_emit_test.go — §6.2c → encoding-axis
- Left in place: internal design-doc cross-references like
  "delta-transmission-design.md §16.2" where the §N refers to a
  section of a design doc rather than the paper. Those are legit
  technical writing, not paper citations.

No substantive content or behaviour change anywhere; pure
framing/naming cleanup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* eval: rename artifacts to match SDK cost evaluation naming

- deploy/eval-results/three-axis/ → deploy/eval-results/sdk-cost/
  to match the design doc (docs/sdk-cost-evaluation.md).
- CSV + log files lose the paper-section prefix:
    6.2a-time-20260423     → time-axis-20260423
    6.2b-label-20260423    → label-axis-20260423
    6.2c-encoding-20260423 → encoding-axis-20260423
- Update every reference in
  docs/sdk-cost-evaluation.md, TODO.md,
  deploy/scripts/run-sdk-cost-sweeps.sh, and the FINDINGS doc.

Data unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sharpen raw-buffer rationale in SDK cost eval

The previous 'Why this baseline exists' blurb framed raw-buffer as
a generic reference point but didn't pin down (a) that OTel's
native aggregators are lossy *specifically over the emit period
W*, or (b) that the experiment's value comes from running
raw-buffer at the *same* W as the sketch encodings so the
comparison is apples-to-apples.

Rewrite leads with those two observations, then closes on the
'without it, the ablation collapses into sketch vs nothing' line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: explain Aggregation<X>Delta as a design decision, not a TODO

The previous 'Adding Aggregation<X>Delta (×5)' block read like a
PR checklist — semantics / encoding / LOC estimate. This is a
design doc, not a task tracker.

Rewrite leads with *what* delta encoding is (full state vs. sparse
diff of changed cells; aggregator keeps prev-tick state; full
re-emit on L / param change) and *why* it deserves a distinct
slot in the encoding-axis evaluation:

  1. The bandwidth claim 'sketch < raw' has two independent
     factors (sketch payload vs raw, and delta vs full sketch).
     Measuring *-full and *-delta as separate rows on the
     encoding axis lets each factor be attributed directly.
  2. Delta is a memory-for-bandwidth trade: the aggregator
     holds prev-tick state alongside current, roughly doubling
     RSS. That tradeoff belongs next to the bandwidth savings
     in the encoding row, not buried as an implementation note.

Also notes that KLL is intentionally absent from the delta column
and points at the existing note above for the rationale.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: shorten section headings

AggregationRawBuffer design → AggregationRawBuffer
Aggregation<X>Delta — what it is and why it's a separate slot →
Aggregation<X>Delta

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* deploy: rename run-three-axis-sweep.sh → run-sdk-cost-sweep.sh

Last paper-shaped artefact in the cost-evaluation tooling. The
inner grid driver is now 'run-sdk-cost-sweep.sh' (singular —
runs one sweep over the grid the caller specifies), and the
outer wrapper stays 'run-sdk-cost-sweeps.sh' (plural — runs the
three canonical sub-sweeps).

Updates every reference in the evaluation doc, findings, TODO,
and the wrapper script itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* deploy: rename sweep scripts by role (grid vs eval)

- run-sdk-cost-sweep.sh  → run-sdk-cost-grid.sh
    Runs a W × L × agg grid the caller specifies via env.
- run-sdk-cost-sweeps.sh → run-sdk-cost-eval.sh
    Runs the three canonical sub-sweeps (time / label / encoding)
    that make up the SDK cost evaluation; calls run-sdk-cost-grid
    three times with different fixed/swept axes.

The singular-vs-plural distinction was too subtle; 'grid' vs
'eval' spells out the role difference directly.

Updates references in docs/sdk-cost-evaluation.md, FINDINGS,
TODO.md, and the eval script's own call site.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: merge TODO.md into PROGRESS.md

Two top-level docs were covering the same ground — PROGRESS
drifted into historical session notes, TODO duplicated the
'what's landed' summary and added 'what's left'. Consolidated
into a single PROGRESS.md with three sections:

  - Implemented          — ground-truthed against current code
  - Outstanding          — paper blockers + non-blocker SDK work
  - Future work          — post-paper

Top-level TODO.md deleted. Cross-references in deploy/TODO.md
and deploy/README.md updated to point at PROGRESS.md instead
(the local 'See TODO.md in this directory' ref in README refers
to deploy/TODO.md, kept as-is).

Sketch-processor and aggregator status tables verified against
the patch trees:
  opentelemetry-go-patch/sdk/metric/aggregation.go lists all six
    Aggregation types we document (DDSketch/KLL/CS/CMS/HLL +
    RawBuffer), matching the table.
  opentelemetry-collector-contrib-patch/processor/ has the five
    per-sketch processors + two new *mergeprocessor variants
    (countsketchmerge, countminsketchmerge) surfaced in the
    table.
  deploy/helm/asap/ has Chart.yaml + values.yaml, no templates/
    dir — reflected honestly.

No semantic change; pure consolidation + refresh.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: merge deploy/TODO.md into PROGRESS.md

deploy/TODO.md was ~90% duplicated with PROGRESS.md's
'Outstanding' section — same instrumentation, query-side,
fault-injection, and Helm items, rephrased. Consolidate.

Genuinely unique content folded into PROGRESS.md:

- Helm template landing order (file-by-file sequence, from
  _helpers.tpl through prometheus.yaml/grafana.yaml) — now
  part of the 'Helm chart templates' future-work bullet.
- Fault-injection script names (controller-kill.sh,
  agent-kill.sh, network-partition.sh) — now bullet-listed
  under paper blocker #8.
- Compose polish (per-agent AGENT_ID label, CI check,
  deploy/k8s/ plain-manifest alternative) — new bullet under
  Future work; didn't previously live anywhere else.

Fixed three dangling refs to deploy/TODO.md:
- deploy/README.md — the Helm 'see TODO.md in this directory'
  line now points at ../PROGRESS.md.
- deploy/docker-compose/base.yml header comment — 'tracked
  in deploy/TODO.md' → 'tracked in PROGRESS.md'.
- PROGRESS.md's own sibling-docs list drops the now-dead
  deploy/TODO.md entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: base.yml comment ref to deploy/TODO.md → PROGRESS.md

Follow-up to previous commit which tried to update this line but
lost it to a read-modify-write race. Pure comment fix.

* eval(sdk-cost): profile per-sample vs batched raw-buffer emit

Answers the follow-up: at the same ground-truth raw-sample
rate (10 Hz per series), does it cost more or less to emit
each sample per tick (W=100ms) vs buffer them and ship in a
batch (W=10s)?

Knob changes:
- deploy/fake-exporter/main.go — new EXPORTER_PPROF_ADDR env
  var. When set, opens net/http/pprof on the given address so
  external tools can sample CPU / heap / goroutines live.
  No-op when unset; production runs stay unchanged.

Findings (full detail in
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md,
4 configs × 15s pprof captures against the live gateway):

- At matching cardinality, per-tick emit (W=100ms) uses ~2×
  the CPU of batched (W=10s). The delta isn't data volume —
  it's the per-flush overhead (gRPC ClientConn.Invoke + flate
  writer init) that W=100ms pays 100× more often.
- Per-tick emit keeps RSS flat. Batched W=10s with card=100
  spikes +39 MiB at flush — the protobuf transform
  intermediates (transform.Value / KeyValue / DataPoints)
  dominate heap.
- Config A (per-tick) CPU hotspots: PeriodicReader.collectAndExport
  → otlpmetricgrpc.Export → grpc.ClientConn.Invoke (54 % cum).
- Config B (batched) CPU hotspots at card=100:
  runtime.gcBgMarkWorker (30 %) + runtime.gcDrain (30 %) —
  GC chases the flush burst.
- Neither scales linearly where you'd expect: A's CPU scales
  with 1/W (flush frequency), B's memory scales with
  cardinality × samples_per_window.
- Design corner: small W + modest cardinality is cheap; large
  W + large cardinality is dangerous (memory bursts that
  container cgroups will clip).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* eval(sdk-cost): fine-grained pprof — mutex contention dominates at 1M events/s

Extends the emit-cadence profile to fine-grained scale:
cardinality=1000 × freq_hz ∈ {100, 1000}, so 100 k and 1 M
events/s. Runs the same A(W=100ms) vs B(W=10s) comparison
as the first pass.

Three findings not visible in the small-scale pass:

1. A-vs-B CPU gap collapses. At tiny scale A was ~2× B
   (per-flush fixed cost dominated). At fine scale per-flush
   overhead amortises over 10k–100k dp batches and the gap
   disappears (A:2.8c vs B:3.0c at 100k/s; A:8.1c vs B:9.8c
   at 1M/s — B slightly worse).

2. Hot path shifts from gRPC export to aggregator mutex
   contention. rawBufferValues.measure takes a single
   valuesMu across the whole attribute map. With 1000
   goroutines firing 1000 Adds/s = 1M Lock acquisitions/s,
   lockSlow dominates (~26% of CPU). This is a real design
   issue — a sharded buffer or per-series lock is needed
   before raw-buffer is usable at ≥1M events/s.

3. Heap is dominated by OTLP transform intermediates, not
   the raw-buffer. At 1M events/s ~1 GiB of KeyValue/Value/
   DataPoints objects are allocated per flush in both
   configs. Pool reuse in the upstream transform would help.

Results table + hot-path diff + heap breakdown appended to
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant