feat(fake-exporter): three-axis SDK knobs + 4-dim label schema - #190
Merged
Merged
Conversation
PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Formalizes the SDK-side decision point as a (W, L, agg_type) triple: time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric. - docs/sdk-aggregation-three-axis-design.md (new) — authoritative design doc for the framework. Maps to existing Mode 1/2/3 vocabulary in delta-transmission-design.md. Enumerates the outstanding SDK-side aggregators (raw-buffer + 5 delta variants) and the SDK runtime hook (hot-reload AttributeFilter) needed to exercise the full planner loop. - docs/paper-outline.md §6 — split §6.2 into four sub-sweeps (a/b/c/d) along the three axes, introduce §6.5 planner-quality as an independent experiment, update claims table to express bandwidth reduction as a three-factor product. - PROGRESS.md — bump date to 2026-04-23; point at new design doc; enumerate outstanding aggregators (~150 LOC raw-buffer + ~500 LOC five delta variants) + fake-exporter knobs + measure-baseline producer-side columns. - docs/n10-bottleneck-rca.md — prepend postscript noting the Options A–D recommendation section is superseded by the three-axis design. Diagnosis content unchanged. - TODO.md (top-level) — retire "N=10 throughput collapse" as P0 blocker (the RCA closed it); replace with the concrete implementation punch list for the three-axis framework. - deploy/TODO.md — add producer-side measurement requirement (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the instrumentation P1 list; the three-axis sweeps read these from fake-exporter container directly rather than inferring from gateway counters. Includes the n10-bottleneck-rca.md content from the earlier diag/n10-bottleneck-rca branch (superseded by this PR — close that one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows that DeltaTransmission is already a field on DDSketch / CountSketch / CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so there's no need to ship five new AggregationDelta<X> types. Only KLL lacks delta support, and its multi-level sample-buffer structure doesn't admit a naive byte-diff — separate design problem, not a §6.2 blocker. Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer (~150 LOC). Proceeding to implement that next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds AggregationRawBuffer to the SDK as the encoding-axis baseline
for the three-axis framework (see docs/sdk-aggregation-three-axis-design.md):
where every Counter.Add / Gauge.Record call becomes its own
NumberDataPoint on the wire instead of being reduced to a Sum or
LastValue per attribute set per tick.
Implementation:
- sdk/metric/aggregation.go — new AggregationRawBuffer{MaxEventsPerSeries}
public type, wired into the Aggregation interface and err()
validation alongside the existing sketch aggregators.
- sdk/metric/internal/aggregate/rawbuffer.go — aggregator impl. Per
attribute.Distinct key, appends (ts, value) to a bounded slice in
measure(); in collect() emits each buffered sample as its own
metricdata.DataPoint[N] inside a Gauge[N], then clears the buffer.
Overflow on a single series is silently dropped with a per-series
drop counter (exposing that via a side-channel metric is a
follow-up tracked in PROGRESS.md).
- sdk/metric/internal/aggregate/aggregate.go — Builder.RawBuffer
method, mirroring the KLLSketch / HLLSketch shape.
- sdk/metric/pipeline.go — dispatch into Builder.RawBuffer +
isAggregatorCompatible.
Tests: four unit tests covering the core contract (every sample
emitted, second collect is empty i.e. no cumulative semantics
retained, per-series cap honoured, many-attribute-set isolation).
Design notes:
- Both delta and cumulative paths call the same collect(). Raw-buffer
has no meaningful cumulative semantics — re-emitting all history
every tick would be useless — so the buffer always resets after
collect regardless of requested temporality.
- We cap per-series, not globally, because the L (label-projection)
axis sweep intentionally drives cardinality down; a global cap
would couple the two axes.
Pre-existing hllsketch build break fixed in passing:
- sketchlib-go renamed HyperLogLog.Insert → InsertValue (float64
arg) and Estimate → EstimateCardinality. Two-line patch in
hllsketch.go to restore a green build. The sketchlib-go
call surface is unchanged at semver zero, so this is
mechanical drift, not a behaviour change.
Stacked on docs/sdk-three-axis-framework — review that first for
the design context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrites the fake-exporter to expose the three-axis knobs defined
in docs/sdk-aggregation-three-axis-design.md. The paper's §6.2
sub-sweeps (time / label / encoding) drive all three via env:
EXPORTER_SDK_WINDOW PeriodicReader interval (default 15s).
Paper's W axis.
EXPORTER_SDK_PROJECTION Comma-separated attribute keys to
keep; "" = keep all, "-" = drop all.
Paper's L axis, implemented via
sdkmetric.View AttributeFilter.
EXPORTER_SDK_AGG Aggregator kind. Paper's encoding
axis. Supported:
default | sum | raw-buffer |
dd-full | dd-delta |
kll |
cms-full | cms-delta |
cs-full | cs-delta |
hll-full | hll-delta
Paper raw-baseline story now wires the SDK-native AggregationRawBuffer
(landed in feat/aggregation-raw-buffer), so every Counter.Add and
Gauge.Record becomes its own NumberDataPoint on the wire at each
window flush — no more conflating the encoding axis with the
per-tick SDK aggregation behaviour.
Workload changes:
- Drops `EXPORTER_RATE` (was a no-op under SDK aggregation; see
docs/n10-bottleneck-rca.md). Warns loudly when set so stale
compose files surface.
- Adds `EXPORTER_FREQ_HZ` (default 10 Hz per series) as the
app-layer event rate. Orthogonal to SDK_WINDOW.
- Widens the synthetic label schema from 2 dims (zone, pod) to
4 dims (zone, rack, node, pod). Default max cardinality is now
4 × 10 × 25 × 10 = 10000; every EXPORTER_*_VALS override is a
separate env. The L-axis sweep needs at least 3 dims to cover
{full, 3-of-4, 2-of-4, 1-of-4, 0-of-4}.
- One goroutine per series, each ticking at period = 1/FREQ_HZ.
Cleaner isolation than the old shared ticker + mu.Lock pattern.
Build:
- go.mod now pins v1.41.0 and replaces every otel sibling at
../../opentelemetry-go (the combined upstream + patch tree
produced by restore_opentelemetry_go_patches.sh). Replaces must
live here because the patched sdk/metric module's own replaces
don't apply when this exporter is the main module.
- Dockerfile.fake-exporter now takes sketchlib-go via
BuildKit --build-context (no submodule hop), copies in the
opentelemetry-go combined tree, and rewrites the sketchlib-go
path in go.mod for /src.
- Locally verified: go build clean, image builds clean,
binary starts and parses all three knobs + deprecation warning.
Follow-up: measure-baseline.py producer-side columns, then
§6.2 sweeps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.
Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:
6.2a time: WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
6.2b label: WINDOWS="60s" PROJECTIONS=": zone,rack zone -"
AGGS="dd-full"
6.2c encoding: WINDOWS="60s" PROJECTIONS="zone,rack"
AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"
Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
labels" (the natural empty-string would be awkward to pass
through a space-separated list) and "-" for "drop all".
decode_projection() maps those back to what
fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
(OTLP → batch(1s) → OTLP), which keeps the agent from running
a sketch pipeline on top and conflating the SDK-side
measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
is trivially splittable back into the grid for plotting.
Smoke-tested locally against the PR #189/#190/#191 stack:
$ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv
baseline,scale,rate,cardinality,producer_cpu_cores,...
w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...
Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.
Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
New sweep driver that iterates the `(W, L, agg_type)` grid defined
in docs/sdk-aggregation-three-axis-design.md, bringing up the
compose stack once per cell, soaking, calling measure-baseline.py,
and tearing down.
Each of the paper's §6.2 sub-sweeps is a thin wrapper fixing two
axes and varying the third:
6.2a time: WINDOWS="1s 15s 60s 300s" PROJECTIONS=":" AGGS="dd-full"
6.2b label: WINDOWS="60s" PROJECTIONS=": zone,rack zone -"
AGGS="dd-full"
6.2c encoding: WINDOWS="60s" PROJECTIONS="zone,rack"
AGGS="raw-buffer dd-full dd-delta kll cms-full hll-full"
Implementation notes:
- PROJECTIONS uses ":" as a bash-friendly escape for "keep all
labels" (the natural empty-string would be awkward to pass
through a space-separated list) and "-" for "drop all".
decode_projection() maps those back to what
fake-exporter's EXPORTER_SDK_PROJECTION expects.
- Uses baseline-b0a-raw-stream.yml as a neutral agent shape
(OTLP → batch(1s) → OTLP), which keeps the agent from running
a sketch pipeline on top and conflating the SDK-side
measurement. The SDK axis is the one under study.
- Row tag is "w${W}-l${PROJ_RAW}-a${AGG}" so the resulting CSV
is trivially splittable back into the grid for plotting.
Smoke-tested locally against the PR #189/#190/#191 stack:
$ WINDOWS=5s PROJECTIONS=":" AGGS=raw-buffer SOAK_S=30 \
CARDINALITY=50 FREQ_HZ=5 BYTES_WIN=3 \
./deploy/scripts/run-three-axis-sweep.sh > /tmp/sweep.csv
baseline,scale,rate,cardinality,producer_cpu_cores,...
w5s-l:-araw-buffer,N1,,50,0.025,9.754,322.269,...
Driver brings up/down correctly, producer-side columns populate,
CSV is well-formed. Numeric validation of the aggregator output
(e.g., "does raw-buffer really emit every sample under gzip
compression?") belongs to the §6.2c sub-sweep PR that uses this
driver to collect actual data.
Stacked on feat/measure-baseline-producer-columns (#191) — that
PR adds the producer columns this script relies on.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
…iner (#194) base.yml was hard-coding EXPORTER_RATE=1000 and ignoring the new EXPORTER_SDK_* knobs introduced in PR #190. Smoke-testing run-three-axis-sweep.sh (#193) produced suspiciously-low producer_bytes_out_per_s values because every cell of the sweep was silently running the SDK at its defaults (window=15s, agg=default, projection="") instead of the swept values. Switches the fake-exporter container's environment block to `${VAR:-default}` interpolation for every sweep-relevant knob so the sweep driver's env assignments actually reach the container: EXPORTER_FREQ_HZ EXPORTER_SDK_WINDOW EXPORTER_SDK_PROJECTION EXPORTER_SDK_AGG EXPORTER_MAX_BUFFER_PER_SERIES Legacy EXPORTER_RATE is kept but drops to empty by default so the deprecation warning doesn't fire spuriously. Before/after on the same cell (W=5s, card=50, freq=5, agg=raw-buffer, no gzip, 60s soak): before after sdk config logged default/15s raw-buffer/5s producer_bytes_out 1923 B/s 48546 B/s (×25) gateway_points_per_s 3.18 280.4 backend_samples_per_s 2.91 247.9 48 KB/s matches the naive estimate (50 card × 5 Hz × 2 instruments × ~80 B/datapoint ≈ 40 KB/s), confirming AggregationRawBuffer is emitting the expected per-event stream. Before the fix, the SDK was running Sum aggregation on the Counter instead — one data point per attribute set per 15s tick = ~7 pts/s, which matches the low gateway rate seen. Also adds `deploy/fake-exporter/sdk_emit_test.go` as a regression test for the aggregator contract: a ManualReader + AggregationRawBuffer should emit exactly `cardinality × instruments × samples_each` data points. This is the counterpart of the unit test in opentelemetry-go-patch/sdk/metric/internal/aggregate/ but runs at the MeterProvider+View level, catching wiring mistakes like the one above. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the fake-exporter to the three-axis framework (depends on #189, which adds
AggregationRawBuffer). The §6.2 sub-sweeps can now drive one axis at a time via env vars.Knobs
EXPORTER_SDK_WINDOW15sW) —PeriodicReader.WithIntervalEXPORTER_SDK_PROJECTION""L) —View.AttributeFilter;""keeps all,"-"drops all,"zone,rack"keeps subsetEXPORTER_SDK_AGGdefaultdefault|sum|raw-buffer|dd-full|dd-delta|kll|cms-full|cms-delta|cs-full|cs-delta|hll-full|hll-deltaPlus:
EXPORTER_FREQ_HZ(new, default 10) controls per-series event rate;EXPORTER_CARDINALITYunchanged semantics (default 1000) but now sits under a 4-dim schemazone × rack × node × pod(default product 10000). Each schema dim has anEXPORTER_*_VALSoverride.Deprecations:
EXPORTER_RATEdropped. Was a no-op under SDK aggregation (seedocs/n10-bottleneck-rca.md). Warns loudly at startup if set so stale compose files surface.Changes
deploy/fake-exporter/main.go— rewritten. One goroutine per attribute set ticking at1/FREQ_HZ;View.AttributeFilter+Aggregationderived from the new env vars;PeriodicReaderperiod =EXPORTER_SDK_WINDOW. Trace-replay path preserved.deploy/fake-exporter/go.mod— now pinsv1.41.0and replaces every otel module at../../opentelemetry-go(the combined upstream+patch tree). Bumped togo 1.24.0to match the patched SDK's toolchain requirement.deploy/fake-exporter/go.sum— generated bygo mod tidy.deploy/docker/Dockerfile.fake-exporter— takessketchlib-goviaDOCKER_BUILDKIT=1 --build-context, copiesopentelemetry-gointo/src, rewrites the sketchlib replace path for/srclayout.Verified locally
Deprecation warning fires, config parsed, 4-dim schema logged.
Still to do (follow-up PRs)
measure-baseline.pyproducer-side columns (producer_cpu_cores,producer_rss_mib,producer_bytes_out_per_s)baseline-b{0a,0b,1,2,3,5}-*.yml) still setEXPORTER_RATE; not changed here so the deprecation warning lands visibly. Clean up in a separate rename PR once the three-axis sweep driver lands.eval-results/🤖 Generated with Claude Code