Adding DDSketch Metrics Type in OTel - #8
Merged
Merged
Conversation
zzylol
added a commit
that referenced
this pull request
Apr 15, 2026
Last of the four processor typed-emission refactors (PR #158: countmin, PR #159: countsketch, PR #160: hll dead-code cleanup, this PR: kll). Replaces Gauge-with-byte-attribute emission with typed `KLLSketchDataPoint` messages so ASAPQuery-backend's modified-OTLP sketch router sees them as `Metric.data = KLLSketch{...}` variants instead of anonymous Gauges. ## Why Before this PR the `TransmitSketch` path called `appendKLLSketchDataPoint` which emitted: dp := metric.Gauge().DataPoints().AppendEmpty() dp.Attributes().PutInt("kll.k", int64(k)) dp.Attributes().PutInt("kll.count", int64(sketch.Count())) dp.Attributes().PutEmptyBytes("kll.sketch_payload").FromRaw(payload) dp.SetDoubleValue(float64(sketch.Count())) ASAPQuery-backend's modified-OTLP decoder matches on `Metric.data = KLLSketch{data_points: [...]}` (oneof tag 14, typed `KLLSketchDataPoint`). The Rust-side KLL decoder (ASAPQuery PR #8 §KLL, with lossy statistical reconstruction via item replay) was already wired up, but no processor produced the typed data points it wanted. The processor and the decoder were both ready for each other but speaking past each other on the wire. ## What changed ### `processor.go` * **Batch path** (around line 226) — `TransmitSketch` branch now calls `findOrCreateKLLSketchMetric` (creates a typed `pmetric.MetricTypeKLLSketch` metric with `AggregationTemporalityDelta`) and `appendTypedKLLSketchDataPoint` (writes `SetSketch` / `SetEncoding` / `SetCount` on a typed `KLLSketchDataPoint`). * **Windowed path** (around line 486) — same two helpers. The inline `m.SetEmptyGauge()` is replaced with `m.SetEmptyKLLSketch().SetAggregationTemporality(...)`. * New helper `findOrCreateKLLSketchMetric` mirrors the existing `findOrCreateGaugeMetric` but creates typed KLL metrics and matches on `MetricType` so legacy Gauge metrics with the same name don't get accidentally reused. * New helper `appendTypedKLLSketchDataPoint` replaces the old `appendKLLSketchDataPoint`. Writes the sketch payload via `SetSketch` and tags the encoding with `KLLSketchEncodingProto`. Still carries `kll.k` on the attribute map for operator visibility (the typed DP has no dedicated setter for it; the Rust side derives k from the serialized payload). Sum/Min/Max fields on the typed DP are left at zero because sketchlib-go's `KLLSketch` type doesn't track them — KLL is quantile-only and the proto fields are observability-only. * Old `appendKLLSketchDataPoint` function deleted. Old `findOrCreateGaugeMetric` retained because the non-TransmitSketch quantile-emission path (lines 233-246 and 503-540) still exports gauge-shaped scalar quantile series. ### `processor_test.go` `TestBatchMode_TransmitSketch` (lines 68-115) previously asserted on `m.Gauge().DataPoints().At(0)` and read the payload from the `kll.sketch_payload` byte attribute. Updated to: * Assert `m.Type() == pmetric.MetricTypeKLLSketch` * Read the payload from `outDP.Sketch()` directly * Assert `outDP.Encoding() == pmetric.KLLSketchEncodingProto` Other tests in the file (TestBatchModeNoStatePersistence and below) all target the quantile-emission path (`TransmitSketch: false`), which still uses Gauge — they're unaffected. ## Validation Same pre-existing `go.opentelemetry.io/collector/processor/selfmonitor` resolution issue as PRs #158/#159/#160 blocks local `go build`. `gofmt -l` clean on both modified files. All API methods used (`pmetric.MetricTypeKLLSketch`, `SetEmptyKLLSketch`, `KLLSketch().DataPoints().AppendEmpty()`, `SetSketch`, `SetEncoding`, `SetCount`, `KLLSketchEncodingProto`) already exist on the pmetric patch. ## Stack * PR #158 (merged) — countminsketchprocessor typed emission * PR #159 — countsketchprocessor typed emission * PR #160 — hllprocessor dead-code cleanup + broken-test fix * **This PR** — kllprocessor typed emission * Next — per-processor MSGPACK config option (one-line branch each, calling sketchlib-go `SerializeMsgpack` from PR #51 and setting `*SketchEncodingMsgpack` from PR #157). Note: for KLL specifically, sketchlib-go #50/#51 out-of-scope'd the msgpack wire format because sketchlib-go's KLL and sketch-core's KLL don't share a byte-level backend — so KLL producers should keep using `_ENCODING_PROTO`. The MSGPACK enum value in PR #157 is reserved but not usable for KLL today. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 15, 2026
Last of the four processor typed-emission refactors (PR #158: countmin, PR #159: countsketch, PR #160: hll dead-code cleanup, this PR: kll). Replaces Gauge-with-byte-attribute emission with typed `KLLSketchDataPoint` messages so ASAPQuery-backend's modified-OTLP sketch router sees them as `Metric.data = KLLSketch{...}` variants instead of anonymous Gauges. ## Why Before this PR the `TransmitSketch` path called `appendKLLSketchDataPoint` which emitted: dp := metric.Gauge().DataPoints().AppendEmpty() dp.Attributes().PutInt("kll.k", int64(k)) dp.Attributes().PutInt("kll.count", int64(sketch.Count())) dp.Attributes().PutEmptyBytes("kll.sketch_payload").FromRaw(payload) dp.SetDoubleValue(float64(sketch.Count())) ASAPQuery-backend's modified-OTLP decoder matches on `Metric.data = KLLSketch{data_points: [...]}` (oneof tag 14, typed `KLLSketchDataPoint`). The Rust-side KLL decoder (ASAPQuery PR #8 §KLL, with lossy statistical reconstruction via item replay) was already wired up, but no processor produced the typed data points it wanted. The processor and the decoder were both ready for each other but speaking past each other on the wire. ## What changed ### `processor.go` * **Batch path** (around line 226) — `TransmitSketch` branch now calls `findOrCreateKLLSketchMetric` (creates a typed `pmetric.MetricTypeKLLSketch` metric with `AggregationTemporalityDelta`) and `appendTypedKLLSketchDataPoint` (writes `SetSketch` / `SetEncoding` / `SetCount` on a typed `KLLSketchDataPoint`). * **Windowed path** (around line 486) — same two helpers. The inline `m.SetEmptyGauge()` is replaced with `m.SetEmptyKLLSketch().SetAggregationTemporality(...)`. * New helper `findOrCreateKLLSketchMetric` mirrors the existing `findOrCreateGaugeMetric` but creates typed KLL metrics and matches on `MetricType` so legacy Gauge metrics with the same name don't get accidentally reused. * New helper `appendTypedKLLSketchDataPoint` replaces the old `appendKLLSketchDataPoint`. Writes the sketch payload via `SetSketch` and tags the encoding with `KLLSketchEncodingProto`. Still carries `kll.k` on the attribute map for operator visibility (the typed DP has no dedicated setter for it; the Rust side derives k from the serialized payload). Sum/Min/Max fields on the typed DP are left at zero because sketchlib-go's `KLLSketch` type doesn't track them — KLL is quantile-only and the proto fields are observability-only. * Old `appendKLLSketchDataPoint` function deleted. Old `findOrCreateGaugeMetric` retained because the non-TransmitSketch quantile-emission path (lines 233-246 and 503-540) still exports gauge-shaped scalar quantile series. ### `processor_test.go` `TestBatchMode_TransmitSketch` (lines 68-115) previously asserted on `m.Gauge().DataPoints().At(0)` and read the payload from the `kll.sketch_payload` byte attribute. Updated to: * Assert `m.Type() == pmetric.MetricTypeKLLSketch` * Read the payload from `outDP.Sketch()` directly * Assert `outDP.Encoding() == pmetric.KLLSketchEncodingProto` Other tests in the file (TestBatchModeNoStatePersistence and below) all target the quantile-emission path (`TransmitSketch: false`), which still uses Gauge — they're unaffected. ## Validation Same pre-existing `go.opentelemetry.io/collector/processor/selfmonitor` resolution issue as PRs #158/#159/#160 blocks local `go build`. `gofmt -l` clean on both modified files. All API methods used (`pmetric.MetricTypeKLLSketch`, `SetEmptyKLLSketch`, `KLLSketch().DataPoints().AppendEmpty()`, `SetSketch`, `SetEncoding`, `SetCount`, `KLLSketchEncodingProto`) already exist on the pmetric patch. ## Stack * PR #158 (merged) — countminsketchprocessor typed emission * PR #159 — countsketchprocessor typed emission * PR #160 — hllprocessor dead-code cleanup + broken-test fix * **This PR** — kllprocessor typed emission * Next — per-processor MSGPACK config option (one-line branch each, calling sketchlib-go `SerializeMsgpack` from PR #51 and setting `*SketchEncodingMsgpack` from PR #157). Note: for KLL specifically, sketchlib-go #50/#51 out-of-scope'd the msgpack wire format because sketchlib-go's KLL and sketch-core's KLL don't share a byte-level backend — so KLL producers should keep using `_ENCODING_PROTO`. The MSGPACK enum value in PR #157 is reserved but not usable for KLL today. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta
pushed a commit
that referenced
this pull request
Apr 17, 2026
* update otel collector make genpdata * compile ddsketch collector * build ddsketchcol with all pipelines * add README * add gitignore * add gitignore * update * update otel collector make genpdata * compile ddsketch collector * build ddsketchcol with all pipelines * add README * add gitignore * add gitignore * update
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
deploy/TODO.md was ~90% duplicated with PROGRESS.md's 'Outstanding' section — same instrumentation, query-side, fault-injection, and Helm items, rephrased. Consolidate. Genuinely unique content folded into PROGRESS.md: - Helm template landing order (file-by-file sequence, from _helpers.tpl through prometheus.yaml/grafana.yaml) — now part of the 'Helm chart templates' future-work bullet. - Fault-injection script names (controller-kill.sh, agent-kill.sh, network-partition.sh) — now bullet-listed under paper blocker #8. - Compose polish (per-agent AGENT_ID label, CI check, deploy/k8s/ plain-manifest alternative) — new bullet under Future work; didn't previously live anywhere else. Fixed three dangling refs to deploy/TODO.md: - deploy/README.md — the Helm 'see TODO.md in this directory' line now points at ../PROGRESS.md. - deploy/docker-compose/base.yml header comment — 'tracked in deploy/TODO.md' → 'tracked in PROGRESS.md'. - PROGRESS.md's own sibling-docs list drops the now-dead deploy/TODO.md entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 30, 2026
* eval(§6.2): first three-axis sweep pass + findings write-up
Runs the §6.2 a/b/c sub-sweeps defined in
docs/sdk-aggregation-three-axis-design.md at CARDINALITY=1000,
FREQ_HZ=10, N=1, SOAK_S=180, BYTES_WIN=20. 17 cells total,
~65 min wall clock.
Artifacts:
- deploy/scripts/run-paper-6.2-sweeps.sh — wrapper that runs
all three sub-sweeps at named env points and drops CSVs.
- deploy/eval-results/three-axis/6.2a-time-20260423.csv
- deploy/eval-results/three-axis/6.2b-label-20260423.csv
- deploy/eval-results/three-axis/6.2c-encoding-20260423.csv
plus .log files for each sub-sweep stderr.
- deploy/eval-results/three-axis/FINDINGS-20260423.md — results
interpretation with three sections:
- Clean findings (encoding-axis ratios, label-axis RSS drop)
- Methodology caveat (BYTES_WIN < W under-counts bytes)
- Surprises worth re-checking (§6.2b CPU-up-with-projection,
cms-delta memory overhead)
Key clean findings:
- §6.2c encoding: raw-buffer holds 108 MiB RSS vs 33-39 MiB for
sketch encodings (~3× memory cost for per-event storage) and
emits ~7× the wire bytes of any sketch. cms-delta costs 2×
the RSS of cms-full because it holds prev-tick snapshot for
diff.
- §6.2b label: projecting to zone,rack cuts producer RSS from
66.8 → 33.7 MiB as attribute-set folding reduces sketch
instance count.
Known caveat: BYTES_WIN=20s < W=60s in most cells, so
producer_bytes_out_per_s is under-reported by factor ~W/BYTES_WIN.
Ratios within a sub-sweep (all at same W) are unaffected. V2
rerun with BYTES_WIN ≥ 2W will fix absolute numbers — doc'd in
FINDINGS as the top-priority follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rewrite FINDINGS to describe the experiment self-contained
Drops paper-section references from the findings doc so it reads
as experimental results standing on their own, not as "§6.2
sub-sweep output". Structure is now:
1. Experiment design — what the (W, L, agg_type) triple means,
what each sub-experiment varies vs holds fixed, how each
cell is measured.
2. Methodology caveat (unchanged — BYTES_WIN < W under-reports).
3. Results per sub-experiment (encoding axis, label axis,
time axis) with the numbers and what to make of them.
4. Non-obvious findings worth a second look.
5. Next actions.
No numeric content changed; CSVs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rename findings title to 'SDK aggregation experiments'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain W + push vs pull in FINDINGS
Adds a 'What W actually is (and push vs pull)' subsection right
after the (W, L, agg_type) definition so the doc doesn't assume
the reader knows OTel's Reader/PeriodicReader model. Compares
OTLP push and Prometheus pull aggregation-equivalence and
calls out the two places they actually differ for this project
(NAT-friendliness + controller-reconfig channel).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: mark AggregationRawBuffer as landed (#189)
docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md
all still had AggregationRawBuffer flagged ❌. It shipped in
#189 back in April — updating the status rows + cross-referencing
the contract test (deploy/fake-exporter/sdk_emit_test.go) +
the implementation path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: expand AggregationRawBuffer design — why + what
Rewrites the AggregationRawBuffer subsection in the design doc to
explain *why* raw-buffer is a first-class encoding slot (not just
what it does). Key points the previous version didn't make:
- Every other upstream Aggregation is lossy by design; raw-buffer
is the only SDK-native option that preserves the full
observation stream. Without it there's no reference point
for sketch-vs-raw claims.
- It's also the encoding that can serve queries no summary
can: exact events for cold-fallback replay, per-sample audit,
downstream sketch computation the SDK policy didn't
anticipate.
- Memory cost scales as O(W × event_rate × cardinality) —
the tradeoff the encoding axis measures.
Contract + overflow + temporality specifics kept from the
earlier version and folded into dedicated subsections.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rename to 'SDK aggregation cost' + strip landed-date stamps
- Rename `docs/sdk-aggregation-three-axis-design.md` →
`docs/sdk-aggregation-cost.md`. Title updated to 'SDK
aggregation cost'. Old name was a framework description;
new name says directly what the doc is measuring.
- Update every cross-reference in docs, PROGRESS.md, TODO.md,
deploy/ (compose, Dockerfile, go.mod, main.go, scripts,
FINDINGS) to point at the new filename.
- Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status
tables and TODO checkboxes. Merged to main = ✅; the dates
and PR numbers are git-log territory, not running-doc
territory.
No substantive content change; only the framing/file name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove stray __pycache__ + ignore it
* docs: rename to 'SDK cost evaluation' to signal this is the eval doc
File: docs/sdk-aggregation-cost.md → docs/sdk-cost-evaluation.md
Title: 'SDK cost evaluation'
All cross-references in docs/, PROGRESS.md, TODO.md, deploy/
updated. No content change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: delete n10-bottleneck-rca.md; rework kll-delta status to 'no need'
- Delete docs/n10-bottleneck-rca.md — the finding (PR #185's
'N=10 throughput collapse' is SDK pre-aggregation, not a
bottleneck) has propagated into the docs that still need it
(TODO §1 and the SDK cost evaluation doc summarise the
conclusion inline). The long write-up was one-shot debug
context, not a reference.
- Drop the dangling link from docs/sdk-cost-evaluation.md's
header block (and the 'Authoritative for §6.2…' framing —
redundant with the doc content).
- Fix the other references that pointed at the deleted file:
TODO.md §1, PROGRESS.md outstanding-aggregators note,
fake-exporter main.go comment and EXPORTER_RATE deprecation
log, docker-compose/base.yml comment. Each now explains the
point inline.
- Rework the kll-delta status in both docs/sdk-cost-evaluation.md
and PROGRESS.md from '❌ not yet' → '**no need**'. kll-full
covers the encoding-axis comparison; implementing a KLL-specific
delta would need a different strategy than the byte-diff the
other four sketches use, and the payoff doesn't justify it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: strip paper-section references; rename sweep driver; drop follow-up list
Across-the-board cleanup so evaluation docs stand on their own and
don't cite paper section numbers. Evaluations describe design +
methodology + results directly; paper framing belongs only in
docs/paper-outline.md.
- Rename deploy/scripts/run-paper-6.2-sweeps.sh →
run-sdk-cost-sweeps.sh. Filename drops the §6.2 embedding.
- Delete docs/design-stateful-protocol.md's "(Paper §4.5)" title
suffix; title is now just the protocol name.
- docs/sdk-cost-evaluation.md:
- remove the obsolete "Implementation order (follow-up PRs)"
section — most items are merged, and the remaining one
lives in TODO.md anyway
- remove the "Non-goals of this doc" section — pointer list
was stale
- drop the paper-§6 framing in the opening; doc now reads
as standalone evaluation design
- rewrite the ablation table: instead of "6.2a Time axis /
6.2b Label axis / …" with "Claim" column, plain "Time /
Label / Encoding / Combined" with "What the sweep
measures" column
- strip the few remaining §6.2 / §6.5 inline refs
- Strip paper-§N from every config / compose / script / TODO /
PROGRESS comment where it appeared. Specifically:
- PROGRESS.md outstanding-aggregators section
- TODO.md §1 header (now "SDK cost evaluation") + all inline
§6.2 references
- deploy/TODO.md instrumentation section headings + inline
- deploy/README.md "Paper §6 mapping" → "Evaluation → metric
mapping"; drop "paper's §6 eval" framing
- deploy/configs/*.yaml — comment prefixes like "Paper §6.2
baseline:" → plain "Baseline:"
- deploy/configs/prometheus.yml, grafana-datasources.yml —
same
- deploy/docker-compose/*.yml — "Paper §6" → "Scale dials"
etc.
- deploy/helm/asap/{values,Chart}.yaml — same
- deploy/fake-exporter/main.go — "Paper's W axis" / "L axis"
/ "encoding axis" → "Time axis" / "Label axis" / "Encoding
axis"
- deploy/fake-exporter/sdk_emit_test.go — §6.2c → encoding-axis
- Left in place: internal design-doc cross-references like
"delta-transmission-design.md §16.2" where the §N refers to a
section of a design doc rather than the paper. Those are legit
technical writing, not paper citations.
No substantive content or behaviour change anywhere; pure
framing/naming cleanup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* eval: rename artifacts to match SDK cost evaluation naming
- deploy/eval-results/three-axis/ → deploy/eval-results/sdk-cost/
to match the design doc (docs/sdk-cost-evaluation.md).
- CSV + log files lose the paper-section prefix:
6.2a-time-20260423 → time-axis-20260423
6.2b-label-20260423 → label-axis-20260423
6.2c-encoding-20260423 → encoding-axis-20260423
- Update every reference in
docs/sdk-cost-evaluation.md, TODO.md,
deploy/scripts/run-sdk-cost-sweeps.sh, and the FINDINGS doc.
Data unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: sharpen raw-buffer rationale in SDK cost eval
The previous 'Why this baseline exists' blurb framed raw-buffer as
a generic reference point but didn't pin down (a) that OTel's
native aggregators are lossy *specifically over the emit period
W*, or (b) that the experiment's value comes from running
raw-buffer at the *same* W as the sketch encodings so the
comparison is apples-to-apples.
Rewrite leads with those two observations, then closes on the
'without it, the ablation collapses into sketch vs nothing' line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain Aggregation<X>Delta as a design decision, not a TODO
The previous 'Adding Aggregation<X>Delta (×5)' block read like a
PR checklist — semantics / encoding / LOC estimate. This is a
design doc, not a task tracker.
Rewrite leads with *what* delta encoding is (full state vs. sparse
diff of changed cells; aggregator keeps prev-tick state; full
re-emit on L / param change) and *why* it deserves a distinct
slot in the encoding-axis evaluation:
1. The bandwidth claim 'sketch < raw' has two independent
factors (sketch payload vs raw, and delta vs full sketch).
Measuring *-full and *-delta as separate rows on the
encoding axis lets each factor be attributed directly.
2. Delta is a memory-for-bandwidth trade: the aggregator
holds prev-tick state alongside current, roughly doubling
RSS. That tradeoff belongs next to the bandwidth savings
in the encoding row, not buried as an implementation note.
Also notes that KLL is intentionally absent from the delta column
and points at the existing note above for the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: shorten section headings
AggregationRawBuffer design → AggregationRawBuffer
Aggregation<X>Delta — what it is and why it's a separate slot →
Aggregation<X>Delta
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deploy: rename run-three-axis-sweep.sh → run-sdk-cost-sweep.sh
Last paper-shaped artefact in the cost-evaluation tooling. The
inner grid driver is now 'run-sdk-cost-sweep.sh' (singular —
runs one sweep over the grid the caller specifies), and the
outer wrapper stays 'run-sdk-cost-sweeps.sh' (plural — runs the
three canonical sub-sweeps).
Updates every reference in the evaluation doc, findings, TODO,
and the wrapper script itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deploy: rename sweep scripts by role (grid vs eval)
- run-sdk-cost-sweep.sh → run-sdk-cost-grid.sh
Runs a W × L × agg grid the caller specifies via env.
- run-sdk-cost-sweeps.sh → run-sdk-cost-eval.sh
Runs the three canonical sub-sweeps (time / label / encoding)
that make up the SDK cost evaluation; calls run-sdk-cost-grid
three times with different fixed/swept axes.
The singular-vs-plural distinction was too subtle; 'grid' vs
'eval' spells out the role difference directly.
Updates references in docs/sdk-cost-evaluation.md, FINDINGS,
TODO.md, and the eval script's own call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: merge TODO.md into PROGRESS.md
Two top-level docs were covering the same ground — PROGRESS
drifted into historical session notes, TODO duplicated the
'what's landed' summary and added 'what's left'. Consolidated
into a single PROGRESS.md with three sections:
- Implemented — ground-truthed against current code
- Outstanding — paper blockers + non-blocker SDK work
- Future work — post-paper
Top-level TODO.md deleted. Cross-references in deploy/TODO.md
and deploy/README.md updated to point at PROGRESS.md instead
(the local 'See TODO.md in this directory' ref in README refers
to deploy/TODO.md, kept as-is).
Sketch-processor and aggregator status tables verified against
the patch trees:
opentelemetry-go-patch/sdk/metric/aggregation.go lists all six
Aggregation types we document (DDSketch/KLL/CS/CMS/HLL +
RawBuffer), matching the table.
opentelemetry-collector-contrib-patch/processor/ has the five
per-sketch processors + two new *mergeprocessor variants
(countsketchmerge, countminsketchmerge) surfaced in the
table.
deploy/helm/asap/ has Chart.yaml + values.yaml, no templates/
dir — reflected honestly.
No semantic change; pure consolidation + refresh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: merge deploy/TODO.md into PROGRESS.md
deploy/TODO.md was ~90% duplicated with PROGRESS.md's
'Outstanding' section — same instrumentation, query-side,
fault-injection, and Helm items, rephrased. Consolidate.
Genuinely unique content folded into PROGRESS.md:
- Helm template landing order (file-by-file sequence, from
_helpers.tpl through prometheus.yaml/grafana.yaml) — now
part of the 'Helm chart templates' future-work bullet.
- Fault-injection script names (controller-kill.sh,
agent-kill.sh, network-partition.sh) — now bullet-listed
under paper blocker #8.
- Compose polish (per-agent AGENT_ID label, CI check,
deploy/k8s/ plain-manifest alternative) — new bullet under
Future work; didn't previously live anywhere else.
Fixed three dangling refs to deploy/TODO.md:
- deploy/README.md — the Helm 'see TODO.md in this directory'
line now points at ../PROGRESS.md.
- deploy/docker-compose/base.yml header comment — 'tracked
in deploy/TODO.md' → 'tracked in PROGRESS.md'.
- PROGRESS.md's own sibling-docs list drops the now-dead
deploy/TODO.md entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: base.yml comment ref to deploy/TODO.md → PROGRESS.md
Follow-up to previous commit which tried to update this line but
lost it to a read-modify-write race. Pure comment fix.
* eval(sdk-cost): profile per-sample vs batched raw-buffer emit
Answers the follow-up: at the same ground-truth raw-sample
rate (10 Hz per series), does it cost more or less to emit
each sample per tick (W=100ms) vs buffer them and ship in a
batch (W=10s)?
Knob changes:
- deploy/fake-exporter/main.go — new EXPORTER_PPROF_ADDR env
var. When set, opens net/http/pprof on the given address so
external tools can sample CPU / heap / goroutines live.
No-op when unset; production runs stay unchanged.
Findings (full detail in
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md,
4 configs × 15s pprof captures against the live gateway):
- At matching cardinality, per-tick emit (W=100ms) uses ~2×
the CPU of batched (W=10s). The delta isn't data volume —
it's the per-flush overhead (gRPC ClientConn.Invoke + flate
writer init) that W=100ms pays 100× more often.
- Per-tick emit keeps RSS flat. Batched W=10s with card=100
spikes +39 MiB at flush — the protobuf transform
intermediates (transform.Value / KeyValue / DataPoints)
dominate heap.
- Config A (per-tick) CPU hotspots: PeriodicReader.collectAndExport
→ otlpmetricgrpc.Export → grpc.ClientConn.Invoke (54 % cum).
- Config B (batched) CPU hotspots at card=100:
runtime.gcBgMarkWorker (30 %) + runtime.gcDrain (30 %) —
GC chases the flush burst.
- Neither scales linearly where you'd expect: A's CPU scales
with 1/W (flush frequency), B's memory scales with
cardinality × samples_per_window.
- Design corner: small W + modest cardinality is cheap; large
W + large cardinality is dangerous (memory bursts that
container cgroups will clip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* eval(sdk-cost): fine-grained pprof — mutex contention dominates at 1M events/s
Extends the emit-cadence profile to fine-grained scale:
cardinality=1000 × freq_hz ∈ {100, 1000}, so 100 k and 1 M
events/s. Runs the same A(W=100ms) vs B(W=10s) comparison
as the first pass.
Three findings not visible in the small-scale pass:
1. A-vs-B CPU gap collapses. At tiny scale A was ~2× B
(per-flush fixed cost dominated). At fine scale per-flush
overhead amortises over 10k–100k dp batches and the gap
disappears (A:2.8c vs B:3.0c at 100k/s; A:8.1c vs B:9.8c
at 1M/s — B slightly worse).
2. Hot path shifts from gRPC export to aggregator mutex
contention. rawBufferValues.measure takes a single
valuesMu across the whole attribute map. With 1000
goroutines firing 1000 Adds/s = 1M Lock acquisitions/s,
lockSlow dominates (~26% of CPU). This is a real design
issue — a sharded buffer or per-series lock is needed
before raw-buffer is usable at ≥1M events/s.
3. Heap is dominated by OTLP transform intermediates, not
the raw-buffer. At 1M events/s ~1 GiB of KeyValue/Value/
DataPoints objects are allocated per flush in both
configs. Pool reuse in the upstream transform would help.
Results table + hot-path diff + heap breakdown appended to
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.