Skip to content

feat: SDK memory optimization — sketch pooling, series ID fast-path, label heap free - #42

Merged
zzylol merged 1 commit into
mainfrom
37-sdk-optimization---only-store-labels-once-not-twice
Mar 15, 2026
Merged

zzylol merged 1 commit into
mainfrom
37-sdk-optimization---only-store-labels-once-not-twice

Conversation

@zzylol

@zzylol zzylol commented Mar 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • sync.Pool per sketch aggregator: series structs and their sketch backing arrays (KLL compactors, HLL registers, CMS count tables, etc.) are recycled via Reset()/Clear() instead of reallocated each export cycle. Eliminates up to ~197 KB of heap churn per series per delta export for CountMinSketch.
  • SeriesIDSink fast-path: after the stateful OTLP Dictionary assigns a series ID on first export, subsequent exports short-circuit before any fingerprint string building or map lookups.
  • AttrsClearer: after first registration, the Dictionary zeroes series.attrs via a back-pointer, permanently freeing the []KeyValue heap array (~224 bytes per 5-label series).
  • Dictionary memory management: sync.Pool for *strings.Builder and *seriesEntry; generation-based eviction (staleGenerations=5) for stale series; idle-series eviction in cumulative aggregators (maxIdleCycles=3) to handle dynamic label churn.

Sketch aggregators added

Type Backing array freed per Reset()
HLL 16 KB (2¹⁴ registers)
KLL (k=200) ~25–50 KB (compactor arrays)
DDSketch ~10–30 KB (bucket map)
CountSketch (5×2048) ~82 KB
CountMinSketch (4×2048) ~197 KB

Test plan

  • go test ./sdk/metric/... passes
  • go test ./sdk/metric/internal/aggregate/... passes
  • Build clean: go build ./...
  • Integration test with collector end-to-end

🤖 Generated with Claude Code

@zzylol zzylol linked an issue Mar 15, 2026 that may be closed by this pull request
Updates patch copies of all five sketch aggregators, dictionary, and
metricdata with the Reset()-based sync.Pool recycling, SeriesIDSink
fast-path, and AttrsClearer label heap free changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the 37-sdk-optimization---only-store-labels-once-not-twice branch from 8f0d591 to 1e9b69c Compare March 15, 2026 07:47
@zzylol
zzylol merged commit 2d86bc5 into main Mar 15, 2026
@zzylol
zzylol deleted the 37-sdk-optimization---only-store-labels-once-not-twice branch March 15, 2026 07:49
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
…ore-labels-once-not-twice

feat: SDK memory optimization — sketch pooling, series ID fast-path, label heap free
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDK optimization - only store labels once, not twice

1 participant