Skip to content

feat: memory pools in otel collector processors - #44

Merged
zzylol merged 5 commits into
mainfrom
43-memory-pool-in-otel-collector
Mar 15, 2026
Merged

feat: memory pools in otel collector processors#44
zzylol merged 5 commits into
mainfrom
43-memory-pool-in-otel-collector

Conversation

@zzylol

@zzylol zzylol commented Mar 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Pool strings.Builder in attributesKey() (called on every data point — hottest path in both processors)
  • Pool hllSeries/kllSeries structs across window flushes; HLL reuses its register array via Reset() to avoid reallocating the sketch's internal storage
  • Pool bytes.Buffer in serializeKLLSketch() (one allocation per series per flush)
  • Bump hllprocessor to local sketchlib-go to gain HyperLogLog.Reset()

Mirrors the sync.Pool pattern already used in the SDK sketch aggregators (PR #42).

Test plan

  • go test ./... passes in both hllprocessor and kllprocessor
  • Run fakemetricload end-to-end with --sketch-type hll and --sketch-type kll in window mode at high cardinality and verify GC pressure reduction with go tool pprof

🤖 Generated with Claude Code

@zzylol @claude
feat: add memory pools to hll/kll collector processors
af5d76c 
Pool strings.Builder in attributesKey() (hot path, called per data point),
hllSeries/kllSeries structs across window flushes (with HLL register-array
reuse via Reset()), and bytes.Buffer in KLL sketch serialization.

Mirrors the sync.Pool pattern already used in the SDK sketch aggregators.
hllprocessor bumps to local sketchlib-go to gain HyperLogLog.Reset().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zzylol zzylol linked an issue Mar 15, 2026 that may be closed by this pull request
Memory pool in otel collector #43
Closed
@zzylol @claude
feat: add memory pools to countsketch/countminsketch processors
3b7ae0f 
countsketchprocessor: pool the two global row/col CountSketch objects
across flushes via Reset() instead of allocating new ones each window.

countminsketchprocessor: pool windowSketch structs with CMS Reset()
across window flushes (same pattern as hll/kll series pools); pool
strings.Builder in encodeAttributesAsKey() and bytes.Buffer in
serializeCMS(). Both processors bump to local sketchlib-go for Reset().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zzylol zzylol changed the title feat: memory pools in hll/kll collector processors feat: memory pools in otel collector processors Mar 15, 2026
zzylol and others added 3 commits March 15, 2026 03:27
@zzylol @claude
refactor: use sketchlib-go public serialization APIs; restore KLL Res…
6a82f18 
…et()

Replace direct struct field access (Compactors, Count/Sum/Sum2/L1/L2
arrays) with SerializeToBytes()/DeserializeXFromBytes() in kllprocessor
and countminsketchprocessor. Removes custom gob snapshot types entirely.

kllprocessor now points at local sketchlib-go, enabling Reset() on KLL
sketch compactor arrays in the series pool (full sketch reuse on flush).
Update tests to deserialize via library APIs instead of snapshot structs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zzylol @claude
chore: sync opentelemetry-go submodule with sketch aggregator memory …
9407c6a 
…pool changes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zzylol @claude
chore: revert opentelemetry-go submodule to 4575a97
45b6008 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zzylol
zzylol merged commit 53fd71c into main Mar 15, 2026
@zzylol
zzylol deleted the 43-memory-pool-in-otel-collector branch March 15, 2026 08:30
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
@zzylol
Merge pull request #44 from ProjectASAP/43-memory-pool-in-otel-collector
60b1f1d 
feat: memory pools in otel collector processors
@zzylol zzylol mentioned this pull request May 5, 2026
Merged
5 tasks
zzylol added a commit that referenced this pull request May 5, 2026
@zzylol @claude
phase3(parity): un-ignore countsketch byte-parity test, route wrapper…
b147538 
… through INT64/L2 (#253)

Pairs with asap_sketchlib PR #44, which ports
`sketchlib-go::common.HashSpec` / `DeriveIndex` / `DeriveSign` into a
new public module `asap_sketchlib::common::hashspec` and refactors
`CountSketch::update` to feed the matrix through the same packed-`u64`
hash-once → bit-slice-rows pipeline that Go uses. With that upstream
fix landed, the matrix layout matches Go cell-for-cell and the
remaining gap is wrapper-side: the Rust wrapper was emitting a
`CountSketchState` with `counter_type = FLOAT64` + `counts_float`,
while Go's `CountSketch.SerializePortable` emits
`counter_type = INT64` + `counts_int` + per-row `l2`.

This commit:

- Updates `CountSketchWrapper::build_state` to mirror Go's
  `SerializePortable`: emit packed sint64 `counts_int` (Opt-2: 4–8×
  smaller than f64 for typical small-integer counter values) and
  per-row L2 norms derived as `l2[r] = sum_c counts[r][c]^2`. Both
  fields are required for byte parity; without them the envelope
  diverges in `counter_type`, `counts_*`, and `l2` simultaneously.
- Updates `integration/parity/golden_test.go::TestGenerateGoldenFixtures`
  → `t.Run("CountSketch", …)` to strip `Producer` / `HashSpec`
  metadata before marshalling (matching the DDSketch / KLL / HLL
  cases) and to clear `HhKeys` (the Rust wrapper does not maintain a
  Space-Saving candidate tracker, so this list cannot be reproduced;
  downstream rebuilds TopK from the merged matrix anyway).
- Removes the `#[ignore]` from
  `cross_language_parity::countsketch_byte_parity_with_go`. Regenerate
  the golden fixture with
  `GOLDEN_REGEN=1 go test -run TestGenerateGoldenFixtures ./integration/parity/...`
  and the test passes byte-equality against
  `integration/parity/golden/countsketch_envelope.bin`.

Closes part of #243.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 5, 2026
@zzylol @claude
docs: record byte-parity completion + five-claim eval framing (#255)
c8a96a4 
Two updates rolled into one PR:

## PROGRESS.md — byte-parity completion (closes #243 follow-on)

Adds a top-of-file dated section "Cross-language byte-format parity,
5/5 sketches (2026-05-05)" covering the chain of work that closed
#243:

- asap_sketchlib PRs #43 / #44 / #45 — HLL hash-seed alignment,
  CountSketch HashSpec / derive_index / derive_sign port (also
  extracts the shared `common::hashspec` module), CountMinSketch
  port routing through the same primitives.
- ASAPCollector PR #254 — un-ignores `cms_byte_parity_with_go` with
  the matching FO-mode wrapper rewrite, and fixes a latent
  inconsistency in the HLL fixture generator that #252 missed
  (HLL was using `SerializeProtoBytes()` directly while every
  other sketch goes through `SerializePortable*` + strip + Marshal;
  the test only "passed" because fixtures are gitignored and rarely
  regenerated alongside a test run).
- Verification: 7/7 cross_language_parity tests pass with
  `--include-ignored`, including all 5 byte-parity tests + 2 sanity
  tests, no ignored remaining.

Also flags the downstream unblock: `ASAPQuery-backend`'s
`edge_runtime_consumes_precompute_rs.rs` HLL / CS / CMS round-trip
tests were `#[ignore = "blocked on ASAPCollector#243"]` per
`design-phase3-asap-precompute-rs.md` — they should now pass without
backend code changes (mechanical un-ignore PR pending, separate
work).

## docs/paper-outline.md — five-claim eval framing

The "Measurable benefits at every layer" bullet was 4 lines listing
benefits without pointing at *how* the paper proves each one.
Replaces it with five named evaluation dimensions (transmission
bandwidth, edge CPU, edge memory, query accuracy, query latency)
plus the combined Pareto headline claim, with explicit
evidence-tooling pointers per claim:

- bandwidth: `run_e2e_sweep.sh` (P7) bytes columns + `e2e_plots.py`
  (P9) bandwidth-vs-N + `cardinality_crossover/` single-host
  pre-compare.
- edge CPU: P7 producer-side cpu column + `bench_2node_sim.sh` +
  the SDK label-axis profile (paper blocker #2).
- edge memory: P7 rss column + `bench_soak.sh` (steady-state
  RSS / heap / fd-count + slope-based leak verdict).
- accuracy: `raw_tee.go` (P4) ground truth + `accuracy_reduce.py`
  (P8) join + `ASAPQuery-backend/TODO.md` "Accuracy-profile library
  per sketch type" + `sketchlib-bench/docs/DESIGN.md` for bound
  derivation.
- query latency: `promql_replay.py` (P5) p50/p99 + P9
  `query_latency_cdf.png` + `plan_transition.py` (P6) transition
  timings.

The combined Pareto claim (`pareto_acc_vs_thru.png` from P9) is
named explicitly as the figure the paper's contribution rests on.
The "Formal correctness" bullet stays unchanged.

Architectural sibling — the existing
[`design-asap-edge-framework.md`](design-asap-edge-framework.md) —
covers the system shape; this rewrite makes the empirical bar for
"done" explicit alongside it.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Memory pool in otel collector

1 participant

@zzylol