Skip to content

fix(kll,cms): use proto decoder on ingest to match emit wire format - #222

Merged
zzylol merged 1 commit into
mainfrom
fix/kll-cms-ingest-proto-decode
May 2, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/kll-cms-ingest-proto-decode

Conversation

@zzylol

@zzylol zzylol commented May 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The kllprocessor and countminsketchprocessor emit proto-encoded sketches (SerializePortable() + proto.Marshal for KLL, SerializeProtoBytesFO() for CMS) but their inbound paths were calling the gob-format decoders (DeserializeKLLSketchFromBytes, DeserializeCountMinSketchFromBytes).
  • All production traffic that emits typed KLLSketch / CountMinSketch data points uses the proto envelope. The wire-format mismatch made every inbound pre-aggregated sketch fail to deserialize and silently drop on merge.
  • Switched both ingest sites to the matching proto decoders: kll.DeserializeKLLSketchFromProtoBytes and cms.DeserializeCountMinSketchFromProtoBytes — symbols that already exist in sketchlib-go (verified at /home/.../sketchlib-go/sketches/KLL/portable.go:58 and .../CountMinSketch/portable.go:111).

Why this surfaced now

PR #220 (3b05c93) audited the processor sink/wire-format expectations end-to-end as part of unbreaking the post-DataDog→sketchlib-go fixtures. While reviewing the existing TestBatchModeTransmitSketch (which already correctly decodes the emit payload via DeserializeKLLSketchFromProtoBytes), the asymmetry on the ingest path stood out: the emit and ingest sides of the same processor disagreed on the wire format. TestProcessor_TumblingWindow_Correctness in CMS shows the same pattern — emit decoded with DeserializeCountMinSketchFromProtoBytes, but deserializeCMS (the function used on ingest) calls the gob path.

What the round-trip test proves

Added TestRoundTripIngestProtoSketch in both packages. Each test:

  1. Builds a source sketch with a known multiset (KLL: 10 floats; CMS: 7 inserts at known hashes).
  2. Serializes it via the processor's own emit-side helper (serializeKLLSketch / serializeCMS) — i.e. exactly the bytes a peer collector would put on the wire.
  3. Feeds those bytes back into the same processor as a typed KLLSketch / CountMinSketch input data point.
  4. Verifies the processor's emitted output decodes to the same Count() (KLL) / per-hash frequency estimates (CMS) as the source.

Pre-fix, both new tests fail (KLL: no output emitted because deserialize errored before merge; CMS: gob: duplicate type received while trying to gob-decode proto bytes). Post-fix both pass, exercising the full ingest → merge → emit → decode round-trip on the production wire format.

No emit-side behavior change

The diff is one-line per processor on the ingest path only:

  • kllprocessor/processor.go lines 208 and 462: DeserializeKLLSketchFromBytesDeserializeKLLSketchFromProtoBytes.
  • countminsketchprocessor/processor.go line 693 (deserializeCMS): same swap.

serializeKLLSketch, serializeCMS, appendTypedKLLSketchDataPoint, buildWindowMetricsAndReset, cloneCMS, the delta-encoding ingest branch, and every other emit-related callsite are untouched. Wire format on the emit side is unchanged; downstream consumers see byte-identical output.

Test plan

  • cd opentelemetry-collector-contrib-patch/processor/kllprocessor && go test ./... — passes (includes new TestRoundTripIngestProtoSketch).
  • cd opentelemetry-collector-contrib-patch/processor/countminsketchprocessor && go test ./... — passes (includes new TestRoundTripIngestProtoSketch).
  • Confirmed both new tests fail when only the ingest deserializer is reverted to the gob path (catches the bug they document).

🤖 Generated with Claude Code

KLL and CMS processors emit proto-encoded sketches but were calling the
gob decoder on inbound. Production traffic is proto, so inbound merges
failed silently. Match emit format on both ends; round-trip test added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit f5d5f8a into main May 2, 2026
@zzylol
zzylol deleted the fix/kll-cms-ingest-proto-decode branch May 2, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant