feat: per-processor encoding config — MSGPACK wire format option (3/4) - #162
Merged
Merged
Conversation
Adds an `encoding: msgpack` config option to countminsketchprocessor, countsketchprocessor, and hllprocessor so operators can opt into the cross-language MessagePack wire format as an alternative to the default sketchlib proto path. This makes real MessagePack traffic flow on the modified-OTLP data plane — ASAPQuery-backend already accepts it (PRs #9, pmetric constants in #157), sketchlib-go already emits it (PR #51), but no processor selected it until now. ## Scope Three of four sketch processors are wired in this PR: * `countminsketchprocessor` — calls `ws.cms.SerializeMsgpack()` when `encoding: msgpack`, tags the data point with `CountMinSketchEncodingMsgpack`. * `countsketchprocessor` — calls `ws.cs.SerializeMsgpack()`, tags with `CountSketchEncodingMsgpack`. * `hllprocessor` — calls `series.sketch.SerializeMsgpack()` via a new `serializeHLLSketch(sketch, enc)` helper that returns `(payload, encodingTag, err)`, tags with `HLLSketchEncodingMsgpack`. `ddsketchprocessor` is intentionally **deferred** — it uses `github.com/DataDog/sketches-go`, not sketchlib-go, so it can't call `SerializeMsgpack` directly. Enabling MSGPACK for ddsketchprocessor requires a conversion shim that walks the DataDog sketch's buckets, builds a sketchlib-go `DDSketchState` proto (the same shape sketchlib-go [PR #52](ProjectASAP/sketchlib-go#52) introduced via `NewFromStateProtoBytes`), and calls `SerializeMsgpack` on the reconstructed sketchlib-go sketch. Tracked as a separate follow-up because (a) the conversion code is ~100 lines of bucket flattening, (b) DataDog's proto doesn't carry Sum/Min/Max so msgpack emission from that source is lossy, and (c) the long-term fix is a full ddsketchprocessor migration to sketchlib-go internally, which is a much bigger refactor. ## Delta transmission stays proto-only All three processors keep delta transmission on the proto path when `delta_transmission: true` is set. Sketchlib-go has `SerializeMsgpack` for full sketch state but no matching delta wire format — tracked upstream until sketchlib-go grows an `apply_delta` API parallel to its proto one. The net effect: `encoding: msgpack` + `delta_transmission: true` emits proto deltas for sparse windows and never falls back to msgpack-full for those. ## Config shape Each processor's `Config` gains: ```yaml encoding: msgpack # "proto" (default) or "msgpack" ``` New `SketchEncoding` string type + `EncodingProto` / `EncodingMsgpack` constants per processor. `Validate()` rejects unknown values with a clear error rather than silently falling back. ## Validation Same pre-existing `go.opentelemetry.io/collector/processor/selfmonitor` module-resolution issue as the earlier typed-DP refactor PRs blocks local `go build`. gofmt is clean on all 6 modified files. All API methods used (`SerializeMsgpack` on each sketch type, `*SketchEncodingMsgpack` on the pmetric patch) are already available: * sketchlib-go `CountMinSketch.SerializeMsgpack` — PR #51 * sketchlib-go `CountSketch.SerializeMsgpack` — PR #51 * sketchlib-go `HyperLogLog.SerializeMsgpack` — PR #51 * pmetric `CountMinSketchEncodingMsgpack` — PR #157 * pmetric `CountSketchEncodingMsgpack` — PR #157 * pmetric `HLLSketchEncodingMsgpack` — PR #157 ## Follow-ups * `ddsketchprocessor` MSGPACK option via a DataDog→sketchlib-go conversion shim (or a full internal migration). Tracked. * Delta msgpack wire format (requires sketchlib-go upstream work). * Full ddsketchprocessor migration from DataDog/sketches-go to sketchlib-go DDSketch — bigger refactor, not in this PR's scope. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an
encoding: msgpackconfig option tocountminsketchprocessor,countsketchprocessor, andhllprocessorso operators can opt into the cross-language MessagePack wire format as an alternative to the default sketchlib proto path.This makes real MessagePack traffic flow on the modified-OTLP data plane — ASAPQuery-backend already accepts it (#9, pmetric constants in #157), sketchlib-go already emits it (#51), but no processor selected it until now.
Scope: 3 of 4 sketch processors
countminsketchprocessorws.cms.SerializeMsgpack()CountMinSketchEncodingMsgpackcountsketchprocessorws.cs.SerializeMsgpack()CountSketchEncodingMsgpackhllprocessorseries.sketch.SerializeMsgpack()via newserializeHLLSketch(sketch, enc)helperHLLSketchEncodingMsgpackddsketchprocessoris intentionally deferred — it usesgithub.com/DataDog/sketches-go, not sketchlib-go, so it can't callSerializeMsgpackdirectly. Enabling MSGPACK there requires a conversion shim that walks the DataDog sketch's buckets, builds a sketchlib-goDDSketchStateproto (the shape sketchlib-go #52 introduced viaNewFromStateProtoBytes), and callsSerializeMsgpackon the reconstructed sketchlib-go sketch. Tracked as a separate follow-up because (a) the conversion is ~100 lines, (b) DataDog's proto doesn't carry Sum/Min/Max so msgpack emission is lossy from that source, and (c) the long-term fix is a full ddsketchprocessor migration to sketchlib-go internally.Delta transmission stays proto-only
All three processors keep delta transmission on the proto path when
delta_transmission: true. Sketchlib-go hasSerializeMsgpackfor full state but no matching delta wire format.encoding: msgpack+delta_transmission: trueemits proto deltas for sparse windows and never falls back to msgpack-full for those. Tracked upstream until sketchlib-go grows anapply_deltaparallel.Config shape
Each processor's
Configgains:New
SketchEncodingstring type +EncodingProto/EncodingMsgpackconstants per processor.Validate()rejects unknown values with a clear error rather than silently falling back.Validation
Same pre-existing
go.opentelemetry.io/collector/processor/selfmonitormodule-resolution issue as the earlier typed-DP refactor PRs (#158/#159/#160/#161) blocks localgo build.gofmt -l: clean on all 6 modified filesCountMinSketch.SerializeMsgpack(#51)CountSketch.SerializeMsgpack(#51)HyperLogLog.SerializeMsgpack(#51)CountMinSketchEncodingMsgpack(#157)CountSketchEncodingMsgpack(#157)HLLSketchEncodingMsgpack(#157)Follow-ups
ddsketchprocessorMSGPACK option via DataDog→sketchlib-go conversion shim (or a full internal migration)ddsketchprocessormigration fromDataDog/sketches-goto sketchlib-go — bigger refactorStack
SerializeMsgpackon each sketch typeNewFromStateProtoBytes(used by the DDSketch follow-up)🤖 Generated with Claude Code