mvp #46: add KLL/DDSketch/HLL merge processors, fix gateway emit - #362
Merged
Merged
Conversation
Adds the three missing per-family gateway merge processors so the controller's typed-stage emit produces YAML the patched collector build can actually load. Without these, the gateway emits names (`kllmerge`, `ddsketchmerge`, `hllmerge`) that have no registered processor — every gateway-merge plan silently failed to load and criterion #2 (controller assigns aggregation tasks to executors) was unverifiable. New processors mirror the existing CMS/CountSketch merge processors but consume typed sketch metric types (`KLLSketch` / `DDSketch` / `HLLSketch`) and key accumulators by serialized attribute set. `processMetrics` is pass-through; the only side effect is updating the in-memory accumulator map (matching existing template). - KLL: Proto/Unspecified replace; Msgpack/MsgpackDelta reserved (warn + drop); no delta variant in sketchlib-go. - DDSketch: Proto via `NewFromStateProtoBytes`; ProtoDelta via `ApplyDelta`; Msgpack via `DeserializeMsgpack`; MsgpackDelta reserved (warn + drop). - HLL: Proto via `DeserializeHyperLogLogFromProtoBytes`; Delta via `DeserializeRegisterDelta` + `ApplyRegisterDelta`; Msgpack supported. Other fixes in same PR: - controller/src/config/stage_config.rs: Cms merge name was `countminmerge` — corrected to `countminsketchmerge`, matching the registered processor type. Doc comments updated to reflect the family-specific naming convention. - deploy/scripts/run_mvp_demo.sh: per-metric capture loop now iterates all 6 distinct metrics in mvp-workload.yaml (was 2, causing 4 sketch families' bindings to be missing from the per-metric snapshot used by the §8 verdict). Replay-class comment count corrected from 6 to 7. - gorillas3processor: removed legacy GORILLA1 wire-format files unused after PR #359's cold-path delivery-modes split: `tsdb_block_writer.go` (test-only shim — runtime uses `gorilla.StreamingTSDBBlockBuilder` directly), `encoder.go`, `window.go`, `postings.go`, and their _test.go siblings. Inlined the 7-line `attributesToMap` helper into `processor.go`; added `readAllSamples` test helper into `processor_test.go`. The 3-mode architecture (best_effort / durable_raw / durable_fragment) is unaffected — runtime path is unchanged. Verification: - `go test ./...` in each new processor dir: 4 tests passing each - `go test ./...` in gorillas3processor: ok - `cargo test --manifest-path controller/Cargo.toml -- --test-threads=1`: 621 passed - `python3 -m pytest deploy/scripts/tests/ -q`: 68 passed Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kllmerge,ddsketchmerge,hllmerge) with no registered processor → criterion CountMinSketch in opentelemetry-collector #2 (controller assigns aggregation tasks to executors) silently failed.stage_config.rs(countminmerge→countminsketchmerge, matching the registered processor type).tsdb_block_writer.go,encoder.go,window.go,postings.go+ their_test.gosiblings). 3-mode architecture (best_effort/durable_raw/durable_fragment) unchanged — runtime still usesgorilla.StreamingTSDBBlockBuilder+StreamingFragmentEncoderdirectly.Merge processor design
countminsketchmergeprocessor/countsketchmergeprocessortemplate.KLLSketch/DDSketch/HLLSketch) instead of Gauge.processMetricsreturns md unchanged — only side effect is the in-memory accumulator map.ApplyDelta/ApplyRegisterDelta(orphan deltas warned + dropped). KLL has no delta variant in sketchlib-go.Companion PR
Backend env-var wiring + freshness probe regex generality: ProjectASAP/ASAPQuery-backend#mvp-46-controller-url-and-freshness-probe
Test plan
cargo test --manifest-path controller/Cargo.toml -- --test-threads=1— 621 passedgo test ./...in each new processor dir — 4 tests passing each (kllmerge, ddsketchmerge, hllmerge)go test ./...in gorillas3processor — ok (post legacy-deletion)python3 -m pytest deploy/scripts/tests/ -q— 68 passedbash deploy/scripts/run_mvp_demo.sh --mode asapand confirmcontroller-emitted-configs/STATUS == liveplus all 6 per-metric snapshots populatedasap-otelbinary loads gateway YAML containingkllmerge:/ddsketchmerge:/hllmerge:/countminsketchmerge:without "unknown processor type" errors🤖 Generated with Claude Code