feat(precompute): opt-in sparse in-memory HLL for warm aggregators - #472
Merged
Merged
Conversation
Add an opt-in sparse base for the HLL precompute wrapper so low-cardinality warm series no longer pay the dense ~16KB/series register-array cost. - sketches/hll.go: add NewHLLWrapperSparse(), backed by hll.NewSparseHyperLogLog() (sketchlib-go #66). A new `sparse` field on HLLWrapper is threaded through the shared newSketch() helper so Reset / Merge / ApplyDelta rebuild a sparse base and a sparse wrapper never reverts to the dense footprint. The wrapper drives the inner sketch only through its public methods and never touches the exported Registers field. Snapshot / Merge / ApplyDelta / Reset / EstimateCardinality are unchanged and remain byte-identical / interoperable with the dense base. - config.go: document the recognized HLL "sparse" SketchParams key. - sketches/hll_sparse_test.go: estimate parity, byte-identical snapshots, dense<->sparse merge interop, Reset, and a low-cardinality heap check. Dependency: sketchlib-go #66 is MERGED to main (fff038b); resolved via the existing `replace => ../../sketchlib-go` directive, so no go.mod change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ssor Complete the processor side of the opt-in sparse HLL feature (engine side landed in the preceding commit). Lives in the contrib-patch overlay (opentelemetry-collector-contrib-patch/processor/asapedgeprocessor), the source of truth committed in this private repo; restore_otel_collector_contrib_patches.sh stages it onto the upstream-pinned submodule for build/test. - config.go: add `HLLSparse bool` (mapstructure "hll_sparse") to MetricFamily. Default false (dense). Opt-in selects the sparse in-memory HLL base for an HLL family so low-cardinality warm series avoid the dense ~16KB/series register array; serialized output is byte-identical to dense (pure in-memory footprint win, no wire change). Only consulted for family=hll. - config_validate.go: reject hll_sparse on any non-HLL family at boot (mirrors the emit_heap family guard) so a misconfiguration surfaces early. - warm_sketch.go: in the FamilyHLL branch, build sketches.NewHLLWrapperSparse() when fam.HLLSparse, else the dense NewHLLWrapper(). The constructor is the source of truth for base selection; the choice is also surfaced as the documented HLL "sparse" SketchParams key (1=sparse, absent=dense) on the emitted PrecomputeConfig for introspection. - hll_sparse_test.go: assert default => dense (SketchParams[sparse] absent), hll_sparse=true => SketchParams[sparse]=1, both factories build a usable *sketches.HLLWrapper, plus config round-trip + the non-HLL family guard. Remove the now-obsolete asapedge-hll-sparse-wiring/README.md placeholder (it captured the pending fragments; the wiring is now committed). Rebased onto merged #471 (dual-mode aggregation) and #474 (HLL delta invariant). The sketchlib-go NewSparseHyperLogLog dependency is merged to main and resolves via the existing local replace; no pin. Build/test: asap-precompute-go `go test ./... && go vet ./...` pass; the asapedgeprocessor module builds, vets, gofmts clean, and `go test ./...` passes (incl. the new sparse tests and #471's whole_stream tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
feat/precompute-hll-sparse-wiring
branch
from
June 1, 2026 13:29
4ca6436 to
142b80b
Compare
zzylol
marked this pull request as ready for review
June 1, 2026 13:40
zzylol
marked this pull request as draft
June 1, 2026 13:41
zzylol
marked this pull request as ready for review
June 1, 2026 13:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@.pr472-body.md