feat(precompute): dual-mode aggregation (per-series vs whole-stream) unified under Mode - #471
Merged
Merged
Conversation
zzylol
force-pushed
the
feat/dual-mode-aggregation
branch
4 times, most recently
from
May 31, 2026 17:25
4bbb1ca to
eba0913
Compare
…unified under Mode Add an explicit per-aggregation scope (AggMode: ModePerSeries / ModeWholeStream) to PrecomputeConfig.Scope, selectable by the control plane. PerSeries (zero value) is today's behavior; WholeStream collapses every matching datapoint into a single sketch per AggID and emits one envelope per AggID per window. The legacy GlobalAggregation bool is subsumed: effectiveScope() folds it into ModeWholeStream so one whole-stream code path drives both (the emit_heap CountSketch keeps working). UpdateConfig resets the in-flight window on a scope flip (the series map is keyed incompatibly across scopes); same-scope changes stay in place. MaxSeries is a no-op in WholeStream (cardinality is 1). Per-family the ingested subject is the metric value by default, or the inner item dimension when asap_edge ItemLabel is set (reconciled with the existing ItemLabel, no new knob). asap_edge surfaces a `mode` mapstructure field on MetricFamily (validated, defaults per_series) and plumbs it to PrecomputeConfig.Scope. Tests: precompute scope_test.go (PerSeries N vs WholeStream 1, Sum grand total, JSON round-trip, GlobalAggregation back-compat, MaxSeries no-op, scope-change window reset) and asapedge whole_stream_test.go (Sum/DDSketch/ HLL per-series-vs-whole-stream fan-out + mode validation). go test ./..., go vet, gofmt all pass for both modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
feat/dual-mode-aggregation
branch
from
May 31, 2026 17:28
eba0913 to
fa4a777
Compare
zzylol
marked this pull request as ready for review
June 1, 2026 13:19
zzylol
added a commit
that referenced
this pull request
Jun 1, 2026
…ssor Complete the processor side of the opt-in sparse HLL feature (engine side landed in the preceding commit). Lives in the contrib-patch overlay (opentelemetry-collector-contrib-patch/processor/asapedgeprocessor), the source of truth committed in this private repo; restore_otel_collector_contrib_patches.sh stages it onto the upstream-pinned submodule for build/test. - config.go: add `HLLSparse bool` (mapstructure "hll_sparse") to MetricFamily. Default false (dense). Opt-in selects the sparse in-memory HLL base for an HLL family so low-cardinality warm series avoid the dense ~16KB/series register array; serialized output is byte-identical to dense (pure in-memory footprint win, no wire change). Only consulted for family=hll. - config_validate.go: reject hll_sparse on any non-HLL family at boot (mirrors the emit_heap family guard) so a misconfiguration surfaces early. - warm_sketch.go: in the FamilyHLL branch, build sketches.NewHLLWrapperSparse() when fam.HLLSparse, else the dense NewHLLWrapper(). The constructor is the source of truth for base selection; the choice is also surfaced as the documented HLL "sparse" SketchParams key (1=sparse, absent=dense) on the emitted PrecomputeConfig for introspection. - hll_sparse_test.go: assert default => dense (SketchParams[sparse] absent), hll_sparse=true => SketchParams[sparse]=1, both factories build a usable *sketches.HLLWrapper, plus config round-trip + the non-HLL family guard. Remove the now-obsolete asapedge-hll-sparse-wiring/README.md placeholder (it captured the pending fragments; the wiring is now committed). Rebased onto merged #471 (dual-mode aggregation) and #474 (HLL delta invariant). The sketchlib-go NewSparseHyperLogLog dependency is merged to main and resolves via the existing local replace; no pin. Build/test: asap-precompute-go `go test ./... && go vet ./...` pass; the asapedgeprocessor module builds, vets, gofmts clean, and `go test ./...` passes (incl. the new sparse tests and #471's whole_stream tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Jun 1, 2026
) * feat(precompute): opt-in sparse in-memory HLL for warm aggregators Add an opt-in sparse base for the HLL precompute wrapper so low-cardinality warm series no longer pay the dense ~16KB/series register-array cost. - sketches/hll.go: add NewHLLWrapperSparse(), backed by hll.NewSparseHyperLogLog() (sketchlib-go #66). A new `sparse` field on HLLWrapper is threaded through the shared newSketch() helper so Reset / Merge / ApplyDelta rebuild a sparse base and a sparse wrapper never reverts to the dense footprint. The wrapper drives the inner sketch only through its public methods and never touches the exported Registers field. Snapshot / Merge / ApplyDelta / Reset / EstimateCardinality are unchanged and remain byte-identical / interoperable with the dense base. - config.go: document the recognized HLL "sparse" SketchParams key. - sketches/hll_sparse_test.go: estimate parity, byte-identical snapshots, dense<->sparse merge interop, Reset, and a low-cardinality heap check. Dependency: sketchlib-go #66 is MERGED to main (fff038b); resolved via the existing `replace => ../../sketchlib-go` directive, so no go.mod change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(asap_edge): wire opt-in sparse in-memory HLL into the edge processor Complete the processor side of the opt-in sparse HLL feature (engine side landed in the preceding commit). Lives in the contrib-patch overlay (opentelemetry-collector-contrib-patch/processor/asapedgeprocessor), the source of truth committed in this private repo; restore_otel_collector_contrib_patches.sh stages it onto the upstream-pinned submodule for build/test. - config.go: add `HLLSparse bool` (mapstructure "hll_sparse") to MetricFamily. Default false (dense). Opt-in selects the sparse in-memory HLL base for an HLL family so low-cardinality warm series avoid the dense ~16KB/series register array; serialized output is byte-identical to dense (pure in-memory footprint win, no wire change). Only consulted for family=hll. - config_validate.go: reject hll_sparse on any non-HLL family at boot (mirrors the emit_heap family guard) so a misconfiguration surfaces early. - warm_sketch.go: in the FamilyHLL branch, build sketches.NewHLLWrapperSparse() when fam.HLLSparse, else the dense NewHLLWrapper(). The constructor is the source of truth for base selection; the choice is also surfaced as the documented HLL "sparse" SketchParams key (1=sparse, absent=dense) on the emitted PrecomputeConfig for introspection. - hll_sparse_test.go: assert default => dense (SketchParams[sparse] absent), hll_sparse=true => SketchParams[sparse]=1, both factories build a usable *sketches.HLLWrapper, plus config round-trip + the non-HLL family guard. Remove the now-obsolete asapedge-hll-sparse-wiring/README.md placeholder (it captured the pending fragments; the wiring is now committed). Rebased onto merged #471 (dual-mode aggregation) and #474 (HLL delta invariant). The sketchlib-go NewSparseHyperLogLog dependency is merged to main and resolves via the existing local replace; no pin. Build/test: asap-precompute-go `go test ./... && go vet ./...` pass; the asapedgeprocessor module builds, vets, gofmts clean, and `go test ./...` passes (incl. the new sparse tests and #471's whole_stream tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds dual-mode aggregation to the ASAP edge precompute runtime: every
sketch/aggregation family now supports both aggregation scopes, selectable
per-aggregation by the control plane.
AggregateBygroup, honoringOmitResourceAttrs); each datapoint folds intoits own series' sketch; one output envelope per series. Today's behavior —
existing plans are byte-compatible.
AggID(per shard,merged at flush) ingests from every matching datapoint regardless of series
identity; one output envelope per AggID per window. Resource attrs + dp labels
are stripped from the series key and the emitted envelope.
Design
AggModeenum inasap-precompute-go/config.go(ModePerSeries=0 /ModeWholeStream) withString()+ParseAggMode. It serializes exactly likethe package's other enums (plain integer, no custom
MarshalJSON), so anomitted field decodes to the default.
PrecomputeConfig.Scope AggMode. Naming note: the field isScope, notMode, becausePrecomputeConfig.Modeis already taken by theexisting
AggregationMode(Tumbling/Sliding/Batch windowing strategy). Scopeis orthogonal to the windowing mode; the enum constants stay
Mode*so usesites read
cfg.Scope == ModeWholeStream.AggID rather than adding a parallel "single sketch slot" — so pooling
(
SketchSink),Reset, snapshot-cache, and delta encoding all work unchangedwith zero new flush/rotate/drain code paths.
Merged / deprecated existing functionality
GlobalAggregation boolis subsumed byScope. A neweffectiveScope()helper folds
GlobalAggregation=trueintoModeWholeStream, so the singleWholeStream code path drives both. The bool is kept as a documented deprecated
alias; the legacy
emit_heapCountSketch path (which setGlobalAggregation)keeps working byte-for-byte. All four SeriesKey helpers + the window
admit/strip path now branch on
isWholeStream()instead of the raw bool, sothe two signals can never disagree. A back-compat test asserts a
GlobalAggregationconfig behaves identically to aScope=WholeStreamone.OmitResourceAttrsand the empty-AggregateBypath compose cleanly:WholeStream is the strict super-collapse (strips both resource + dp labels);
the other flags only matter in PerSeries.
mechanisms. The only one in these repos is
GlobalAggregation(subsumed). Thelegacy standalone processors were already retired (Sum became first-class in
Sum as first-class AggregationType (edge + vendored pdata) + google_cluster E2E harness #468); there is now ONE scope abstraction.
Per-family WholeStream semantics
Mode only changes routing; no sketchlib-go math changes. The ingested
subject is the metric value by default, or the inner item dimension when
ItemLabelis set (reconciled with the existingItemLabel— no redundantknob):
UpdateConfig
A live scope flip cannot hot-swap in place (the series map is keyed
incompatibly across scopes), so
UpdateConfigresets the in-flight window on ascope transition; same-scope changes stay in place (preserving accumulated
window bytes, per the documented contract).
asap_edge wiring
MetricFamilygains amodemapstructure field (per_seriesdefault /whole_stream), validated at config boot and plumbed toPrecomputeConfig.Scope.MaxSeriesis a no-op under WholeStream.Controller follow-up (not in these repos)
The scope is "decided by the control plane given the queries." The wire field
(
PrecomputeConfig.Scope) and the operator-facing knob (MetricFamily.Mode)now exist and are documented. The query→PrecomputeConfig mapping lives in the
controller's deployment-model layer (
/mydata/ASAPControlleremits L3AggIntent), which is NOT present in these repos, so no controller code was
fabricated. The controller must set
Scope/modeper aggregation:distinct-count / global-quantile / grand-total / global-top-k queries →
whole_stream; per-label-group queries →per_series.Tests
asap-precompute-go/scope_test.go: AggMode String/parse, JSON round-trip(with + without the field),
effectiveScopefolding, PerSeries→N vsWholeStream→1 with pooled semantics, GlobalAggregation back-compat parity,
MaxSeries no-op, scope-change window reset (and same-scope preservation).
asapedgeprocessor/whole_stream_test.go: Sum grand-total, DDSketch pooledquantile, HLL distinct (item dimension projected + grouping collapsed),
per-series-vs-whole-stream fan-out, and
modevalidation.go test ./...,go vet ./...,gofmt -lall pass for both theasap-precompute-gomodule and theasapedgeprocessormodule.🤖 Generated with Claude Code