Skip to content

feat(precompute): dual-mode aggregation (per-series vs whole-stream) unified under Mode - #471

Merged
zzylol merged 1 commit into
mainfrom
feat/dual-mode-aggregation
Jun 1, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/dual-mode-aggregation

Conversation

@zzylol

@zzylol zzylol commented May 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds dual-mode aggregation to the ASAP edge precompute runtime: every
sketch/aggregation family now supports both aggregation scopes, selectable
per-aggregation by the control plane.

  • PerSeries (default / zero value): one sketch per series key (per
    AggregateBy group, honoring OmitResourceAttrs); each datapoint folds into
    its own series' sketch; one output envelope per series. Today's behavior —
    existing plans are byte-compatible.
  • WholeStream: collapse grouping — a single sketch per AggID (per shard,
    merged at flush) ingests from every matching datapoint regardless of series
    identity; one output envelope per AggID per window. Resource attrs + dp labels
    are stripped from the series key and the emitted envelope.

Design

  • New AggMode enum in asap-precompute-go/config.go (ModePerSeries=0 /
    ModeWholeStream) with String() + ParseAggMode. It serializes exactly like
    the package's other enums (plain integer, no custom MarshalJSON), so an
    omitted field decodes to the default.
  • New field PrecomputeConfig.Scope AggMode. Naming note: the field is
    Scope, not Mode, because PrecomputeConfig.Mode is already taken by the
    existing AggregationMode (Tumbling/Sliding/Batch windowing strategy). Scope
    is orthogonal to the windowing mode; the enum constants stay Mode* so use
    sites read cfg.Scope == ModeWholeStream.
  • WholeStream reuses the single series-map mechanism with a constant key per
    AggID rather than adding a parallel "single sketch slot" — so pooling
    (SketchSink), Reset, snapshot-cache, and delta encoding all work unchanged
    with zero new flush/rotate/drain code paths.

Merged / deprecated existing functionality

  • GlobalAggregation bool is subsumed by Scope. A new effectiveScope()
    helper folds GlobalAggregation=true into ModeWholeStream, so the single
    WholeStream code path drives both. The bool is kept as a documented deprecated
    alias; the legacy emit_heap CountSketch path (which set GlobalAggregation)
    keeps working byte-for-byte. All four SeriesKey helpers + the window
    admit/strip path now branch on isWholeStream() instead of the raw bool, so
    the two signals can never disagree. A back-compat test asserts a
    GlobalAggregation config behaves identically to a Scope=WholeStream one.
  • OmitResourceAttrs and the empty-AggregateBy path compose cleanly:
    WholeStream is the strict super-collapse (strips both resource + dp labels);
    the other flags only matter in PerSeries.
  • Searched the whole tree for parallel "global/whole/scope/buildPartitionKey"
    mechanisms. The only one in these repos is GlobalAggregation (subsumed). The
    legacy standalone processors were already retired (Sum became first-class in
    Sum as first-class AggregationType (edge + vendored pdata) + google_cluster E2E harness #468); there is now ONE scope abstraction.

Per-family WholeStream semantics

Mode only changes routing; no sketchlib-go math changes. The ingested
subject is the metric value by default, or the inner item dimension when
ItemLabel is set (reconciled with the existing ItemLabel — no redundant
knob):

Family value subject (default) item subject (ItemLabel set)
Sum grand total n/a
DDSketch / KLL pooled value distribution n/a
HLL distinct values distinct items
CountMinSketch value frequency per-item frequency
CountSketch (+heap) value frequency / heavy values heavy items (top-k)

UpdateConfig

A live scope flip cannot hot-swap in place (the series map is keyed
incompatibly across scopes), so UpdateConfig resets the in-flight window on a
scope transition; same-scope changes stay in place (preserving accumulated
window bytes, per the documented contract).

asap_edge wiring

MetricFamily gains a mode mapstructure field (per_series default /
whole_stream), validated at config boot and plumbed to
PrecomputeConfig.Scope. MaxSeries is a no-op under WholeStream.

Controller follow-up (not in these repos)

The scope is "decided by the control plane given the queries." The wire field
(PrecomputeConfig.Scope) and the operator-facing knob (MetricFamily.Mode)
now exist and are documented. The query→PrecomputeConfig mapping lives in the
controller's deployment-model layer (/mydata/ASAPController emits L3
AggIntent), which is NOT present in these repos, so no controller code was
fabricated. The controller must set Scope/mode per aggregation:
distinct-count / global-quantile / grand-total / global-top-k queries →
whole_stream; per-label-group queries → per_series.

Tests

  • asap-precompute-go/scope_test.go: AggMode String/parse, JSON round-trip
    (with + without the field), effectiveScope folding, PerSeries→N vs
    WholeStream→1 with pooled semantics, GlobalAggregation back-compat parity,
    MaxSeries no-op, scope-change window reset (and same-scope preservation).
  • asapedgeprocessor/whole_stream_test.go: Sum grand-total, DDSketch pooled
    quantile, HLL distinct (item dimension projected + grouping collapsed),
    per-series-vs-whole-stream fan-out, and mode validation.
  • go test ./..., go vet ./..., gofmt -l all pass for both the
    asap-precompute-go module and the asapedgeprocessor module.

Build note: the asapedgeprocessor source of truth is
opentelemetry-collector-contrib-patch/processor/asapedgeprocessor/ (committed
in this private repo); restore_otel_collector_contrib_patches.sh stages it
onto the upstream-pinned contrib submodule for the build/test.

🤖 Generated with Claude Code

@zzylol
zzylol force-pushed the feat/dual-mode-aggregation branch 4 times, most recently from 4bbb1ca to eba0913 Compare May 31, 2026 17:25
…unified under Mode

Add an explicit per-aggregation scope (AggMode: ModePerSeries /
ModeWholeStream) to PrecomputeConfig.Scope, selectable by the control
plane. PerSeries (zero value) is today's behavior; WholeStream collapses
every matching datapoint into a single sketch per AggID and emits one
envelope per AggID per window.

The legacy GlobalAggregation bool is subsumed: effectiveScope() folds it
into ModeWholeStream so one whole-stream code path drives both (the
emit_heap CountSketch keeps working). UpdateConfig resets the in-flight
window on a scope flip (the series map is keyed incompatibly across
scopes); same-scope changes stay in place. MaxSeries is a no-op in
WholeStream (cardinality is 1). Per-family the ingested subject is the
metric value by default, or the inner item dimension when asap_edge
ItemLabel is set (reconciled with the existing ItemLabel, no new knob).

asap_edge surfaces a `mode` mapstructure field on MetricFamily (validated,
defaults per_series) and plumbs it to PrecomputeConfig.Scope.

Tests: precompute scope_test.go (PerSeries N vs WholeStream 1, Sum grand
total, JSON round-trip, GlobalAggregation back-compat, MaxSeries no-op,
scope-change window reset) and asapedge whole_stream_test.go (Sum/DDSketch/
HLL per-series-vs-whole-stream fan-out + mode validation). go test ./...,
go vet, gofmt all pass for both modules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the feat/dual-mode-aggregation branch from eba0913 to fa4a777 Compare May 31, 2026 17:28
@zzylol
zzylol marked this pull request as ready for review June 1, 2026 13:19
@zzylol
zzylol merged commit 68a9609 into main Jun 1, 2026
@zzylol
zzylol deleted the feat/dual-mode-aggregation branch June 1, 2026 13:21
zzylol added a commit that referenced this pull request Jun 1, 2026
…ssor

Complete the processor side of the opt-in sparse HLL feature (engine side
landed in the preceding commit). Lives in the contrib-patch overlay
(opentelemetry-collector-contrib-patch/processor/asapedgeprocessor), the
source of truth committed in this private repo; restore_otel_collector_contrib_patches.sh
stages it onto the upstream-pinned submodule for build/test.

- config.go: add `HLLSparse bool` (mapstructure "hll_sparse") to MetricFamily.
  Default false (dense). Opt-in selects the sparse in-memory HLL base for an
  HLL family so low-cardinality warm series avoid the dense ~16KB/series
  register array; serialized output is byte-identical to dense (pure in-memory
  footprint win, no wire change). Only consulted for family=hll.
- config_validate.go: reject hll_sparse on any non-HLL family at boot
  (mirrors the emit_heap family guard) so a misconfiguration surfaces early.
- warm_sketch.go: in the FamilyHLL branch, build sketches.NewHLLWrapperSparse()
  when fam.HLLSparse, else the dense NewHLLWrapper(). The constructor is the
  source of truth for base selection; the choice is also surfaced as the
  documented HLL "sparse" SketchParams key (1=sparse, absent=dense) on the
  emitted PrecomputeConfig for introspection.
- hll_sparse_test.go: assert default => dense (SketchParams[sparse] absent),
  hll_sparse=true => SketchParams[sparse]=1, both factories build a usable
  *sketches.HLLWrapper, plus config round-trip + the non-HLL family guard.

Remove the now-obsolete asapedge-hll-sparse-wiring/README.md placeholder
(it captured the pending fragments; the wiring is now committed).

Rebased onto merged #471 (dual-mode aggregation) and #474 (HLL delta
invariant). The sketchlib-go NewSparseHyperLogLog dependency is merged to
main and resolves via the existing local replace; no pin.

Build/test: asap-precompute-go `go test ./... && go vet ./...` pass; the
asapedgeprocessor module builds, vets, gofmts clean, and `go test ./...`
passes (incl. the new sparse tests and #471's whole_stream tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Jun 1, 2026
)

* feat(precompute): opt-in sparse in-memory HLL for warm aggregators

Add an opt-in sparse base for the HLL precompute wrapper so low-cardinality
warm series no longer pay the dense ~16KB/series register-array cost.

- sketches/hll.go: add NewHLLWrapperSparse(), backed by
  hll.NewSparseHyperLogLog() (sketchlib-go #66). A new `sparse` field on
  HLLWrapper is threaded through the shared newSketch() helper so Reset / Merge
  / ApplyDelta rebuild a sparse base and a sparse wrapper never reverts to the
  dense footprint. The wrapper drives the inner sketch only through its public
  methods and never touches the exported Registers field. Snapshot / Merge /
  ApplyDelta / Reset / EstimateCardinality are unchanged and remain
  byte-identical / interoperable with the dense base.
- config.go: document the recognized HLL "sparse" SketchParams key.
- sketches/hll_sparse_test.go: estimate parity, byte-identical snapshots,
  dense<->sparse merge interop, Reset, and a low-cardinality heap check.

Dependency: sketchlib-go #66 is MERGED to main (fff038b); resolved via the
existing `replace => ../../sketchlib-go` directive, so no go.mod change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(asap_edge): wire opt-in sparse in-memory HLL into the edge processor

Complete the processor side of the opt-in sparse HLL feature (engine side
landed in the preceding commit). Lives in the contrib-patch overlay
(opentelemetry-collector-contrib-patch/processor/asapedgeprocessor), the
source of truth committed in this private repo; restore_otel_collector_contrib_patches.sh
stages it onto the upstream-pinned submodule for build/test.

- config.go: add `HLLSparse bool` (mapstructure "hll_sparse") to MetricFamily.
  Default false (dense). Opt-in selects the sparse in-memory HLL base for an
  HLL family so low-cardinality warm series avoid the dense ~16KB/series
  register array; serialized output is byte-identical to dense (pure in-memory
  footprint win, no wire change). Only consulted for family=hll.
- config_validate.go: reject hll_sparse on any non-HLL family at boot
  (mirrors the emit_heap family guard) so a misconfiguration surfaces early.
- warm_sketch.go: in the FamilyHLL branch, build sketches.NewHLLWrapperSparse()
  when fam.HLLSparse, else the dense NewHLLWrapper(). The constructor is the
  source of truth for base selection; the choice is also surfaced as the
  documented HLL "sparse" SketchParams key (1=sparse, absent=dense) on the
  emitted PrecomputeConfig for introspection.
- hll_sparse_test.go: assert default => dense (SketchParams[sparse] absent),
  hll_sparse=true => SketchParams[sparse]=1, both factories build a usable
  *sketches.HLLWrapper, plus config round-trip + the non-HLL family guard.

Remove the now-obsolete asapedge-hll-sparse-wiring/README.md placeholder
(it captured the pending fragments; the wiring is now committed).

Rebased onto merged #471 (dual-mode aggregation) and #474 (HLL delta
invariant). The sketchlib-go NewSparseHyperLogLog dependency is merged to
main and resolves via the existing local replace; no pin.

Build/test: asap-precompute-go `go test ./... && go vet ./...` pass; the
asapedgeprocessor module builds, vets, gofmts clean, and `go test ./...`
passes (incl. the new sparse tests and #471's whole_stream tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant