asapedge sketch stack: CountSketch heap/topk + wire-tag fix + mvp-multinode config - #463
Merged
Merged
Conversation
…rden factory; heap + delta-heap - configDimensions clamps rows so rows*log2(cols)<=64 (was: SketchFactory dropped NewCountSketchWrapper's error → nil wrapper → SIGSEGV on first observe). - metric_name config + resolveOutputMetricName so the sketch registers under the queried name (e.g. top_endpoint_qps) instead of the fixed "countsketch_partition". - item_label keying; emit_heap/heap_size → heap-bearing CountSketch (FrequencyTopk) with a msgpack delta-heap frame (full first window, delta after). - stampDPMetadata: group-aware envelope pairing (was a fragile flat idx++). - tests: clamp/naming/heap round-trip; proto-refactor compile repairs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t CS/CMS encoding tag
- CountSketch heap-msgpack wrapper: ComputeDeltaAgainst/DeltaAgainstEmptyBase emit
the per-window delta-heap msgpack frame (PWR, delta-against-empty); ApplyDelta
handles full + delta-heap; EncodingMsgpackDelta host-neutral encoding.
- otel/encode.go: hostNeutralTo{CountSketch,CountMin}Encoding now switch on the
encoding (Proto/Delta/Msgpack/MsgpackDelta) like DDSketch, instead of hardcoding
Proto — fixes proto-delta + heap-msgpack CS/CMS being mistagged on the wire.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
emit_heap/heap_size/item_label on a CountSketch MetricFamily build a NewCountSketchWithHeapWrapper keyed by the item label, with GlobalAggregation + OmitResourceAttrs so the top-k heap ranks items together, emitting MSGPACK / MSGPACK_DELTA frames the backend promotes to FrequencyTopk. Non-heap families unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…h/kll/countmin) The 2026-05 proto refactor removed precomputed scalar getters (dp.Count/Epsilon/ Delta/Dimension/Cardinality/...) from the typed sketch DataPoints. Repaired the test suites to read params off the parent container / reconstruct from the payload, and to expect preserved input metric names. ddsketch 27/0, kll 22/0, hll green. NOTE: 3 countminsketch delta/window tests (TestCMSDelta_RoundTrip, MultipleWindowsConvergence, TumblingWindow_Correctness) are left RED — a test HARNESS flaw (assertCMSCellsEqual compares two independently-constructed CMS instances cell-by-cell, but CMS hashes per-instance; the layout lives in the serialized bytes). The CMS delta wire format itself is verified correct by the ASAPQuery-backend cross-language golden tests. Seed-invariant rewrite tracked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…andalone-5-sketch) Single asap_edge processor: 6 metric families (sum/kll/hll/countsketch/countmin) with per-family tier + delta + the CountSketch warm-topk heap keys, one shared cold tier via gorilla-merger. run_demo.sh asap arm points here. latency_ms uses KLL per the workload's canonical sketch_family_override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
feat/asapedge-sketch-stack
branch
from
May 27, 2026 14:23
9ee1493 to
491acf1
Compare
…n + CMS query surface
Runtime validation of the asapedge mvp-multinode stack root-caused and fixed:
- KLL k=0 (asap-precompute-go/sketches/kll.go): KLLWrapper.Snapshot() used
SerializePortable's value-offset fixed-point encoding (empty items[]) for
exactly-representable samples, which the backend KLL decoder cannot read
("KllState.k must be >= 8 (got 0)") — so latency quantiles read 0. Now
uses SerializePortableRawF64() (raw items[] the backend decodes) for all
values. Empty windows still emit nothing.
- HLL/CMS item_label (asapedgeprocessor/warm_sketch.go + asap-precompute-go
hll.go): new obsKindItemHLL / obsKindItemCMS projects the configured
item_label (e.g. user_id / endpoint) OUT of the series key and feeds its
VALUE to the sketch as the hash subject — one sketch per group counting
distinct values, instead of one cardinality-1 sketch per value.
- mvp-workload.yaml: item_label keys for the HLL/CMS/CountSketch metrics;
CMS query changed from rate(...) to count_over_time(endpoint_request_freq
[30s]) — the CMS FrequencyEstimate query surface (rate() is a counter op
the analyzer routes to ExactAgg(Sum)→archive).
Pairs with ASAPQuery-backend controller item_label/aggregate_by emit + HLL
sparse decode. Validated: all 6 query families resolve warm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
Runtime validation — all 6 query families now resolve warmValidated the asapedge mvp-multinode stack end-to-end on localhost. This commit (+ ASAPQuery-backend #352) takes it from 2/6 → 6/6 families resolving against the warm asap_query tier (SUM, KLL latency, KLL request_size, HLL, CMS, TopK). This PR's changes:
🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
asapedge sketch stack: CountSketch heap/topk + wire-tag fix + asapedge mvp-multinode config
Part of the asapedge
mvp-multinodeinitiative. Five reviewable commits:metric_namepreservation (sketch registers under the queried name, notcountsketch_partition),item_labelkeying,emit_heap/heap_sizeheap-bearing CountSketch + msgpack delta-heap, and group-awarestampDPMetadata.CS/CMSnow tagged Proto/Delta/Msgpack/MsgpackDelta per the actual encoding, instead of hardcoded Proto — fixes mistagged proto-delta + heap frames).asapedgeprocessoremits the heap-bearing CountSketch (MSGPACK/MSGPACK_DELTA) foremit_heapfamilies, withGlobalAggregation+OmitResourceAttrsso the top-k heap ranks items together.assertCMSCellsEqualcompares two independently-built CMS instances cell-by-cell; CMS hashes per-instance, layout lives in the bytes). The CMS delta wire format is verified correct by the backend cross-language goldens (tune: fake-exporter cardinality down (5K aggregate target, was 25K+) #352). Seed-invariant rewrite tracked.asap_edgeagent config replacing the stale standalone-5-sketch path (latency_ms→KLL per the workload override).Cross-repo
Status
DRAFT — code-review first. Go packages build/test green (except the 3 flagged CMS harness tests). Not yet validated end-to-end on the asapedge
mvp-multinodestack (build asap-otel/data-plane/control-plane → run → verify warm DDSketch?/KLL/HLL/sum + warm-topk via the heap + controller-applied config + cold coverage). I'll push validation fixes here.🤖 Generated with Claude Code