Sum AggregationType ingest + CMS per-item estimate(key) - #355
Merged
Merged
Conversation
sketch_reducer: decode_frequency_estimate (calls CountMinSketch/CountSketch::estimate(key); heap path rebuilds via from_legacy_matrix) + thread item_key through evaluate_for_capability/evaluate_core; FrequencyEstimate branch dispatches estimate(key) vs the per-window bucket total. engine + index: sid->item_label side-table (set_item_label/item_label_for); engine extract_filter_value resolves the item value from a hit sid's registered item_label and lifts the keyed-frequency safe-miss only for item_label-mode sids. Tests: cms_per_item_estimate, extract_filter_value. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
asap_otel_proto: add SumAgg/SumAggDataPoint/SumAggEncoding + Data::SumAgg (sum_agg=18, matching the collector pdata oneof tag). SumAccumulator::from_sum_bytes decodes the self-contained 16-byte Sum payload (float64 sum LE || uint64 count LE) the edge emits — Sum is an aggregation, NOT a sketch, so the payload deliberately does not use the sketchlib envelope proto. otel.rs ingest routes Data::SumAgg through SumAccumulator into the existing ExactAgg(Sum) MetricPoint path (no new SketchKindHandle; sid stays exact_agg:Sum). Cross-language golden test (Go SumWrapper bytes -> sum=100). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
|
Companion PR: ProjectASAP/ASAPCollector#468 (edge Sum + vendored pdata SumAgg + google_cluster harness). Coordinated cross-repo change joined by the wire contract (16-byte Sum payload + SumAgg metric field tag 18) — land together. |
control_plane: thread the per-metric item_label into the backend streaming-config. BackendAggregation gains an item_label field (patched from collect_metric_to_item_label in both plan-handler loops), emitted into the policy parameters["item_label"] by build_backend_aggregation_json. data_plane: at sketch ingest registration, read the matched policy's parameters["item_label"] and record it on the sid via set_item_label — so the FrequencyEstimate safe-miss gate resolves the item key and serves per-item estimate(key). Previously the gate was inert (set_item_label was never called). control_plane 794 lib tests pass; data_plane Sum/CMS tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`SketchStore::approx_memory_bytes` is the flusher's evictable-payload gauge (current_epoch + sealed payloads). At idle it correctly reads ~0 because payloads are flushed to disk — but the per-sid registry (`instances` metadata + per-series `InternTable` label caches) stays resident, so the MEMORY_DIAG line read "0.00 KB" while process RSS sat in the hundreds of MB, and there was no relief valve for that residue (stale sketch sids are pinned until config-driven retirement). Two cohesive changes, both scoped to the SketchStore: 1. Honest memory diagnostic. Adds `InternTable::approx_heap_bytes` and `SketchStore::approx_resident_bytes` (registry metadata + intern caches + live payload) and `process_resident_bytes()` (/proc/self/statm). The 30s MEMORY_DIAG line now logs payload (evictable) / registry+intern (resident, not flushable) / process RSS, so the real footprint is visible. The flusher's payload-pressure trigger is intentionally left on the payload gauge — feeding non-flushable registry memory into it would livelock the flush loop (it can only evict payload). 2. Idle-sid eviction (opt-in --idle-sid-evict-secs / ASAP_IDLE_SID_EVICT_SECS, default 0 = off). A periodic sweep drops the in-memory `SidStoreData` (epoch columns + intern cache + series slot) for sketch sids that are persistence-backed, write-idle past the threshold, AND fully durable on disk (sealed_epochs + current_epoch empty), while KEEPING the queryable `SketchInstanceMetadata`. `union_disk_parts_into` reads the durable tier via the retained metadata only (never `self.series`), so an evicted series stays answerable from disk; the append path's `entry().or_insert_with` rehydrates a fresh store on the next write. Bounds resident registry memory under series churn. Effective idle horizon is max(threshold, persistence_hot_window) since eviction waits for a sid's windows to seal+flush first. Adds `SidStoreData::last_write_unix_ms`, stamped under the append write lock the hot path already holds (no extra cost). 5 unit tests cover resident accounting and the eviction predicate / rehydration / durability guard. Validated live on the Google-cluster trace: 31.5k stale sids reclaimed, in-memory sid state held to a few thousand vs pinning all 36k; MEMORY_DIAG now shows payload≈KB / registry≈MB / RSS≈MB instead of a misleading 0.00 KB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
data-plane: honest SketchStore memory diagnostic + idle-sid eviction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related backend features completing the asapedge warm-aggregation set (the edge producer side is in the ASAPCollector PR).
Sum AggregationType envelope ingest
asap_otel_proto: newSumAgg/SumAggDataPoint/SumAggEncoding+Data::SumAgg(sum_agg = 18), matching the collector pdata oneof tag.SumAccumulator::from_sum_bytesdecodes the self-contained 16-byte Sum payload (float64 sumLE ‖uint64 countLE) the edge emits — Sum is an aggregation, not a sketch, so the payload deliberately does not use the sketchlib envelope proto.otel.rsingest routesData::SumAggthroughSumAccumulatorinto the existingExactAgg(Sum)MetricPoint path (no newSketchKindHandle; sid staysexact_agg:Sum).SumWrapperbytes →sum=100).CMS per-item
estimate(key)sketch_reducer:decode_frequency_estimatecallsCountMinSketch/CountSketch::estimate(key);item_keythreaded throughevaluate_for_capability/evaluate_core; the FrequencyEstimate branch dispatches per-item estimate vs the per-window bucket total.item_labelside-table; engineextract_filter_valueresolves the item value from a hit sid's registereditem_labeland lifts the keyed-frequency safe-miss only for item_label-mode sids.cargo build -p data_planeclean; sum + CMS reducer tests pass.🤖 Generated with Claude Code