Sum as first-class AggregationType (edge + vendored pdata) + google_cluster E2E harness - #468
Merged
Merged
Conversation
…oogle_cluster E2E harness + agent-accuracy
asap-precompute-go: AggregationKind {Sketch,Sum} umbrella + AggKind field on SketchEnvelope/PrecomputeConfig (EffectiveAggKind dual-read, stamped in serializeSeries). sketches/sum.go: SumWrapper + SumObserver (additive Merge). Sum is an aggregation, NOT a sketch — its payload is a self-contained 16-byte {float64 sum, uint64 count} little-endian, deliberately NOT the sketchlib sketch-envelope proto, so it touches neither the sketchlib-go nor asap_sketchlib repo. otel adapter encodes/decodes the modified-OTLP SumAgg metric. Tests: SumWrapper round-trip, SumAgg otel round-trip, real-data sketch accuracy.
opentelemetry-collector-patch: vendor the modified-OTLP SumAgg typed metric (pdatagen model + generated SumAgg files + SumAggEncoding + MetricTypeSumAgg + DataPointCount), alongside the existing sketch types, captured via backup_otel_collector_patches.sh. The submodule itself is not committed (restore_otel_collector_patches.sh re-applies the tree).
asapedgeprocessor: route FamilySum through the precompute path (SumWrapper -> SumAgg, unsuffixed name); retire warm_sum.go + the per-shard sum machinery; relocate shared capMetrics; add FamilySum to all-families test.
datasets_eval/google_cluster: backend-query E2E harness + queries.json v2 (+ CMS frequency) + run.py schema.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
|
Companion PR: ProjectASAP/ASAPQuery-backend#355 (backend SumAgg ingest + CMS estimate). Coordinated cross-repo change joined by the wire contract (16-byte Sum payload + SumAgg metric field tag 18) — land together. |
… OTLP frame The fused asap_edge warm path emits DDSketch envelopes via oteladapter.Encode, but the encoder never set the output pmetric.DDSketch container's relative_accuracy — unlike the standalone ddsketchprocessor (shim_helpers SetRelativeAccuracy). So every fused-edge DDSketch frame shipped relative_accuracy=0.0, and the backend registered the sid as a degenerate ε=0 sketch. A ε=0 DDSketch can't produce bucketed quantiles, so `quantile_over_time(...)` capability-misses to the archive and returns empty (observed live: all google_cluster DDSketch sids registered with relative_accuracy 0.0 while the controller policy correctly declared alpha=0.02). Thread the alpha from the sketch instance through to the wire: - DDSketchWrapper.RelativeAccuracy() exposes the configured alpha. - SketchEnvelope gains an in-process RelativeAccuracy field, populated in serializeSeries from the sketch (0 for non-DDSketch families). - otel/encode.go sets dst.SetRelativeAccuracy(env.RelativeAccuracy) on the DDSketch container when > 0. Verified by relative_accuracy_integration_test.go through the real DDSketchWrapper: envelope, in-memory pmetric, AND a full OTLP proto marshal->unmarshal round-trip all preserve alpha=0.02, on both the full-snapshot and delta-transmission paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cy-on-wire asap-precompute-go: emit DDSketch relative_accuracy on the wire (fixes empty warm quantile queries)
This was referenced May 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sum as a first-class AggregationType on the edge, the vendored modified-OTLP
SumAggpdata type, plus the google_cluster E2E harness and a real-data agent-accuracy test. (Backend ingest side is in ASAPQuery-backend.)Sum (asap-precompute-go + asapedgeprocessor)
AggregationKind {Sketch,Sum}umbrella +AggKindfield onSketchEnvelope/PrecomputeConfig(EffectiveAggKinddual-read, stamped inserializeSeries).sketches/sum.go:SumWrapper+SumObserver(additiveMerge). Sum is an aggregation, not a sketch — its payload is a self-contained 16-byte{float64 sum, uint64 count}(LE), deliberately not the sketchlib sketch-envelope proto, so it touches neither the sketchlib-go nor asap_sketchlib repo.SumAggmetric.asapedgeprocessor: routeFamilySumthrough the precompute path (SumWrapper→SumAgg, unsuffixed name); retirewarm_sum.go+ the per-shard sum machinery; relocate the sharedcapMetricstest helper.Vendored pdata
SumAgg(opentelemetry-collector-patch/)SumAggtyped metric (pdatagen model + generatedSumAggfiles +SumAggEncoding+MetricTypeSumAgg+DataPointCount), captured viabackup_otel_collector_patches.shalongside the existing sketch types. The submodule itself is not committed —restore_otel_collector_patches.shre-applies the tree.google_cluster E2E harness + agent-accuracy
datasets_eval/google_cluster/e2e/: backend-query E2E harness (gt_eval/compare/query_client/run_e2e+ workload yaml).queries.jsonv2 (+ CMS frequency query, structured ground-truth specs);run.pyschema.All touched Go modules build; edge + Sum + otel adapter tests pass.
🤖 Generated with Claude Code