fix(series): send full attrs until collector confirms series ID (multi-hop fix) - #59
Merged
zzylol merged 1 commit intoMar 19, 2026
Conversation
…multi-hop tests The series ID optimisation was premature: assignSeriesID set the local ID and marked entry.registered=true before any collector response, causing seriesIdentity to drop dimension attributes even on the first export. In a multi-hop deployment (SDK → Agent → Gateway) every intermediate hop therefore received data points with attrs=nil and an opaque numeric ID it had no mapping for, silently losing all dimension labels in the backend. Fix: - assignSeriesID now only switches to ID-only mode (sets *seriesID, clears *attrs) when entry.registered is already true, i.e. after Apply() has been called with a collector-confirmed SeriesAssignment. Until that point, seriesID stays 0 so the transform serialises full attributes onto the wire. - All five sketch annotate functions (DDSketch, KLL, CountSketch, CountMinSketch, HLL) now guard AttrsClearer behind dp.SeriesID != 0, preventing the aggregator's attrs field from being zeroed out before confirmation — which would have caused attrs to disappear on the next export cycle. Add two tests in the series package: - TestMultiHop_AttrsUntilCollectorConfirms: drives the full three-phase lifecycle (unconfirmed → Apply → confirmed) and asserts that SeriesID==0 + full attrs are forwarded on every pre-confirmation export, and that the aggregator state (SeriesIDSink / AttrsClearer) transitions correctly only after confirmation. - TestMultiHop_ResourceAttrsAlwaysSent: regression guard that Annotate never touches ResourceMetrics.Resource attributes (service.instance.id, host.name, k8s.pod.name), which are the labels used for agent counting and Prometheus scrape discovery. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zzylol
deleted the
45-distributed-sdks---1-collector-distributed-collectors
branch
March 19, 2026 21:59
SieDeta
pushed a commit
that referenced
this pull request
Apr 17, 2026
…ctor-distributed-collectors fix(series): send full attrs until collector confirms series ID (multi-hop fix)
zzylol
added a commit
that referenced
this pull request
Apr 21, 2026
Backend's /metrics endpoint is now populated (ASAPQuery-backend PR #59), so the scrape job can come back. Point at `backend:9091` (query server port, --query-port) instead of the stale `:9465` that was never bound — ingest port 9090 is data-only and does not expose /metrics. Live-verified: * `asap_ingest_samples_total` at 18.5M through Prometheus * Backend target UP alongside controller + gateway * Query histograms visible for P99 calculation (§6.3) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
assignSeriesIDwas setting the local series ID and markingentry.registered=truebefore any collector response. This causedseriesIdentityto drop dimension attributes even on the first export — including in Agent → Gateway deployments where the gateway never received any attribute mapping.assignSeriesIDnow only switches to ID-only mode afterApply()setsentry.registered=true(collector-confirmed). All five sketchannotate*functions guardAttrsClearerbehinddp.SeriesID != 0, preventing the aggregator's attrs from being cleared prematurely.docs/otel-distributed-setup.mdsummarising OTel distributed collector architecture with link references.What breaks without this fix
In a multi-hop deployment (
SDK → Agent → Gateway):{series_id: N, attrs: nil}— even the first one{series_id: N, attrs: nil}to the GatewayBehaviour after fix
series_id=1, attrs=nil❌series_id=0, attrs=real✅series_id=1, attrs=nil❌series_id=0, attrs=real✅Apply()confirmsseries_id=1, attrs=nilseries_id=42, attrs=nil✅Test plan
TestMultiHop_AttrsUntilCollectorConfirms— drives full three-phase lifecycle (unconfirmed → Apply → confirmed), asserts SeriesID==0 + full attrs on pre-confirmation exports, correct aggregator state transitions after confirmationTestMultiHop_ResourceAttrsAlwaysSent— regression guard thatAnnotatenever touchesResourceMetrics.Resourceattributes (service.instance.id, host.name, k8s.pod.name)cd opentelemetry-go/exporters/otlp/otlpmetric/otlpmetricgrpc && go test ./internal/series/... -run TestMultiHop🤖 Generated with Claude Code