Eric.firth/backport pr 12350 1.65.x - #12415
Draft
ericfirth wants to merge 32 commits into
Draft
Conversation
Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
(cherry picked from commit c55cb16) Co-authored-by: gh-worker-campaigns-3e9aa4[bot] <244854796+gh-worker-campaigns-3e9aa4[bot]@users.noreply.github.com>
Handle gracefully malformed flag evaluations
Implements the pre-instrumentation config accessor for the data streams feature flag, mirroring the existing usmEnabled pattern. Reads the same data.streams.enabled key that Config.isDataStreamsEnabled() uses, enabling DSM to work independently of the general DD_INTEGRATIONS_ENABLED APM gate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ledOverridesFalse Save and restore the original dataStreamsEnabled value instead of hardcoding false in the finally block. This prevents test pollution where an incorrect state could leak to the next test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add DATA_STREAMS sampling mechanism (byte value 14) to support decoupling Kafka DSM from APM tracing. This mechanism enforces USER_DROP priority for DSM-only spans and allows overriding locked priority when DSM is enabled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Use qualified PrioritySampling.USER_DROP form in validateWithSamplingPriority to match the DATA_JOBS case style (line 70) which uses qualified form. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…reams base class Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When Kafka tracing is disabled but Data Streams Monitoring is enabled, produce-path spans created without a propagated trace context should be marked with PrioritySampling.USER_DROP to avoid billing as APM spans. This implements the 'local-root spans only' scoping from the spec: spans with an extracted parent context (genuine distributed traces) are left completely untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eams base class Change extends InstrumenterModule.Tracing to extends InstrumenterModule.DataStreams for all 8 instrumentation classes in the kafka-clients-3.8 module. This decouples Data Streams Monitoring from APM tracing and prepares for billing-suppression logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…me paths) Add TRACING_ENABLED constant and billing-suppression logic for produce and consume paths to suppress unnecessary billing when APM tracing is disabled but DSM is enabled. The suppression only applies to local-root spans (no extracted parent context) per the DSM-Kafka decoupling plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion to DataStreams This class populates StreamingContextUpdater's source/disabled-topic registry via BuildAdvice.exit, which KafkaStreamTaskInstrumentation's DSM checkpoint logic reads. Switching from InstrumenterModule.Tracing to DataStreams ensures the registry is populated when DSM is the only reason Kafka is instrumented, allowing DSM's STREAMING_CONTEXT.isSourceTopic()/isDisabledForTopic() checks to work correctly instead of silently falling back to defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Change ConnectWorkerInstrumentation to extend InstrumenterModule.DataStreams instead of InstrumenterModule.Tracing. This class only wraps TaskStatus.Listener for forwarding task status callbacks and does not create spans, making it appropriate for the DataStreams module instead of the Tracing module. Task 11 of DSMS-161 (kafka-connect-0.11: base-class swap) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…d tests Converts the Groovy/Spock InstrumenterConfigTest to a JUnit 5 Java test (per repo workflow), preserving all 44 existing cases, and adds explicit coverage for InstrumenterConfig.isDataStreamsEnabled(): default false, and true via DD_DATA_STREAMS_ENABLED. Part of the DSM/Kafka decoupling test task (Task 12a). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Convert the Spock SamplingMechanismTest to a parameterized JUnit 5 Java test, and add DATA_STREAMS cases mirroring the existing DATA_JOBS coverage for validateWithSamplingPriority() and canAvoidSamplingPriorityLock(), per DSMS-161 task 12. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…TREAMS flag Test that AgentInstaller.getEnabledSystems() correctly includes or excludes TargetSystem.DATA_STREAMS based on the data.streams.enabled configuration flag. Also test USM and LLMOBS for consistency. Uses @WithConfig extension to manage configuration state across test cases. Tests verify: - DATA_STREAMS not included when data.streams.enabled is false (default) - DATA_STREAMS included when data.streams.enabled is true - USM not included when usm.enabled is false (default) - USM included when usm.enabled is true - LLMOBS not included when llmobs.enabled is false (default) - LLMOBS included when llmobs.enabled is true Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lients-0.11 Mirrors the fix already applied to kafka-clients-3.8's TracingIterator: when Kafka APM tracing is disabled but Data Streams Monitoring is enabled, and no genuine propagated trace context was extracted from the record headers, the local-root consume span is marked USER_DROP so it is not billed as a normal APM span on kafka-clients < 3.8.0.
Cover the local-root sampling-priority suppression (USER_DROP) that ProducerAdvice and TracingIterator apply when kafka tracing is disabled (via integrations.enabled or the per-integration trace.kafka.enabled override) but DSM is enabled, plus a regression guard confirming spans that join a real propagated trace are not suppressed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… trace `AgentSpan.setSamplingPriority` is trace-level: it resolves to the local root span. The DSM billing-suppression guard was gated on "no context was extracted from the record headers", which is not the same predicate as "this span is the local root". Because the kafka-clients producer and the kafka-streams StreamTask advices create their spans with scope-honouring `startSpan` overloads (USE_SCOPE), the created span can inherit the locally active span -- and since the new DATA_STREAMS mechanism bypasses the sampling priority lock, the guard could force-drop a genuine customer trace (e.g. producing a Kafka message inside an instrumented HTTP request). Producer sites (kafka-clients-0.11 and kafka-clients-3.8) now additionally require `span.getLocalRootSpan() == span`. Entering the else branch already rules out a header-extracted parent, so the local root check is exactly the remaining condition. The kafka-streams sites need a stronger predicate: - `TIME_IN_QUEUE_ENABLED` (on by default) parents the consume span onto the queue span, so `getLocalRootSpan() == span` is never true there; the root candidate is the queue span when one exists. - An `ExtractedContext` parent allocates a fresh TraceCollector, so a span continuing a header-propagated trace is still its own local root. The sibling ContextPropagationAdvice attaches that extracted context to the scope before this advice runs, so the active span captured on entry is what detects it. Both sites therefore require no active span on entry and that the local root is the first span the advice created. Also switches the producer sites from the static `Config.get()` to the remote-config-aware `traceConfig()`, matching the consumer/streams sites so a dynamic DSM toggle cannot make them disagree. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`MessageListenerInstrumentation` applies Code Origin (span origin) advice to Spring Kafka message listeners -- a purely-APM concern with no Data Streams behaviour. Swapping it to `InstrumenterModule.DataStreams` ORed `isDataStreamsEnabled()` into `isEnabled()`, so with `code.origin.enabled=true` and `data.streams.enabled=true` it installed in DSM-only deployments where Kafka tracing is off. Revert this one module to `InstrumenterModule.Tracing` and record why in a class comment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…and streams Regression guards for the two billing-suppression bugs, plus the coverage gaps the final review flagged. All four new specs were confirmed red against a temporarily reverted guard. kafka-streams-0.11 (new spec, module previously had no working DSM coverage -- its only end-to-end test is @ignore'd): - a local-root streams consume span is still forced to USER_DROP - a streams consume span continuing a header-propagated trace is not. The local-root case needs kafka.client.propagation.disabled.topics, since an in-JVM producer would otherwise inject its own context into the headers. kafka-clients-0.11: - producing inside an active local trace does not force that trace to USER_DROP -- the one-line test that would have caught the producer bug - strengthen the extracted-parent scenario from "!= USER_DROP" to the concrete expected SAMPLER_KEEP, and assert the consume span's trace id too kafka-clients-3.8 (new spec, module had no suppression coverage at all): the local-root USER_DROP scenario and the active-local-trace regression guard. These extend InstrumentationSpecification rather than KafkaClientTestBase, whose inherited test asserts Code Origin tags that correctly no longer apply once Kafka tracing is off. SamplingMechanismTest: the only DATA_STREAMS row in the parameterized lock-bypass test left DSM at its disabled default, so it passed for a config reason rather than a mechanism one. Add dedicated tests with data.streams.enabled set both ways, including that enabling DSM does not unlock any other mechanism. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eview findings Fixes a billing gap where the standalone kafka.poll span/trace created for DSM was never forced to USER_DROP, so it would be billed as APM even with Kafka tracing disabled. Also delegates Config.isDataStreamsEnabled() to InstrumenterConfig, dedupes each KafkaDecorator's hardcoded integration-name list into shared constants used by TRACING_ENABLED and every module constructor, extracts the duplicated DSM-drop guard in KafkaStreamTaskInstrumentation into a helper, and rewrites InstrumenterModuleTest to use @WithConfig instead of raw reflection. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Muzzle's static-reference check expects public_or_protected static for a method invoked across nested advice classes, since those classes may be injected into a different classloader than their enclosing class. The package-private helper caused KafkaStreamTaskInstrumentation to be muzzled (blocked) at runtime; making it public resolves the mismatch.
Ensures the cross-language contract holds: when APM tracing is disabled and DSM is enabled, no span is ever written to the agent for Kafka instrumentations. Replaces AgentTracer.startSpan with a lightweight Utils.newPathwayOnlySpan shim across kafka-clients-0.11, kafka-clients-3.8, and kafka-streams-0.11, and extracts shared DSM checkpoint/inject and transaction-tracking logic into helper methods to remove duplication. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
temporary to give a custom build of #12350 to a customer. will close once CI gives an artifact
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]