Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .agents/skills/apm-integrations/references/advice-class.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,34 @@ Exit method:
6. `scope.close()`
7. `span.finish()`

### Batch-consume operations: one span per item, not one span for the whole batch

Some client APIs return a batch of items from a single call — a message broker's poll returning N records, a search client returning a page of hits, a bulk API returning multiple results. If the caller iterates the batch and does further per-item work (deserializing, dispatching to a handler, downstream calls), a single span around the whole batch call is wrong: it cannot attach any of that follow-on work to the specific item that triggered it, and it does not reflect where the actual work happens or ends.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict per-item spans to message consumption

When this guidance is applied to the explicitly listed search pages or bulk API results, it creates a synthetic active span for every returned element even though those elements are parts of one outbound operation rather than independently received messages carrying separate trace contexts. Existing Elasticsearch instrumentation instead creates one span around performRequest/execute (for example, Elasticsearch7RestClientInstrumentation.java:57-104); wrapping a page containing thousands of hits would therefore produce thousands of misleading spans and attribute arbitrary follow-on application work to them. Limit this canonical pattern to domains such as messaging where each item represents an independent consume operation.

Useful? React with 👍 / 👎.


**The pattern**: wrap the returned `Iterable`/`Iterator`/`List` so that advancing to the next item closes the previous item's span and opens a new one for the current item. Do not span the method that returns the batch; span the act of consuming each item from it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Preserve the batch client-operation span

An integration can omit latency and error data for the remote client operation.

Assertion details
  • Input: A future integration follows this guidance for a search or bulk client method that sends a remote request and returns results.
  • Expected: Limit the rule to consumer item or delivery spans. State that a separate client-operation span can cover the remote batch request.
  • Actual: The rule says not to span a method that returns a batch. Its examples include search and bulk client calls, which are remote client operations.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session


```java
// WRONG — one span covers the whole batch; no way to attach per-item follow-on work
@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
public static void exit(@Advice.Return Iterable<Record> records) {
AgentSpan span = startSpan(DECORATE.operationName(), ...);
// ... consume the whole Iterable under one span — individual item work has no span of its own
}

// CORRECT — wrap the Iterable so each item gets its own span, started on next() and
// closed when the following item starts (or when iteration ends)
@Advice.OnMethodExit(suppress = Throwable.class)
public static void exit(@Advice.Return(readOnly = false) Iterable<Record> records) {
if (records != null) {
records = new TracingIterable(records, DECORATE.operationName(), DECORATE);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Extract each item's distributed parent before starting its span

For a messaging batch whose records carry different propagation headers, the shown wrapper API provides only an operation name and decorator, so an implementation following it can start every consumer span from the currently active context rather than from the corresponding producer context. The cited Kafka implementation crucially calls extractContextAndGetSpanContext(val.headers(), GETTER) inside TracingIterator.startNewRecordSpan before startSpan; omitting that requirement disconnects or misparents every per-record trace. Make item-specific context extraction (or the transport's documented batch-parent semantics) part of the required pattern and wrapper contract.

Useful? React with 👍 / 👎.

}
}
```

The wrapping iterator's `next()` starts the span for the item it returns, after first closing whichever span was opened for the previous item. Its `hasNext()` closes the last open span when the delegate has no more items — this is what closes out the final item's span if the caller finishes iterating normally, since there's no explicit "close" call for the last item otherwise. If the caller abandons the iteration partway through (stops calling `next()`/`hasNext()` before reaching the end), the last opened span is left unclosed by this mechanism alone — this is an accepted, known gap (spans opened this way are not finished by a background timeout), not something the advice needs to additionally guard.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the claim that abandoned spans never time out

With the default legacy context manager, the cited Kafka iterator activates each record span through activateNext, whose API explicitly closes root iteration scopes after trace.scope.iteration.keep.alive (30 seconds by default); IterationSpansForkedTest.rootIterationScopeLifecycle verifies that behavior. The parenthetical therefore misstates the canonical lifecycle and may lead future implementations to omit the iteration-scope primitive or devise unnecessary cleanup; document the legacy timeout and any different non-legacy behavior explicitly.

AGENTS.md reference: AGENTS.md:L44-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Correct the iteration timeout guidance

The false cleanup model can cause unnecessary or conflicting cleanup code in future integrations.

Assertion details
  • Input: A caller stops iteration after next() while the default legacy context manager is active.
  • Expected: State that the cleaner finishes an overdue root iteration span after the configured keep-alive time, which is 30 seconds by default.
  • Actual: The guidance says no background timeout finishes an abandoned span. The default legacy context manager schedules a cleaner for root iteration spans.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session


**How to discover the right hook point**: don't span the accessor that *returns* the batch (e.g. a `records()`/`poll()` method returning `Iterable<T>` or `List<T>`) — span the iteration over it. If the batch is returned as an `Iterable`, wrap the `Iterable` (whose `iterator()` produces a wrapping `Iterator`). If it's returned as a `List`, the same wrapping applies to `List.iterator()`/`listIterator()`. See `dd-java-agent/instrumentation/kafka/kafka-clients-0.11/src/main/java/datadog/trace/instrumentation/kafka_clients/{TracingIterable,TracingIterator,TracingList,TracingListIterator}.java` for the canonical implementation — this is the reference pattern for any future batch-consume instrumentation (message queues, but not limited to them).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require List wrappers to preserve the List contract

When a batch API returns a List and an integration follows this recommendation, replacing it with the cited Kafka TracingList changes application-visible collection behavior: that class delegates most operations but does not override equals or hashCode, so two content-equal lists can stop comparing equal after instrumentation (and equality can become asymmetric depending on operand order). Either limit this pattern to Iterable/Iterator return types or require a transparent List wrapper that preserves the complete List contract before presenting it as the canonical pattern for future integrations.

Useful? React with 👍 / 👎.


### onExit handling when the target method throws

The `onThrowable = Throwable.class` attribute on `@Advice.OnMethodExit` controls whether the exit advice fires when the **instrumented target method** throws. You **must** set it explicitly to `Throwable.class` for any exit advice that closes a scope or finishes a span — the default skips exceptional termination, which leaks active scopes when the instrumented method throws.
Expand Down
Loading