Skip to content

fix(telemetry): enrich EventStore spans with full OTel semantic attributes - #92

Merged
yordis merged 1 commit into
mainfrom
yordis/fix-otel-event-store-parity
Apr 21, 2026
Merged

yordis merged 1 commit into
mainfrom
yordis/fix-otel-event-store-parity

Conversation

@yordis

@yordis yordis commented Apr 21, 2026 •

Copy link
Copy Markdown
Member
  • ack_event, subscribe, subscribe_to, unsubscribe, and delete_subscription were removed from EventStore instrumentation because they are lifecycle/control-plane operations — they don't carry domain data and their spans add noise without actionable tracing value (e.g., ack_event fires on every processed event, subscribe/unsubscribe are one-shot setup calls). The remaining operations (append_to_stream, stream_forward, read_snapshot, record_snapshot, delete_snapshot) are the ones that actually touch persisted data and benefit from span-level observability
  • error.type displayed Elixir.RuntimeError instead of RuntimeError in APM tools due to to_string() on module atoms
  • EventStore spans lacked connection-level attributes (server.address, server.port, db.namespace, peer.service) needed for proper service topology mapping in Datadog
  • db.system was not set, preventing APM tools from categorizing spans by database technology
  • event_count, start_version, and read_batch_size were missing from span attributes, reducing observability into EventStore operations

@cursor

cursor Bot commented Apr 21, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Telemetry instrumentation behavior changes (span attributes, destination/connection attribution, and removal of spans for some EventStore lifecycle operations) may affect observability dashboards and alerting expectations, but does not impact core domain behavior.

Overview
EventStore OpenTelemetry instrumentation is refocused to data-path operations only (drops spans for ack_event, subscribe*, unsubscribe, and delete_subscription) while enriching the remaining spans with additional semantic attributes.

append_to_stream now emits commanded.event.count, and stream_forward spans now include commanded.stream.start_version and commanded.stream.batch_size. Connection-level attributes (server.address, server.port, db.namespace, peer.service) are added when available, and error type formatting is normalized (e.g., RuntimeError and :atom_error via inspect/1) with tests updated accordingly.

Reviewed by Cursor Bugbot for commit fdf1b3c. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Apr 21, 2026 •

Copy link
Copy Markdown

Warning

Rate limit exceeded

@yordis has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 41 minutes and 54 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 41 minutes and 54 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 68d43224-9d60-462f-8fbc-35a2fc309f5c

📥 Commits

Reviewing files that changed from the base of the PR and between a3cf3cd and fdf1b3c.

📒 Files selected for processing (11)
  • lib/commanded/event_store.ex
  • lib/commanded/opentelemetry/commanded_attributes.ex
  • lib/commanded/opentelemetry/event_store.ex
  • lib/commanded/opentelemetry/helpers.ex
  • test/opentelemetry/aggregate_populate_test.exs
  • test/opentelemetry/aggregate_snapshot_test.exs
  • test/opentelemetry/aggregate_test.exs
  • test/opentelemetry/application_test.exs
  • test/opentelemetry/event_handler_test.exs
  • test/opentelemetry/event_store_test.exs
  • test/support/factory.ex

Walkthrough

The PR enhances OpenTelemetry instrumentation for Commanded's event store by adding detailed span attributes for streaming operations, extracting attribute-construction logic into reusable helpers, and reformatting exception type reporting to remove the "Elixir." prefix convention.

Changes

Cohort / File(s) Summary
OpenTelemetry Attribute Infrastructure
lib/commanded/opentelemetry/commanded_attributes.ex
Added five new EventStore-related attribute functions: eventstore_read_count, eventstore_stream_start_version, eventstore_stream_direction, eventstore_stream_batch_size, and eventstore_stream_delete_type.
OpenTelemetry Helper Functions
lib/commanded/opentelemetry/helpers.ex
Extracted attribute-building logic into reusable helpers (maybe_add_operation_type, maybe_add_destination_name, maybe_add_event_count, maybe_add_subscription_name, maybe_add_source_uuid, etc.), added to_destination_name/1, simplified maybe_add_connection_attributes/2, and changed error-type formatting from to_string/1 to inspect/1.
Event Store Instrumentation
lib/commanded/event_store.ex, lib/commanded/opentelemetry/event_store.ex
Added event_count to telemetry metadata; refactored span attribute construction to delegate to Helpers module; removed instrumentation support for ack_event, delete_subscription, subscribe, subscribe_to, and unsubscribe operations.
Test Expectations
test/opentelemetry/aggregate_test.exs, test/opentelemetry/application_test.exs, test/opentelemetry/event_handler_test.exs, test/opentelemetry/event_store_test.exs
Updated error-type assertions to match new inspect/1-based formatting (e.g., "ArgumentError" instead of "Elixir.ArgumentError"); extended span attribute expectations to include event_count, eventstore.stream.start_version, and eventstore.stream.batch_size.
Test Support
test/support/factory.ex, test/support/opentelemetry_case.ex
Extended event store append-to-stream metadata factory to support and include event_count; expanded telemetry handler detachment in cleanup to cover all EventStore operation variants.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • straw-hat-team/commanded#37: Modifies EventStore OpenTelemetry instrumentation to add telemetry metadata including event_count and initial span attribute infrastructure.
  • straw-hat-team/commanded#91: Refactors EventStore OpenTelemetry span attribute construction and helper functions for connection/database attributes.
  • straw-hat-team/commanded#45: Modifies OpenTelemetry Helpers module, including the to_error_type/2 function that is updated in this PR.

Poem

🐰 With whiskers twitched and nose in code,
I trace the spans down telemetry's road,
New attributes bloom like carrots in spring,
Event counts and stream starts—oh what they sing!
Helper functions hop, no duplication in sight,
The OpenTelemetry path shines ever bright! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 21.62% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: enriching EventStore spans with full OpenTelemetry semantic attributes as part of fixing telemetry instrumentation.
Description check ✅ Passed The pull request description clearly explains the rationale and specific changes made to enrich EventStore telemetry spans with OpenTelemetry semantic attributes, including removal of noisy operations, error type formatting fixes, and new connection and operational attributes.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch yordis/fix-otel-event-store-parity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@yordis
yordis force-pushed the yordis/fix-otel-event-store-parity branch 2 times, most recently from b71eccb to 1e6f2db Compare April 21, 2026 01:54

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
lib/commanded/opentelemetry/commanded_attributes.ex (1)

188-201: Align these new attribute names with the module contract.

The module docs say all attributes MUST use the commanded. prefix, but these new public constants return eventstore.*. Either rename them to the Commanded namespace or update the docs to explicitly carve out EventStore-specific attributes.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@lib/commanded/opentelemetry/commanded_attributes.ex` around lines 188 - 201,
The public attribute functions eventstore_read_count,
eventstore_stream_start_version, eventstore_stream_direction,
eventstore_stream_batch_size, and eventstore_stream_delete_type currently return
atoms under the "eventstore.*" namespace which violates the module contract that
all attributes MUST use the "commanded." prefix; update each function and its
`@spec` to return the corresponding "commanded.eventstore.*" atom (e.g.
:"commanded.eventstore.read.count", etc.) so they conform to the module naming
convention, and adjust any callers/tests or the module docs if you instead
decide to explicitly declare these as EventStore-specific (but prefer renaming
for consistency).
test/opentelemetry/event_store_test.exs (1)

94-96: LGTM — new attribute assertions match the enriched span schema.

commanded.event.count, eventstore.stream.start_version, eventstore.stream.batch_size, and db.system are consistently asserted across success, missing-stream, and error/exception paths.

One note: the stream_forward assertion hardcodes "eventstore.stream.batch_size": 1000. If the default batch size is sourced from config and ever changes, this test will be brittle. Consider deriving from the same constant/config the production code uses.

Also applies to: 120-123, 219-222

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/opentelemetry/event_store_test.exs` around lines 94 - 96, The test
hardcodes "eventstore.stream.batch_size": 1000 in the stream_forward assertion
which will break if the production default changes; update the test to derive
the expected batch size from the same source the runtime uses (e.g., call the
production accessor or read the application config) instead of the literal 1000
so the assertion in the stream_forward test (and the other occurrences at the
referenced assertion blocks) stays in sync with the production constant/config.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/commanded/opentelemetry/helpers.ex`:
- Around line 50-55: The fallback atom clause returns "Elixir.RuntimeError" for
module atoms; add a specific clause in Helpers.to_error_type to handle module
atoms before the generic atom fallback: add a clause like def
to_error_type(module, _tracer_id) when is_atom(module) and
String.starts_with?(Atom.to_string(module), "Elixir."), do: inspect(module)
(placed above def to_error_type(error, _tracer_id) when is_atom(error)), so
module atoms use inspect/1 while non-module atoms (e.g., :validation_failed)
still fall through to to_string/1.

In `@test/support/opentelemetry_case.ex`:
- Around line 61-69: The cleanup is generating telemetry names as [:eventstore,
op, suffix] but Commanded.OpenTelemetry.EventStore.setup/0 registers handlers
under [:commanded, :event_store, op, suffix], so update the eventstore_events
construction (which currently builds eventstore_events from
eventstore_operations and suffixes) to produce [:commanded, :event_store, op,
suffix] instead, ensuring the loop that iterates for event <- commanded_events
++ eventstore_events will detach the exact handlers registered by
Commanded.OpenTelemetry.EventStore.setup/0.

---

Nitpick comments:
In `@lib/commanded/opentelemetry/commanded_attributes.ex`:
- Around line 188-201: The public attribute functions eventstore_read_count,
eventstore_stream_start_version, eventstore_stream_direction,
eventstore_stream_batch_size, and eventstore_stream_delete_type currently return
atoms under the "eventstore.*" namespace which violates the module contract that
all attributes MUST use the "commanded." prefix; update each function and its
`@spec` to return the corresponding "commanded.eventstore.*" atom (e.g.
:"commanded.eventstore.read.count", etc.) so they conform to the module naming
convention, and adjust any callers/tests or the module docs if you instead
decide to explicitly declare these as EventStore-specific (but prefer renaming
for consistency).

In `@test/opentelemetry/event_store_test.exs`:
- Around line 94-96: The test hardcodes "eventstore.stream.batch_size": 1000 in
the stream_forward assertion which will break if the production default changes;
update the test to derive the expected batch size from the same source the
runtime uses (e.g., call the production accessor or read the application config)
instead of the literal 1000 so the assertion in the stream_forward test (and the
other occurrences at the referenced assertion blocks) stays in sync with the
production constant/config.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cd16093c-cbeb-4bfd-9d8b-9b829fd539b5

📥 Commits

Reviewing files that changed from the base of the PR and between d09a91e and a3cf3cd.

📒 Files selected for processing (10)
  • lib/commanded/event_store.ex
  • lib/commanded/opentelemetry/commanded_attributes.ex
  • lib/commanded/opentelemetry/event_store.ex
  • lib/commanded/opentelemetry/helpers.ex
  • test/opentelemetry/aggregate_test.exs
  • test/opentelemetry/application_test.exs
  • test/opentelemetry/event_handler_test.exs
  • test/opentelemetry/event_store_test.exs
  • test/support/factory.ex
  • test/support/opentelemetry_case.ex

Comment thread lib/commanded/opentelemetry/helpers.ex Outdated
Comment thread test/support/opentelemetry_case.ex Outdated
Comment thread lib/commanded/opentelemetry/helpers.ex
Comment thread lib/commanded/opentelemetry/commanded_attributes.ex Outdated
@yordis
yordis force-pushed the yordis/fix-otel-event-store-parity branch 4 times, most recently from 3fbc2ae to b0441c2 Compare April 21, 2026 02:05
…butes

- Orphaned operations (ack_event, subscribe, subscribe_to, unsubscribe,
  delete_subscription) removed from EventStore instrumentation since they
  produce noise without meaningful tracing value
- error.type now uses inspect() to drop the Elixir. prefix for cleaner
  APM display
- Connection attributes (server.address, server.port, db.namespace) and
  peer.service set to the actual database name for proper service mapping
- db.system derived from adapter type (postgresql for EventStore adapter,
  in_memory for InMemory)
- event_count, start_version, read_batch_size added to span attributes
- Shared attribute helpers extracted to Helpers module for reuse

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis
yordis force-pushed the yordis/fix-otel-event-store-parity branch from b0441c2 to fdf1b3c Compare April 21, 2026 02:08

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fdf1b3c. Configure here.

Comment thread lib/commanded/opentelemetry/helpers.ex
@yordis
yordis merged commit 0d0763a into main Apr 21, 2026
3 checks passed
@yordis
yordis deleted the yordis/fix-otel-event-store-parity branch April 21, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant