Skip to content

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation - #7050

Draft
pabloDeputter wants to merge 6 commits into
masterfrom
pablo/fix-botocore-sigv4-trace-propagation
Draft

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation#7050
pabloDeputter wants to merge 6 commits into
masterfrom
pablo/fix-botocore-sigv4-trace-propagation

Conversation

@pabloDeputter

@pabloDeputter pabloDeputter commented Aug 5, 2026

Copy link
Copy Markdown
Member

Description

Summary of what lead to this issue

  • pre-4.12.0 ddtrace basically behaved like Sentry does now: propagation headers were added by the HTTP client after botocore already had calculate the SigV4 signature. So baggage was not included in SignedHeaders. Any later modifications to the value did not invalidate the request.

  • after 4.12.0 moves the header injection to botocore's before-sign event. It adds baggage, ... and x-datadog-* before signing, then suppresses its later HTTP-client injection to avoid duplicate headers.

  • How failure happens:

    1. An incoming FastAPI request contains Sentry or other W3C baggage.
    2. Datadog extracts that baggage into its active context.
    3. A boto3 request is created.
    4. Datadog’s before-sign handler writes the baggage to the AWS request.
    5. Botocore includes baggage in the SigV4 signature.
    6. Sentry’s generic HTTP instrumentation runs afterward and updates or adds another baggage value.
    7. AWS receives a different canonical baggage value from the one that was signed. Because signed baggage was modified after signing, AWS rejects the request with 403 Forbidden or SignatureDoesNotMatch.
  • merge Sentry baggage with existing vendor (e.g. Datadog) baggage in botocore'sbefore-sign hook; avoiding post-sign header tampering that invalidates the SigV4 signature.

  • Skip propagation for presigned requests

Issues

Resolves: #7031 & PY-2667

Reminders

- merge Sentry baggage with existing vendor (e.g. Datadog) baggage in botocore's`before-sign` hook; avoiding post-sign header tampering that invalidates the SigV4 signature.
- Skip propagation for presigned requests

Fixes: #7031 & PY-2667
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codecov Results 📊

99287 passed | ⏭️ 6494 skipped | Total: 105781 | Pass Rate: 93.86% | Execution Time: 354m 38s

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +140
Passed Tests 📈 +126
Failed Tests 📉 -1
Skipped Tests 📈 +15

➖ Removed Tests (1)

View removed tests
  • test_continuous_profiler_auto_start_and_stop_sampled[experiment-gevent]
    • File: tests.profiler.test_continuous_profiler

All tests are passing successfully.

✅ Patch coverage is 95.06%. Project has 2508 uncovered lines.
✅ Project coverage is 89.98%. Comparing base (base) to head (head).

Files with missing lines (2)
File Patch % Lines
sentry_sdk/integrations/stdlib.py 94.55% ⚠️ 3 Missing and 4 partials
sentry_sdk/integrations/boto3.py 96.15% ⚠️ 1 Missing and 1 partials
Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
+ Coverage    89.95%    89.98%    +0.03%
==========================================
  Files          193       193         —
  Lines        24955     25029       +74
  Branches      9004      9038       +34
==========================================
+ Hits         22448     22521       +73
- Misses        2507      2508        +1
- Partials      1437      1440        +3

Generated by Codecov Action

- trace-header injection is delayed until `endheaders()` after all request headers are available.
- avoid duplicate propagation headers.
- no tampering/modifying of already signed baggage

Fixes: #7031 & PY-2667
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 HeadObject returns 403 when sentry_sdk.start_span and Datadog botocore instrumentation are both active

1 participant