Repository navigation
[release/11.0] Match old layouts and feature selection for named mutexes for cross-TFM compatibility - #134595
Merged
Conversation
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/area-system-diagnostics-process |
jkoritzinsky
approved these changes
Sep 24, 2026
jkotas
approved these changes
Sep 24, 2026
adamsitnik
added
area-System.Threading
and removed
area-System.Diagnostics.Process
labels
Contributor
|
Tagging subscribers to this area: @JulieLeeMSFT, @VSadov |
JulieLeeMSFT
added
the
Servicing-consider
rbhanda
added
Servicing-approved
Member
|
@jkoritzinsky, please check test failures and do ba-g if all good. |
Member
|
/ba-g WASM timeouts (this PR doesn't touch code that runs on WASM) |
andrewlock
added a commit
to DataDog/dd-trace-dotnet
that referenced
this pull request
Oct 15, 2026
## Summary of changes Update to building and testing with .NET 11 RC2 ## Reason for change We want to test with the latest. This version also includes some bug fixes so we can revert some hacks we added. ## Implementation details - Bump the version we use from RC1 to RC2 - Rebuild GitLab image using RC2 - Revert #9350 and #9321, as these were workarounds for bugs in RC1 which are fixed in RC2 - The [code coverage bug](dotnet/runtime#134595) does _not_ appear to have shipped in RC2, and will ship in the .NET 11 GA instead. - Tried to re-enable x64 alpine .NET 11 crash tests, in case the issue was resolve, but still seeing the same problem so reversed - Fixed an unrelated issue with Debian archive mirror failing, which was required in order to buid the image on arm64 (getting 404s, so was unable to build the image) - Exclude the `network.peer.port`, `http.client_ip`, `network.client.ip`, and `url.query` tags that are now added by default to the aspnetcore activity in .NET 11 - We have explicit settings to enable/disable including these (`DD_TRACE_CLIENT_IP_ENABLED` and `DD_HTTP_SERVER_TAG_QUERY_STRING`) which we need to respect. - Given we also _don't_ overwrite the tag with the activity version if we set it ourselves, this translates to just "never copy it" Additionally - [Installed RC2 in the macro-benchmarks](DataDog/benchmarking-platform@7f4c52e) and rebuilt the docker images and AMI - Rebuilt the micro-benchmark AMI - Updated serverless tools. Requires merging: - DataDog/serverless-tools#139 - Updated the Windows VMs, as just added the extra SDK for simplicity/compatibility - I'll update the Linux VMs _after_ merging this PR, to avoid impacting others. - Only downside is builds will be a bit slow immediately after merging the PR ## Test coverage This is the test. Once everything is green, we're good ## Other details Follows on from - #9317 Required for the .NET 11 work. https://datadoghq.atlassian.net/browse/APMLP-1455 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
tonyredondo
added a commit
to DataDog/dd-trace-dotnet
that referenced
this pull request
Oct 15, 2026
) ## Summary of changes Restore MSTest 4.4.0 discovery counts and record each native retry attempt, assigning `final_status` only to the final span for each test. Preserve older MSTest cleanup paths and report cleanup failures before the suite closes. Builds on #9189, already in master. ## Reason for change MSTest 4.4 changed discovery and native retry execution. Its retry policy selects results before class cleanup, so Datadog retries must run while row arguments and the framework lifecycle remain available. The compatibility review also found missing cleanup hooks in MSTest 2.0–2.2.7 and returned cleanup exceptions in 3.8–3.10. ## Implementation details - Count discovered tests before filtering, including isolated .NET Framework AppDomains; retain the 4.3.x discovery hook. - Intercept retry policy completion and restore row arguments and a fresh TestContext. MSTest creates and cleans up a fresh class instance for each retry. Track duplicate rows/results by position so identical names and arguments retain separate final outcomes and initial EFD decisions. - Finish execution coverage and callbacks once, retain the captured duration, then assign final retry tags and close the span under the same guard as shutdown. Selenium uses the execution-completion callback to flush RUM while the attempt context is active. - Register the existing `TestOptimization.ShutdownAsync` hook before agent discovery can initialize the tracer. No extra shutdown hook or `TracerManager` changes. - Match legacy cleanup signatures, consume returned exceptions and mark the suite failed before closure. Preserve the runner fallback for open suites and prevent writes after closure. - Keep CI event compression consistent while agent discovery completes: capture the encoding once, then use the same header for those bytes on every retry. Static callbacks avoid a closure per batch. - Recover IPC after an abandoned mutex: `WaitOne` already grants ownership when it throws, so the reader must continue through its existing release `finally`. The previous early return blocked child processes sending the EFD `faulty` tag. This pre-existing handler defect is exposed by the new MSTest discovery scenarios; the correction changes only that catch and adds a regression to `CircularChannelTests`. Temporarily skip the 17 non-empty VSTest discovery cases per affected Linux ARM64 net8/net9/net10 CI leg when the selected SDK is one of the two known .NET 11 RC1/RC2 builds. [dotnet/runtime#134491](dotnet/runtime#134491) prevents the runner and testhost from sharing named mutexes: IPC initialization also fails in cases that do not assert `faulty`. The 14 MTP/empty-run cases and other platforms remain enabled. Comments link the [GA backport](dotnet/runtime#134595) and require removing the skip after final-runtime validation on Debian/Alpine ARM64. Selecting the final SDK already stops the skip; no production workaround or infrastructure change is included. Windows net48 test setup adds only the Visual Studio `18\BuildTools` entry to the existing runner lookup. The new MSTest discovery, retry and cleanup fixtures opt into Windows execution, exposing a missing-runner path that the base net48 CI job did not exercise. The lookup structure, installation roots and fallback are unchanged. All regression scenarios reuse existing sample projects; this PR adds **no sample projects**: - `Samples.MSTestTestsRetries`: native retry scenarios under `NativeRetries`, alongside the existing Datadog retry tests. One binary serves VSTest and MTP; a shared class filter keeps the existing retry and quarantine suites separate from native retry scenarios. Compile 4.4-only APIs only for compatible package versions. - `Samples.MSTestTests` and `Samples.MSTestTests2`: discovery tests select existing passing cases while checking the full discovery count, including repeated and multiple input assemblies. - `Samples.Console`: an early command exercises shutdown before normal tracer initialization. Removed four separate projects and three package matrices. For one local net8.0/MSTest 4.4.0 configuration, their corresponding sample outputs fell from 41.7 to 17.7 MiB (about 58%). This measures build output only, not total CI disk usage. The shared MSTest/retry defaults use 4.4.0 on .NET 8+/Framework and 3.11.1 on older .NET; explicit package versions are unchanged. ## Test coverage Current local validation on master `4a653769`: **106 sender/IPC/MSTest lifecycle unit cases and 34 MSTest integration cases pass without skips**. Three integration cases cover class-cleanup failures with MSTest 3.11.1 (both lifecycles) and 4.4.0; the other 31 cover discovery/EFD. All six compression regressions also pass on .NET 11. The 31 discovery cases validate the current managed tracer on Linux x64 with MSTest 4.3.3, 4.4.0 and 4.4.1, including VSTest/MTP and repeated/multiple input assemblies. An isolated check of the exact guard with simulated platform/SDK inputs verifies the affected RC combinations emit the skip reason and final/other SDKs remain enabled; actual ARM64 skip execution is pending CI. Another **72 IPC cases pass on .NET 11**, including the abandoned-mutex regression. The new abandoned-mutex regression failed against the preceding reader before the correction and passes afterward. All four managed tracer targets, the net8 integration project and the net48 unit project compile without warnings; native registrations are unchanged at 856. Integration uses the branch managed tracer with native/loader artifacts from Azure 211224's merge `3fc4c6b1`. The local invocation disables SDK telemetry to avoid inheriting an external session; temporary source diagnostics have been removed. [Azure build 211267](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=211267) tested merge `4705dc4b` for the preceding head `c8a3eef7` against `2cc84520`: all Windows and Linux x64 CIVisibility jobs passed with the IPC recovery correction. Its six Debian/Alpine ARM64 net8/net9/net10 CIVisibility jobs each had the 11 discovery failures explained above. These results precede the temporary skip. [Azure build 211296](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=211296), merge `93b8fc44` for `a56c5e6f`, exposed six further non-empty VSTest cases whose IPC errors fail the log check; this correction includes them. That build passed 59 of 60 Windows integration jobs; `Win x64_net5.0_CIVisibility` lost the cleanup suite event after the mock agent rejected the payload with a gzip exception and five HTTP 500 responses. Current-head CI and ARM64 skip execution remain pending. - Native retries: 86 VSTest/MTP cases covering custom policies, parameterized rows, fresh instances, initialization failures, final tags, result history and Test Optimization policies. Eight regressions cover duplicate rows and identical custom-executor results with automatic retries on/off; all eight failed against the previous production binaries before the identity fix. - Quarantine: all five combinations of quarantine and automatic retries, including an unquarantined failure, run only the original retry suite. The existing assertions check event counts, final status, retry tags and process exit codes. - Cleanup: supported signatures across MSTest 2.0.0–4.4.0, EndOfClass/EndOfAssembly, returned exceptions, suite status and late-tag warnings. - Discovery/shutdown: filtering, empty runs, EFD thresholds, isolated AppDomains, repeated/multiple sources, and session delivery on explicit close, process exit and exception shutdown. Historical validation before the latest rebase: after rebasing onto master `67afbadd`, **160 local integration cases and 25 lifecycle/instrumentation unit cases pass without skips**: 86 native retries, 17 discovery cases, 16 versioned legacy retry/cleanup/ExceptionReplay cases, and 41 default-mode cases. The default run also covers the existing SubmitTraces/EFD snapshots and Console initialization. Relevant samples compile with MSTest 2.0.0, 2.2.10, 3.7.3, 3.8.3, 3.11.1, 4.3.3 and 4.4.0; net6/net8 defaults pass. All four managed tracer targets and the net8/net48 integration projects compile without warnings. Native registrations were regenerated from the four tracer builds and verified against the combined master/PR definitions (858 registrations, no missing or unexpected entries). Before the latest rebase, the shared-sample filter correction passed **56 fixture cases without skips**: five quarantine variants plus retries and ExceptionReplay for each of seven MSTest versions (2.0.0, 2.2.10, 3.7.3, 3.8.3, 3.11.1, 4.3.3 and 4.4.0), plus seven default-mode cases. The five default quarantine variants failed before the filter change. This validation uses the branch managed tracer, the native tracer from Azure 209332 (checkout `61c0d5cf`), and the existing loader from Azure 208877. This predates the current rebase and Windows corrections. Those historical local runtime tests used the managed tracer built from that revision and native/loader artifacts from Azure 208877's merge revision `edc2a7bf`; those native inputs differ from the branch. These historical results do not validate the current head. [Azure build 208877](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=208877) passed for the previous head `31c3e4fa`; it does not validate this consolidation. ## Other details Temporary gzip diagnostics remain removed. The Windows cleanup investigation reproduced a shared sender race: discovery can enable gzip after the body was encoded, causing a plain MessagePack body to be sent with `Content-Encoding: gzip`. This PR now includes the minimal sender correction and six real-payload regression cases. Three cases fail with the original sender and the same decompression exception, including discovery completing before a retry. Attribution of the historical Windows x86/.NET 8 and x64/.NET 5 failures to this race still requires CI confirmation; no rejected wire payload was captured. Unrelated IIS snapshot changes remain excluded. The [microbenchmark report](#9192 (comment)) for `a56c5e6f` against `2cc84520` flags 12 regressions, including allocations and throughput; attribution remains unconfirmed. The [execution-time report](#9192 (comment)) compares the same revisions and reports no regressions. Both predate the current sender correction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #134541 to release/11.0
/cc @jkoritzinsky
Customer Impact
Using a named mutex on Unix between .NET 11 RC1 apps and .NET 10 and lower apps results in spurious
AbandondedMutexExceptionexceptions for non-abandoned mutexes.Regression
#117635
Testing
Manual validation.
Risk
Low. Only touches Named Mutex logic.