Fix negative tmp-buffer size in vxsort align_vectorized by janvorli · Pull Request #134881 · dotnet/runtime

janvorli · 2026-09-29T21:10:28Z

Issue

Server/Workstation GC background threads can crash with an access violation (read) inside coreclr!memcpy, called from vxsort<T,M,Unroll,Shift>::vectorized_partition, itself called from do_vxsort during mark_phase/plan_phase sorting of the GC mark list. This reproduces the crash originally reported in #86548 ("The size calculated to be negative and cause Access Violation when doing vxsort vectorized_partition"), which was closed without a fix in 2023 due to lack of a reliable repro. Analysis of a production Windows crash dump (Second Chance Exception c0000005, AMD EPYC/Zen4 AVX-512) root-caused the same failure independently and pinpointed the exact algorithmic defect below.

Root cause

In vxsort::align_vectorized(), when the left read pointer isn't vector-aligned, the code pre-aligns by reading a full vector starting before left. The low L = -leftAlign lanes of that vector are "junk" elements that lie before the current [left, right] partition range (already settled by an ancestor partition step), while the remaining lanes are real elements of the current range. Both junk and real lanes are compared against the current pivot and compacted together (via compress-store or an order-preserving permute), so junk elements can land on either side of the pivot - they are not guaranteed to all be <= pivot.

The code advanced tmpLeft by the true, data-dependent count of <=pivot lanes (ltPopCountLeftPart), but advanced tmpStartLeft - the base pointer later used to size the copy-back region (leftTmpSize = tmpLeft - tmpStartLeft) - by the constant L, implicitly assuming all L junk lanes are <=pivot. When fewer than L junk lanes are actually <=pivot, tmpStartLeft overshoots tmpLeft, producing a negative leftTmpSize. That negative value, reinterpreted as an unsigned byte count in the subsequent memcpy, causes a massive out-of-bounds read and the access violation.

The formula was validated against both the original #86548 report (L=7, ltMask=0x3f -> ltPopCountLeftPart=2 -> deficit of -5 elements = -20 bytes, exactly matching the reported value) and against the investigated crash dump (1-element/4-byte deficit reproduced from register/stack state captured in the dump).

The right-side counterpart (tmpStartRight -= rightAlign & rai) has the same class of defect for junk elements read beyond right. It was partially papered over by rtPopCountRightPart = max(mask_popcount( rtMask), rightAlign), which avoids the crash but can instead leave an uninitialized "hole" in the tmp buffer that gets silently copied back into the array as if it were valid sorted data.

Fix

Replace the constant junk-lane skip on both sides with the exact, data-dependent count of junk lanes that landed on the "keep" side of the pivot, computed via a popcount masked to just the junk-lane bits (low L bits for the left side, high rightAlign bits for the right side). This removes the incorrect assumption entirely instead of papering over its symptoms, and makes the previous max() band-aid on the right side unnecessary (removed).

Validated by a full CoreCLR Debug x64 native build (build-runtime.cmd), confirming the change compiles cleanly across the WKS/SVR GC, gcsample, and NativeAOT runtime consumers, for both AVX2 and AVX-512 machine traits and both int32/int64 element types.

Fixes #86548

…torized sort) Issue: Server/Workstation GC background threads can crash with an access violation (read) inside coreclr!memcpy, called from vxsort<T,M,Unroll,Shift>::vectorized_partition, itself called from do_vxsort during mark_phase/plan_phase sorting of the GC mark list. This reproduces the crash originally reported in dotnet#86548 ("The size calculated to be negative and cause Access Violation when doing vxsort vectorized_partition"), which was closed without a fix in 2023 due to lack of a reliable repro. Analysis of a production Windows crash dump (Second Chance Exception c0000005, AMD EPYC/Zen4 AVX-512) root-caused the same failure independently and pinpointed the exact algorithmic defect below. Root cause: In vxsort::align_vectorized(), when the left read pointer isn't vector-aligned, the code pre-aligns by reading a full vector starting before `left`. The low `L = -leftAlign` lanes of that vector are "junk" elements that lie before the current [left, right] partition range (already settled by an ancestor partition step), while the remaining lanes are real elements of the current range. Both junk and real lanes are compared against the current pivot and compacted together (via compress-store or an order-preserving permute), so junk elements can land on either side of the pivot - they are not guaranteed to all be <= pivot. The code advanced `tmpLeft` by the true, data-dependent count of <=pivot lanes (`ltPopCountLeftPart`), but advanced `tmpStartLeft` - the base pointer later used to size the copy-back region (`leftTmpSize = tmpLeft - tmpStartLeft`) - by the *constant* `L`, implicitly assuming all L junk lanes are <=pivot. When fewer than L junk lanes are actually <=pivot, tmpStartLeft overshoots tmpLeft, producing a negative leftTmpSize. That negative value, reinterpreted as an unsigned byte count in the subsequent memcpy, causes a massive out-of-bounds read and the access violation. The formula was validated against both the original dotnet#86548 report (L=7, ltMask=0x3f -> ltPopCountLeftPart=2 -> deficit of -5 elements = -20 bytes, exactly matching the reported value) and against the investigated crash dump (1-element/4-byte deficit reproduced from register/stack state captured in the dump). The right-side counterpart (`tmpStartRight -= rightAlign & rai`) has the same class of defect for junk elements read beyond `right`. It was partially papered over by `rtPopCountRightPart = max(mask_popcount( rtMask), rightAlign)`, which avoids the crash but can instead leave an uninitialized "hole" in the tmp buffer that gets silently copied back into the array as if it were valid sorted data. Fix: Replace the constant junk-lane skip on both sides with the exact, data-dependent count of junk lanes that landed on the "keep" side of the pivot, computed via a popcount masked to just the junk-lane bits (low L bits for the left side, high `rightAlign` bits for the right side). This removes the incorrect assumption entirely instead of papering over its symptoms, and makes the previous `max()` band-aid on the right side unnecessary (removed). Validated by a full CoreCLR Debug x64 native build (build-runtime.cmd), confirming the change compiles cleanly across the WKS/SVR GC, gcsample, and NativeAOT runtime consumers, for both AVX2 and AVX-512 machine traits and both int32/int64 element types. Fixes dotnet#86548 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b029aa4e-2a19-4582-b9e5-4918cb4c3095

janvorli · 2026-09-29T21:10:56Z

cc: @damageboy

azure-pipelines · 2026-09-29T21:11:15Z

Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-09-29T21:12:20Z

Tagging subscribers to this area: @anicka-net, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue: The previous commit fixed the negative-tmp-size access violation from dotnet#86548 by making tmpStartLeft/tmpStartRight advance by the true, data-dependent count of junk lanes that landed on their own expected side, instead of an assumed constant count. That fix is necessary but was shown (via review) to be insufficient: it only prevents the crash, it does not prevent junk elements from a boundary read from being silently duplicated into the sorted output when they land on the *opposite* side of the pivot. Root cause (refined): In align_vectorized(), when leftAlign < 0, LT0's low (-leftAlign) lanes are "junk" elements read from before `left` - they belong to an already-settled, ancestor partition range, not the current [left, right] range. Symmetrically, when rightAlign > 0, RT0's high (rightAlign) lanes are junk elements read from beyond `right`. Junk elements are still compared against the *current* pivot, and that comparison can go either way. The shared tmp buffers are filled in a fixed order: RT0's ">pivot" contribution is written first, ending exactly at tmpStartRight (the only position from which trimming happens), and LT0's ">pivot" contribution is written second, immediately below it - never adjacent to tmpStartRight. So while tmpStartRight can correctly trim away RT0's own junk (My previous commit's fix), any LT0 junk lane that compares as ">pivot" and ends up compress-stored into the same ">pivot" buffer is wedged in the middle of it, with no boundary adjustment able to reach it. It is therefore included, unaccounted for, in the final memcpy copy-back - duplicating an out-of-range element into the array instead of leaving it untouched. The mirror case applies to RT0 junk lanes that compare as "<=pivot" leaking into the left buffer. Fix: Force the classification of junk lanes themselves, before it is used for anything: LT0's junk-lane bits are cleared in ltMask (forced to "<=pivot"), and RT0's junk-lane bits are set in rtMask (forced to ">pivot"). This guarantees junk lanes always land on their designated, trimmable side (adjacent to tmpStartLeft/tmpStartRight, in lane order) regardless of their real value versus the pivot, so they can never cross to the opposite bucket. With this in place, the boundary trims revert to the simple, original constant forms (tmpStartLeft += -leftAlign, tmpStartRight -= rightAlign), which are now provably exact instead of merely assumed. This is the "controlled placement" principle (masking/forcing boundary-lane classification ahead of partitioning, akin to damageboy's masked-load approach in the upstream vxsort project), rather than trying to reconcile mismatched counts after the fact. Verified by hand against a worked example (pivot=50, leftAlign=-7, rightAlign=+6, with several boundary lanes on the "wrong" side of the pivot): with this fix, all 8 LT0 lanes are forced left and trimmed down to just the 1 genuinely in-range element, and all 8 RT0 lanes are forced right and trimmed down to just the 2 genuinely in-range elements, with zero cross-contamination in either direction. Also validated by a full CoreCLR Debug x64 native build (build-runtime.cmd), confirming the change compiles cleanly across the WKS/SVR GC, gcsample, and NativeAOT runtime consumers, for both AVX2 and AVX-512 machine traits and both int32/int64 element types. Refines the fix for dotnet#86548 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b029aa4e-2a19-4582-b9e5-4918cb4c3095

janvorli · 2026-10-05T17:29:07Z

/backport to release/11.0

janvorli · 2026-10-05T17:29:16Z

/backport to release/10.0

github-actions · 2026-10-05T17:29:18Z

Started backporting to release/11.0 (link to workflow run)

github-actions · 2026-10-05T17:29:26Z

Started backporting to release/10.0 (link to workflow run)

github-actions · 2026-10-05T17:31:14Z

@janvorli backporting to release/10.0 failed, the patch most likely resulted in conflicts. Please backport manually!

git am output

$ git cherry-pick 9110af0ee4da5e7b472157668b6a1e5489773f03

Auto-merging src/coreclr/gc/vxsort/vxsort.h
CONFLICT (content): Merge conflict in src/coreclr/gc/vxsort/vxsort.h
error: could not apply 9110af0ee4d... Fix negative tmp-buffer size in vxsort align_vectorized (#134881)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"


$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: Fix negative tmp-buffer size in vxsort align_vectorized (AV in GC vectorized sort)
Using index info to reconstruct a base tree...
M	src/coreclr/gc/vxsort/vxsort.h
Falling back to patching base and 3-way merge...
Auto-merging src/coreclr/gc/vxsort/vxsort.h
CONFLICT (content): Merge conflict in src/coreclr/gc/vxsort/vxsort.h
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Fix negative tmp-buffer size in vxsort align_vectorized (AV in GC vectorized sort)
Error: The process '/usr/bin/git' failed with exit code 128

Link to workflow output

@janvorli

#135230) Backport of #134881 to release/10.0 /cc @janvorli ## Customer Impact - [x] Customer reported - #86548, - [ ] Found internally Server/Workstation GC background threads can crash with an access violation inside coreclr!memcpy, called from vxsort code during mark_phase/plan_phase sorting of the GC mark list. The issue was first reported in May 2023, but we were never able to reproduce it. Now a customer started to hit this issue on a regular basis in Azure App Service powering a multi-tenant SaaS solution. It resulted in SLA violations impacting the customer. The culprit is a bug in the vxsort that has been in vxsort code ever since it was implemented in .NET Core 6 ## Regression - [x] Yes, introduced in .NET 6 in #37159 - [ ] No ## Testing coreclr tests executed with server GC with GC stress C ## Risk Low, the change is local to boundary-lane classification in vxsort and preserves the existing sorting and buffer-management logic. It forces out-of-range lanes onto the side from which they are subsequently trimmed, preventing invalid copy sizes and contamination of the sorted output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b029aa4e-2a19-4582-b9e5-4918cb4c3095

@janvorli

#135229) Backport of #134881 to release/11.0 /cc @janvorli ## Customer Impact - [x] Customer reported - #86548, - [ ] Found internally Server/Workstation GC background threads can crash with an access violation inside coreclr!memcpy, called from vxsort code during mark_phase/plan_phase sorting of the GC mark list. The issue was first reported in May 2023, but we were never able to reproduce it. Now a customer started to hit this issue on a regular basis in Azure App Service powering a multi-tenant SaaS solution. It resulted in SLA violations impacting the customer. The culprit is a bug in the vxsort that has been in vxsort code ever since it was implemented in .NET Core 6 ## Regression - [x] Yes, introduced in .NET 6 in #37159 - [ ] No ## Testing coreclr tests executed with server GC with GC stress C ## Risk Low, the change is local to boundary-lane classification in vxsort and preserves the existing sorting and buffer-management logic. It forces out-of-range lanes onto the side from which they are subsequently trimmed, preventing invalid copy sizes and contamination of the sorted output Co-authored-by: Jan Vorlicek <janvorli@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b029aa4e-2a19-4582-b9e5-4918cb4c3095

janvorli requested a review from kkokosa September 29, 2026 21:10

janvorli self-assigned this Sep 29, 2026

janvorli added the area-GC-coreclr label Sep 29, 2026

This was referenced Oct 1, 2026

System.Net.Quic.QuicException : Connection handshake was canceled due to the configured timeout #133645

Open

[linux-bionic] TimeZoneInfoTests.IsIanaIdTest fails - GMT has wrong IANA Id indicator #133999

Open

ReubenBond mentioned this pull request Oct 1, 2026

test: investigate Windows net10 Serialization test host crash dotnet/orleans#11175

Open

kkokosa approved these changes Oct 1, 2026

View reviewed changes

janvorli merged commit 9110af0 into dotnet:main Oct 1, 2026
109 of 111 checks passed

janvorli deleted the fix-vxsort-align-junk-lane-popcount branch October 1, 2026 13:23

dotnet-maestro Bot mentioned this pull request Oct 2, 2026

[main] Source code updates from dotnet/runtime dotnet/dotnet#9905

Merged

dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Oct 2, 2026

github-actions Bot mentioned this pull request Oct 5, 2026

[release/11.0] Fix negative tmp-buffer size in vxsort align_vectorized #135229

Merged

2 of 4 tasks

janvorli mentioned this pull request Oct 5, 2026

[release/10.0] Fix negative tmp-buffer size in vxsort align_vectorized #135230

Merged

2 of 4 tasks

Navigation Menu

Repository navigation

Fix negative tmp-buffer size in vxsort align_vectorized - #134881

Fix negative tmp-buffer size in vxsort align_vectorized#134881
janvorli merged 2 commits into
dotnet:mainfrom
janvorli:fix-vxsort-align-junk-lane-popcount

janvorli commented Sep 29, 2026

Uh oh!

janvorli commented Sep 29, 2026

Uh oh!

azure-pipelines Bot commented Sep 29, 2026

Uh oh!

dotnet-policy-service Bot commented Sep 29, 2026

Uh oh!

Uh oh!

janvorli commented Oct 5, 2026

Uh oh!

janvorli commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

janvorli commented Sep 29, 2026

Issue

Root cause

Fix

Uh oh!

janvorli commented Sep 29, 2026

Uh oh!

azure-pipelines Bot commented Sep 29, 2026

Uh oh!

dotnet-policy-service Bot commented Sep 29, 2026

Uh oh!

Uh oh!

janvorli commented Oct 5, 2026

Uh oh!

janvorli commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants