[HTTP/3] SIGABRT in stress tests

[CarnaViire]

Today's stress test run crashed with segmentation fault after 28 mins https://dev.azure.com/dnceng/public/_build/results?buildId=1897576&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=1570

Funny enough, this run was scheduled with this ed5aa3d commit on top 😄 @rzikm (might be totally unrelated)

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Today's stress test run crashed with segmentation fault after 28 mins https://dev.azure.com/dnceng/public/_build/results?buildId=1897576&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=1570

Funny enough, this run was scheduled with this ed5aa3d commit on top 😄 @rzikm (might be totally unrelated)

Author:	CarnaViire
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[ManickaP]

I'm not able to reproduce it and the crash dumps from the pipeline are useless since the native is in release build.
I'm keeping the draft PR open if anyone want to have a go on this. It builds msquic in Debug.

[karelz]

@CarnaViire can you please write down when was the first hit, and how often it happens / happened?

[CarnaViire]

It turned out that only the first occurrence was a segfault (exit code 139), all others are sigabrt (exit code 134). So it is still happening and it was not fixed by Mana's copying.

7/18-8/9 we've had ~32 not-crashing runs (30 min) and 5 6 crashing runs most of them gathered around 7/22 (see below)

Date	Link	Exit code	Crashing after
7/22	Run #20220722.1	139	28 min
7/22	Run #20220722.3	134	1 min
7/23	Run #20220723.2	134	2 min
8/3	Run #20220803.6	134	8 min
8/5	Run #20220805.1	134	13 min
8/9	Run #20220809.3	134	7 min

UPD: new occurrences since 8/9

Date	Link	Exit code	Crashing after
8/12	Run #20220812.4	134	23 min
8/12	Run #20220812.7	134	13 min

[wfurt]

sigabrt is likely coming from Assert or unhandled exception. That should be visible IMHO from the dump.

[CarnaViire]

Abort is coming from MsQuic assert in MsQuicStreamSend https://github.com/microsoft/msquic/blob/d50ee4b831fca4f057031c50d0b31a7484e7208b/src/core/api.c#L1089

It seems like pthread_mutex_lock returned a non-zero exit code https://github.com/microsoft/msquic/blob/a7e98a6bbc609efb5fb2f7ef484827a0f453e816/src/inc/quic_platform_posix.h#L346