[NativeAOT] Enable Reflection library tests in Linux-ARM64

[LakshanF]

This test hang in CI Linux-arm64 container without much debugging info and currently its disabled in our CI runs.

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This test hang in CI Linux-arm64 container without much debugging info and currently its disabled in our CI runs.

Author:	LakshanF
Assignees:	-
Labels:	area-Infrastructure-libraries, untriaged
Milestone:	-

[Beau-Gosse-dev]

@LakshanF or @MichalStrehovsky is this the same failure even though it happened in "System.Runtime.Tests"?

#71001

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71001-merge-3cee02da652d46f592/System.Runtime.Tests/1/console.4fc9eef5.log?helixlogtype=result

Edit, for posterity, since I think the helix logs get cleaned up after a bit:

Console log: 'System.Runtime.Tests' from job 3cee02da-652d-46f5-9273-6ff1dd0f2051 (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652 on ddvsotx2l063
running $HELIX_CORRELATION_PAYLOAD/scripts/5ae7405b2fe6414dba9dbc70978675a1/execute.sh in /ssd/work/B3100970/w/B47909EA/e max 2700 seconds

Output:
[BEGIN EXECUTION]
+ sudo python -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
WARNING: The directory '/home/helixbot/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: certifi==2020.6.20 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: ndg-httpsclient==0.5.1 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 2)) (0.5.1)
Requirement already satisfied: psutil==5.8.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 3)) (5.8.0)
Requirement already satisfied: pyasn1==0.4.5 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 4)) (0.4.5)
Requirement already satisfied: pyopenssl==19.0.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 5)) (19.0.0)
Requirement already satisfied: six>=1.11.0 in /usr/lib/python3/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 6)) (1.11.0)
Requirement already satisfied: cryptography<=3.4.6,>=2.8 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 8)) (3.4.6)
Requirement already satisfied: asn1crypto==0.24.0 in /usr/lib/python3/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 9)) (0.24.0)
Requirement already satisfied: idna==2.8 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 10)) (2.8)
Requirement already satisfied: cffi==1.14.5 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 11)) (1.14.5)
Requirement already satisfied: pycparser==2.19 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 12)) (2.19)
Requirement already satisfied: docker==3.7.2 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 14)) (3.7.2)
Requirement already satisfied: docker-pycreds==0.4.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 15)) (0.4.0)
Requirement already satisfied: websocket-client==0.56.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 16)) (0.56.0)
Requirement already satisfied: requests>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 18)) (2.27.1)
Requirement already satisfied: urllib3==1.24.3 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 19)) (1.24.3)
Requirement already satisfied: azure-storage-blob==12.5.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 21)) (12.5.0)
Requirement already satisfied: azure-core==1.7.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 22)) (1.7.0)
Requirement already satisfied: azure-devops==5.0.0b9 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 23)) (5.0.0b9)
Requirement already satisfied: msrest==0.6.10 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 24)) (0.6.10)
Requirement already satisfied: isodate==0.6.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 25)) (0.6.0)
Requirement already satisfied: requests-oauthlib==1.2.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 26)) (1.2.0)
Requirement already satisfied: oauthlib==3.1.0 in /usr/local/lib/python3.6/dist-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 27)) (3.1.0)
Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from requests>=2.21.0->-r /root/helix/scripts/runtime_python_requirements.txt (line 18)) (2.0.12)
+ export PYTHONPATH=/root/helix/scripts:
+ cd /root/helix/work/workitem/e
+ mkdir -p /home/helixbot/dotnetbuild/dumps/
+ /root/helix/work/correlation/scripts/5ae7405b2fe6414dba9dbc70978675a1/execute.sh
+ ./RunTests.sh --runtime-path /root/helix/work/correlation
----- start Mon Jun 20 16:14:34 UTC 2022 =============== To repro directly: =====================================================
pushd .
chmod +rwx System.Runtime.Tests ^&^& ./System.Runtime.Tests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -xml testResults.xml 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Running assembly:System.Runtime.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
[SKIP] System.Tests.StringGetHashCodeTests.GetHashCodeWithStringComparer_UseSameStringInTwoProcesses_ReturnsDifferentHashCodes
[SKIP] System.Reflection.Tests.MethodBodyTests.Test_MethodBody_ExceptionHandlingClause
[SKIP] System.Tests.ExceptionDataTests.ICollection_NonGeneric_CopyTo_NonZeroLowerBound
[SKIP] System.Tests.GCExtendedTests.GetGCMemoryInfo
[SKIP] System.Runtime.Tests.UseResourceKeysTest.ReturnsResourceKeyWhenFeatureSwitchIsEnabled
[SKIP] System.Runtime.Tests.UseResourceKeysTest.ReturnsResourceWhenFeatureSwitchIsDisabled
[SKIP] System.Tests.ArraySegment_Tests_string.IGenericSharedAPI_SerializeDeserialize
[SKIP] System.Tests.UnitySerializationHolderTests.UnitySerializationHolderWithAssemblySingleton

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 2700 seconds and was killed
['System.Runtime.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

[MichalStrehovsky]

@LakshanF or @MichalStrehovsky is this the same failure even though it happened in "System.Runtime.Tests"?

It's quite likely the same issue. It happens a bit less often with System.Runtime, but we've seen apparent hangs there too.

[eerhardt]

Happened in my PR -

• Use crypto.subtle for HMAC on Browser WASM #70745
• https://helix.dot.net/api/2019-06-17/jobs/cb5e6f23-6a92-4ed8-acb1-bdcaa36eaaba/workitems/System.Runtime.Tests/console

[Beau-Gosse-dev]

I'm still trying to reproduce this NativeAOT library test hang on Linux, but it's turning out to be very difficult for me. I was able to get reliable test runs (always passing) on Ubuntu 20.04 x64, but after looking at the logs, it looks like the hang is actually reproducing on Ubuntu 18.04 arm64. So I thought I'd try that OS/arch to see if I can actually hit the hang locally.

I saw that there is a docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652 being used in CI, but I am having a lot of issues with that. Is there a script or build file that I can use to set up the test environment in a container? I've tried to piece it together from the CI logs, but it's not straight forward how everything is getting set up in Helix.

I also just tried spinning up my own VM and setting it up with these commands, based partially on instructions here. Is there an issue with my setup steps below?

sudo bash -c "$(wget -O - https://apt.kitware.com/kitware-archive.sh)"
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"

sudo apt-get install -y cmake llvm-9 clang build-essential python curl git lldb-6.0 liblldb-6.0-dev libunwind8 libunwind8-dev gettext libicu-dev liblttng-ust-dev libssl-dev libnuma-dev libkrb5-dev zlib1g-dev ninja-build git

git clone https://github.com/dotnet/runtime.git
cd runtime

./build.sh clr.alljits+clr.tools+clr.nativeaotlibs+clr.nativeaotruntime+libs+libs.tests -rc Release -lc Release /p:TestNativeAot=true

When set up that way, it results in a Segmentation fault (core dumped) during build. How can I examine the seg fault deeper to see what the problem is? And why would my local VM be throwing it, but the same OS/Arch on Helix isn't. I probably have some dependency installed wrong, but I'm not sure which one.

109:5>Done Building Project "/home/ubuntu/code/runtime/src/libraries/System.Runtime.Caching/tests/System.Runtime.Caching.Tests.csproj" (default targets).
  65:6>IlcCompile:
         Segmentation fault (core dumped)
  65:6>/home/ubuntu/code/runtime/artifacts/bin/coreclr/Linux.arm64.Release/build/Microsoft.NETCore.Native.targets(270,5): error MSB3073: The command ""/home/ubuntu/code/runtime/artifacts/bin/coreclr/Linux.arm64.Release/ilc/ilc" @"/home/ubuntu/code/runtime/artifacts/obj/System.Linq.Expressions.Tests/Release/net7.0/native/System.Linq.Expressions.Tests.ilc.rsp"" exited with code 139. [/home/ubuntu/code/runtime/src/libraries/System.Linq.Expressions/tests/System.Linq.Expressions.Tests.csproj]
       Done executing task "Exec" -- FAILED.
  65:6>Done building target "IlcCompile" in project "System.Linq.Expressions.Tests.csproj" -- FAILED.
  65:6>Done Building Project "/home/ubuntu/code/runtime/src/libraries/System.Linq.Expressions/tests/System.Linq.Expressions.Tests.csproj" (default targets) -- FAILED.

[MichalStrehovsky]

The build machines do a crossbuild - the prereqs:ubuntu-18.04-helix-arm64v8 container is an x64 image that has an ARM64 cross rootfs. We build the product, including the tests, with that, and then the resulting binary is sent to execution on an ARM64 machine.

Doing the build on an ARM64 is not something the infra regularly does. It should work. Looks like there might be a bug. That one would be interesting to look at too. Could you re-run the failing command under a debugger? The command line is in the log you pasted and can be used verbatim.

Also make sure you update to the latest main.

I've submitted #71177 to try to get a crashdump from the infra. I'm not sure if I'll be successful. We basically need dotnet/dnceng#1216.

[Beau-Gosse-dev]

Thanks for the explanation @MichalStrehovsky. I tried to get a stack trace of the build seg fault (unrelated to this GitHub issue) but after syncing from master and building debug instead of release this time, something fixed it. I'll try again on a clean machine tomorrow and also try to run the tests a few times on ARM now that I have it built (without crossbuild)

EDIT: Found a way to reproduce reliably and opened this to track it: #71270

[MichalStrehovsky]

I got dump of the hang in System.Runtime.

There are 8 threads.

Thread 1

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae3907d7c)
    at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae3907d7c)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae3907d7c) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae3907d7c, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae3907d20,
    cond=0xaaaae3907d50) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae3907d50, mutex=0xaaaae3907d20) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacc59c6a4 in S_P_CoreLib_System_Threading_WaitSubsystem_ThreadWaitInfo__Wait ()
dotnet/arcade#6  0x0000aaaacc4cfcd0 in S_P_CoreLib_System_Threading_WaitHandle__WaitOneNoCheck ()
dotnet/arcade#7  0x0000aaaacbf60f1c in System_Runtime_Tests_SingleFileTestRunner__Main ()
dotnet/arcade#8  0x0000aaaacccdb3e8 in __managed__Main ()
dotnet/arcade#9  0x0000aaaacbdc1fb8 in main (argc=5, argv=0xffffdeacb258) at /__w/1/s/src/coreclr/nativeaot/Bootstrap/main.cpp:205

Thread 2

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f2f2c)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae38f2f2c) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae38f2f2c, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae38f2f30,
    cond=0xaaaae38f2f00) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae38f2f00, mutex=0xaaaae38f2f30) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacbe0cf38 in UnixEvent::Wait (this=0xaaaae38f2f00, milliseconds=4294967295)
    at /__w/1/s/src/coreclr/nativeaot/Runtime/unix/PalRedhawkUnix.cpp:246
dotnet/arcade#6  0x0000aaaacbdc2c30 in PalWaitForSingleObjectEx (arg1=0xaaaae38f2f2c, arg2=4294967295, arg3=0)
    at /__w/1/s/src/coreclr/nativeaot/Runtime/./PalRedhawkFunctions.h:161
dotnet/arcade#7  RhpWaitForFinalizerRequest () at /__w/1/s/src/coreclr/nativeaot/Runtime/FinalizerHelpers.cpp:149
dotnet/arcade#8  0x0000aaaacc5386b0 in S_P_CoreLib_System_Runtime___Finalizer__ProcessFinalizers ()
dotnet/arcade#9  0x0000aaaacbdc2afc in FinalizerStart (pContext=0xaaaae38f2ef0)
    at /__w/1/s/src/coreclr/nativeaot/Runtime/FinalizerHelpers.cpp:68
dotnet/arcade#10 0x0000ffff898dd5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
dotnet/arcade#11 0x0000ffff89945d1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 3

#0  __GI___libc_read (nbytes=1, buf=0xffff6891e690, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) bt
#0  __GI___libc_read (nbytes=1, buf=0xffff6891e690, fd=<optimized out>) at ../sysdeps/unix/sysv/linux/read.c:26
dotnet/arcade#1  __GI___libc_read (fd=<optimized out>, buf=0xffff6891e690, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
dotnet/arcade#2  0x0000aaaacbe25930 in SignalHandlerLoop (arg=<optimized out>)
    at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
dotnet/arcade#3  0x0000ffff898dd5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
dotnet/arcade#4  0x0000ffff89945d1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 4

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f9628)
    at ./nptl/futex-internal.c:57
57      ./nptl/futex-internal.c: No such file or directory.
(gdb) bt
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f9628)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae38f9628) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae38f9628, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae38f95d0,
    cond=0xaaaae38f9600) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae38f9600, mutex=0xaaaae38f95d0) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacc4c7104 in S_P_CoreLib_System_Threading_LowLevelLock__WaitAndAcquire ()
dotnet/arcade#6  0x0000aaaacc59d008 in S_P_CoreLib_System_Threading_WaitSubsystem_WaitableObject__Wait ()
dotnet/arcade#7  0x0000aaaacc4cfcd0 in S_P_CoreLib_System_Threading_WaitHandle__WaitOneNoCheck ()
dotnet/arcade#8  0x0000aaaacc3a2dbc in xunit_execution_dotnet_Xunit_Sdk_MessageBus__ReporterWorker ()
dotnet/arcade#9  0x0000aaaacc3b9f28 in xunit_execution_dotnet_Xunit_Sdk_XunitWorkerThread___c___QueueUserWorkItem_b__5_0 ()
dotnet/arcade#10 0x0000aaaacc4c65bc in S_P_CoreLib_System_Threading_ExecutionContext__RunInternal ()
dotnet/arcade#11 0x0000aaaacc4d57c8 in S_P_CoreLib_System_Threading_Tasks_Task__ExecuteWithThreadLocal ()
dotnet/arcade#12 0x0000aaaacc4c3718 in S_P_CoreLib_System_Threading_Thread__StartThread ()
dotnet/arcade#13 0x0000aaaacc4c3bc0 in S_P_CoreLib_System_Threading_Thread__ThreadEntryPoint ()
dotnet/arcade#14 0x0000ffff898dd5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
dotnet/arcade#15 0x0000ffff89945d1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 5

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f9628)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae38f9628) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae38f9628, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae38f95d0,
    cond=0xaaaae38f9600) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae38f9600, mutex=0xaaaae38f95d0) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacc4c7104 in S_P_CoreLib_System_Threading_LowLevelLock__WaitAndAcquire ()
dotnet/arcade#6  0x0000aaaacc4d2c48 in S_P_CoreLib_System_Threading_WaitSubsystem__SetEvent_0 ()
dotnet/arcade#7  0x0000aaaacc4c613c in S_P_CoreLib_System_Threading_EventWaitHandle__Set ()
dotnet/arcade#8  0x0000aaaacc3a2d3c in xunit_execution_dotnet_Xunit_Sdk_MessageBus__QueueMessage ()
dotnet/arcade#9  0x0000aaaaccbd8a1c in xunit_execution_dotnet_Xunit_Sdk_TestRunner_1__RunAsync_d__43<System___Canon>__MoveNext ()
dotnet/arcade#10 0x0000aaaacce59d74 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestRunner_1__RunAsync_d__43<System___Canon>> ()
dotnet/arcade#11 0x0000aaaaccbd1c20 in xunit_execution_dotnet_Xunit_Sdk_TestRunner_1<System___Canon>__RunAsync ()
dotnet/arcade#12 0x0000aaaacc3b7950 in xunit_execution_dotnet_Xunit_Sdk_XunitTheoryTestCaseRunner__RunTestAsync_d__11__MoveNext ()
dotnet/arcade#13 0x0000aaaaccd051e0 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_XunitTheoryTestCaseRunner__RunTestAsync_d__11> ()
dotnet/arcade#14 0x0000aaaacc39a174 in xunit_execution_dotnet_Xunit_Sdk_XunitTheoryTestCaseRunner__RunTestAsync ()
dotnet/arcade#15 0x0000aaaaccbd48a4 in xunit_execution_dotnet_Xunit_Sdk_TestCaseRunner_1__RunAsync_d__19<System___Canon>__MoveNext
    ()
dotnet/arcade#16 0x0000aaaacce58264 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestCaseRunner_1__RunAsync_d__19<System___Canon>> ()
dotnet/arcade#17 0x0000aaaaccbcfcec in xunit_execution_dotnet_Xunit_Sdk_TestCaseRunner_1<System___Canon>__RunAsync ()
dotnet/arcade#18 0x0000aaaaccbd84b0 in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunTestCasesAsync_d__32<System___Canon>__MoveNext ()
dotnet/arcade#19 0x0000aaaacce59a74 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunTestCasesAsync_d__32<System___Canon>> ()
dotnet/arcade#20 0x0000aaaaccbd1740 in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1<System___Canon>__RunTestCasesAsync ()
dotnet/arcade#21 0x0000aaaaccbd7e54 in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunAsync_d__31<System___Canon>__MoveNext
    ()
dotnet/arcade#22 0x0000aaaacce59784 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunAsync_d__31<System___Canon>> ()
dotnet/arcade#23 0x0000aaaaccbd16cc in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1<System___Canon>__RunAsync ()

Thread 6

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f9628)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae38f9628) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae38f9628, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae38f95d0,
    cond=0xaaaae38f9600) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae38f9600, mutex=0xaaaae38f95d0) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacc4c7104 in S_P_CoreLib_System_Threading_LowLevelLock__WaitAndAcquire ()
dotnet/arcade#6  0x0000aaaacc59d008 in S_P_CoreLib_System_Threading_WaitSubsystem_WaitableObject__Wait ()
dotnet/arcade#7  0x0000aaaacc4cfcd0 in S_P_CoreLib_System_Threading_WaitHandle__WaitOneNoCheck ()
dotnet/arcade#8  0x0000aaaacc4c1124 in S_P_CoreLib_System_Threading_Lock__TryAcquireContended ()
dotnet/arcade#9  0x0000aaaacc4c1c54 in S_P_CoreLib_System_Threading_Monitor__TryAcquireContended ()
dotnet/arcade#10 0x0000aaaacc4c19e8 in S_P_CoreLib_System_Threading_Monitor__Enter_0 ()
dotnet/arcade#11 0x0000aaaacc483f6c in S_P_CoreLib_System_SR__InternalGetResourceString ()
dotnet/arcade#12 0x0000aaaacc4844c8 in S_P_CoreLib_System_SR__GetResourceString ()
dotnet/arcade#13 0x0000aaaacc59de68 in S_P_CoreLib_System_Threading_WaitSubsystem_WaitableObject__SignalMutex ()
dotnet/arcade#14 0x0000aaaacc4d2ed4 in S_P_CoreLib_System_Threading_WaitSubsystem__ReleaseMutex_0 ()
dotnet/arcade#15 0x0000aaaacc4c834c in S_P_CoreLib_System_Threading_Mutex__ReleaseMutex ()
dotnet/arcade#16 0x0000aaaacbf55174 in xunit_assert_Xunit_Assert__RecordException ()
dotnet/arcade#17 0x0000aaaacce72760 in xunit_assert_Xunit_Assert__Throws<System___Canon> ()
dotnet/arcade#18 0x0000aaaacbf6547c in System_Runtime_Tests_System_Threading_Tests_WaitHandleTests__VerifySignaled ()
dotnet/arcade#19 0x0000aaaacbf655e4 in System_Runtime_Tests_System_Threading_Tests_WaitHandleTests__SignalAndWait ()
dotnet/arcade#20 0x0000aaaacceecebc in Internal_CompilerGenerated__Module___InvokeRetVIIII<System___Canon__System___Canon__System___Canon__Bool> ()
dotnet/arcade#21 0x0000aaaacc4161fc in S_P_CoreLib_System_InvokeUtils__CallDynamicInvokeMethod ()
dotnet/arcade#22 0x0000aaaacc7da400 in S_P_Reflection_Execution_Internal_Reflection_Execution_MethodInvokers_StaticMethodInvoker__Invoke ()
dotnet/arcade#23 0x0000aaaacc57aa94 in S_P_CoreLib_Internal_Reflection_Core_Execution_MethodInvoker__Invoke ()
dotnet/arcade#24 0x0000aaaacc560a0c in S_P_CoreLib_System_Reflection_Runtime_MethodInfos_RuntimeMethodInfo__Invoke ()
dotnet/arcade#25 0x0000aaaaccbd9718 in xunit_execution_dotnet_Xunit_Sdk_TestInvoker_1___c__DisplayClass48_0___InvokeTestMethodAsync_b__1_d<System___Canon>__MoveNext ()
dotnet/arcade#26 0x0000aaaacce5a074 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestInvoker_1___c__DisplayClass48_0___InvokeTestMethodAsync_b__1_d<System___Canon>> ()
dotnet/arcade#27 0x0000aaaaccbd767c in xunit_execution_dotnet_Xunit_Sdk_TestInvoker_1___c__DisplayClass48_0<System___Canon>___InvokeTestMethodAsync_b__1 ()
dotnet/arcade#28 0x0000aaaacc3b33a8 in xunit_execution_dotnet_Xunit_Sdk_ExecutionTimer__AggregateAsync_d__4__MoveNext ()

Thread 7

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f9628)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae38f9628) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae38f9628, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae38f95d0,
    cond=0xaaaae38f9600) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae38f9600, mutex=0xaaaae38f95d0) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacc4c7104 in S_P_CoreLib_System_Threading_LowLevelLock__WaitAndAcquire ()
dotnet/arcade#6  0x0000aaaacc4d2c48 in S_P_CoreLib_System_Threading_WaitSubsystem__SetEvent_0 ()
dotnet/arcade#7  0x0000aaaacc4c613c in S_P_CoreLib_System_Threading_EventWaitHandle__Set ()
dotnet/arcade#8  0x0000aaaacc3a2d3c in xunit_execution_dotnet_Xunit_Sdk_MessageBus__QueueMessage ()
dotnet/arcade#9  0x0000aaaaccbd8a1c in xunit_execution_dotnet_Xunit_Sdk_TestRunner_1__RunAsync_d__43<System___Canon>__MoveNext ()
dotnet/arcade#10 0x0000aaaacce59d74 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestRunner_1__RunAsync_d__43<System___Canon>> ()
dotnet/arcade#11 0x0000aaaaccbd1c20 in xunit_execution_dotnet_Xunit_Sdk_TestRunner_1<System___Canon>__RunAsync ()
dotnet/arcade#12 0x0000aaaacc3b7950 in xunit_execution_dotnet_Xunit_Sdk_XunitTheoryTestCaseRunner__RunTestAsync_d__11__MoveNext ()
dotnet/arcade#13 0x0000aaaaccd051e0 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_XunitTheoryTestCaseRunner__RunTestAsync_d__11> ()
dotnet/arcade#14 0x0000aaaacc39a174 in xunit_execution_dotnet_Xunit_Sdk_XunitTheoryTestCaseRunner__RunTestAsync ()
dotnet/arcade#15 0x0000aaaaccbd48a4 in xunit_execution_dotnet_Xunit_Sdk_TestCaseRunner_1__RunAsync_d__19<System___Canon>__MoveNext
    ()
dotnet/arcade#16 0x0000aaaacce58264 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestCaseRunner_1__RunAsync_d__19<System___Canon>> ()
dotnet/arcade#17 0x0000aaaaccbcfcec in xunit_execution_dotnet_Xunit_Sdk_TestCaseRunner_1<System___Canon>__RunAsync ()
dotnet/arcade#18 0x0000aaaaccbd84b0 in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunTestCasesAsync_d__32<System___Canon>__MoveNext ()
dotnet/arcade#19 0x0000aaaacce59a74 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunTestCasesAsync_d__32<System___Canon>> ()
dotnet/arcade#20 0x0000aaaaccbd1740 in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1<System___Canon>__RunTestCasesAsync ()
dotnet/arcade#21 0x0000aaaaccbd7e54 in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunAsync_d__31<System___Canon>__MoveNext
    ()
dotnet/arcade#22 0x0000aaaacce59784 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1__RunAsync_d__31<System___Canon>> ()
dotnet/arcade#23 0x0000aaaaccbd16cc in xunit_execution_dotnet_Xunit_Sdk_TestMethodRunner_1<System___Canon>__RunAsync ()
dotnet/arcade#24 0x0000aaaaccbd5a4c in xunit_execution_dotnet_Xunit_Sdk_TestClassRunner_1__RunTestMethodsAsync_d__38<System___Canon>__MoveNext ()

Thread 8

#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0xaaaae38f9628)
    at ./nptl/futex-internal.c:57
dotnet/arcade#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0,
    futex_word=0xaaaae38f9628) at ./nptl/futex-internal.c:87
dotnet/arcade#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0xaaaae38f9628, expected=expected@entry=0,
    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
dotnet/arcade#3  0x0000ffff898dc8fc in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae38f95d0,
    cond=0xaaaae38f9600) at ./nptl/pthread_cond_wait.c:503
dotnet/arcade#4  ___pthread_cond_wait (cond=0xaaaae38f9600, mutex=0xaaaae38f95d0) at ./nptl/pthread_cond_wait.c:627
dotnet/arcade#5  0x0000aaaacc4c7104 in S_P_CoreLib_System_Threading_LowLevelLock__WaitAndAcquire ()
dotnet/arcade#6  0x0000aaaacc4d2c48 in S_P_CoreLib_System_Threading_WaitSubsystem__SetEvent_0 ()
dotnet/arcade#7  0x0000aaaacc4c613c in S_P_CoreLib_System_Threading_EventWaitHandle__Set ()
dotnet/arcade#8  0x0000aaaacc4c1308 in S_P_CoreLib_System_Threading_Lock__ReleaseContended ()
dotnet/arcade#9  0x0000aaaacc4c1280 in S_P_CoreLib_System_Threading_Lock__Release ()
dotnet/arcade#10 0x0000aaaacc484400 in S_P_CoreLib_System_SR__InternalGetResourceString ()
dotnet/arcade#11 0x0000aaaacc4844c8 in S_P_CoreLib_System_SR__GetResourceString ()
dotnet/arcade#12 0x0000aaaacc47b6a0 in S_P_CoreLib_System_Number__GetExceptionUInt128 ()
dotnet/arcade#13 0x0000aaaacc47b38c in S_P_CoreLib_System_Number__ThrowOverflowOrFormatExceptionUInt128 ()
dotnet/arcade#14 0x0000aaaacc33e240 in System_Runtime_Tests_System_Tests_UInt128Tests___c__DisplayClass16_0___Parse_Invalid_b__1 ()
dotnet/arcade#15 0x0000aaaacbf551f4 in xunit_assert_Xunit_Assert__RecordException_0 ()
dotnet/arcade#16 0x0000aaaacc09fcc4 in System_Runtime_Tests_System_Tests_UInt128Tests__Parse_Invalid ()
dotnet/arcade#17 0x0000aaaaccebd41c in Internal_CompilerGenerated__Module___InvokeRetVIIII<System___Canon__Int32__System___Canon__System___Canon> ()
dotnet/arcade#18 0x0000aaaacc4161fc in S_P_CoreLib_System_InvokeUtils__CallDynamicInvokeMethod ()
dotnet/arcade#19 0x0000aaaacc7da400 in S_P_Reflection_Execution_Internal_Reflection_Execution_MethodInvokers_StaticMethodInvoker__Invoke ()
dotnet/arcade#20 0x0000aaaacc57aa94 in S_P_CoreLib_Internal_Reflection_Core_Execution_MethodInvoker__Invoke ()
dotnet/arcade#21 0x0000aaaacc560a0c in S_P_CoreLib_System_Reflection_Runtime_MethodInfos_RuntimeMethodInfo__Invoke ()
dotnet/arcade#22 0x0000aaaaccbd9718 in xunit_execution_dotnet_Xunit_Sdk_TestInvoker_1___c__DisplayClass48_0___InvokeTestMethodAsync_b__1_d<System___Canon>__MoveNext ()
dotnet/arcade#23 0x0000aaaacce5a074 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_TestInvoker_1___c__DisplayClass48_0___InvokeTestMethodAsync_b__1_d<System___Canon>> ()
dotnet/arcade#24 0x0000aaaaccbd767c in xunit_execution_dotnet_Xunit_Sdk_TestInvoker_1___c__DisplayClass48_0<System___Canon>___InvokeTestMethodAsync_b__1 ()
dotnet/arcade#25 0x0000aaaacc3b33a8 in xunit_execution_dotnet_Xunit_Sdk_ExecutionTimer__AggregateAsync_d__4__MoveNext ()
dotnet/arcade#26 0x0000aaaaccd03440 in S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore__Start<xunit_execution_dotnet_Xunit_Sdk_ExecutionTimer__AggregateAsync_d__4> ()

The interesting threads are thread 6 and thread 8.

Both of them are trying to get a resource string. Thread 8 already got the string and now is trying to release a lock and calls into the wait subsystem. Apparently we need to acquire s_lock to do that. Thread 6 is holding the s_lock and trying to get a resource string because this was one of the tests that are trying to do bogus things and we need to throw an exception with a message.

Unfortunately, the System.Reflection.Tests hang will likely be something different because we don't test bogus operations with threading primitives there, but I might be mistaken.

I'm not currently looking into this more, so this is up for grabs. We could temporarily work around by disabling the tests that do bogus things to threading primitives, but I don't have a good sense on how to find them.

Cc @dotnet/ilc-contrib

[VSadov]

I’d like to look at the locking/threading issue next week.
Perhaps we should disable the test for now.