Segmentation fault in LibraryImportGenerator.Unit.Tests

[adamsitnik]

Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1676597&view=results:

  Discovering: LibraryImportGenerator.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  LibraryImportGenerator.Unit.Tests (found 117 test cases)
  Starting:    LibraryImportGenerator.Unit.Tests (parallel test collections = on, max threads = 4)
    LibraryImportGenerator.UnitTests.Compiles.ValidateSnippetsWithMarshalType [SKIP]
      No current scenarios to test.
./RunTests.sh: line 168:    23 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig LibraryImportGenerator.Unit.Tests.runtimeconfig.json --depsfile LibraryImportGenerator.Unit.Tests.deps.json xunit.console.dll LibraryImportGenerator.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Mar 22 21:09:57 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

For more details please go to the console logs

@dotnet/interop-contrib

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1676597&view=results:

  Discovering: LibraryImportGenerator.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  LibraryImportGenerator.Unit.Tests (found 117 test cases)
  Starting:    LibraryImportGenerator.Unit.Tests (parallel test collections = on, max threads = 4)
    LibraryImportGenerator.UnitTests.Compiles.ValidateSnippetsWithMarshalType [SKIP]
      No current scenarios to test.
./RunTests.sh: line 168:    23 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig LibraryImportGenerator.Unit.Tests.runtimeconfig.json --depsfile LibraryImportGenerator.Unit.Tests.deps.json xunit.console.dll LibraryImportGenerator.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Mar 22 21:09:57 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

For more details please go to the console logs

@dotnet/interop-contrib

Author:	adamsitnik
Assignees:	-
Labels:	arch-arm64, area-System.Runtime.InteropServices, os-linux-musl
Milestone:	7.0.0

[danmoseley]

Looking at the last few they're either

Assert failure(PID 25 [0x00000019], Thread: 49 [0x0031]): m_alignpad == 0
    File: /__w/1/s/src/coreclr/vm/syncblk.cpp Line: 2952
    Image: /root/helix/work/correlation/dotnet

or

Assert failure(PID 24 [0x00000018], Thread: 34 [0x0022]): Assertion failed 'arg.IsTemp()' in 'System.Type:op_Equality(System.Type,System.Type):bool' during 'Optimize Valnum CSEs' (IL size 38; hash 0xa9f2805d; Tier1)

    File: /__w/1/s/src/coreclr/jit/morph.cpp Line: 3207
    Image: /root/helix/work/correlation/dotnet

They seemed to stop on 4/15 though (the 1st kind was last). @jkotas do you recall either of these getting fixed? I can't find them in some quick searching.

Execute: Web | Desktop | Web (Lens) | Desktop (SAW)

https://engsrvprod.kusto.windows.net/engineeringdata

WorkItems
| where FriendlyName == "LibraryImportGenerator.Unit.Tests"
//| where Queued > ago(1d)
| where Status == "BadExit"
| where ExitCode  == 134
| join Jobs on JobId
| project
  Queued,
  FriendlyName, ExitCode,
  ConsoleUri,
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  BuildId = tostring(parse_json(Properties).BuildId),
  QueueName, Source
| where Pipeline == "runtime"

[danmoseley]

There is also a Windows dump (no Unix dumps for some reason). It may be unrelated. This was 4/16 on Windows 10 x64. @AndyAyersMS is this something already fixed?

0:004> k
 # ChildEBP RetAddr  
00 0ed4cbf8 70f02174 coreclr!TerminateOnAssert+0x17 [D:\a\_work\1\s\src\coreclr\utilcode\debug.cpp @ 189]
01 0ed4ccd0 70f017a9 coreclr!_DbgBreakCheck+0x411 [D:\a\_work\1\s\src\coreclr\utilcode\debug.cpp @ 427]
02 0ed4cd40 70f01ad1 coreclr!_DbgBreakCheckNoThrow+0x51 [D:\a\_work\1\s\src\coreclr\utilcode\debug.cpp @ 534]
03 0ed4cddc 70bdd6c0 coreclr!DbgAssertDialog+0x20b [D:\a\_work\1\s\src\coreclr\utilcode\debug.cpp @ 695]
04 0ed4cec4 70b89e7c coreclr!CLRLastThrownObjectException::Validate+0x144 [D:\a\_work\1\s\src\coreclr\vm\clrex.cpp @ 2170]
05 0ed4f0a4 70b8bcc1 coreclr!TieredCompilationManager::CompileCodeVersion+0x251 [D:\a\_work\1\s\src\coreclr\vm\tieredcompilation.cpp @ 919]
06 0ed4f120 70b8aa28 coreclr!TieredCompilationManager::OptimizeMethod+0xc1 [D:\a\_work\1\s\src\coreclr\vm\tieredcompilation.cpp @ 877]
07 0ed4f314 70b89b1a coreclr!TieredCompilationManager::DoBackgroundWork+0x635 [D:\a\_work\1\s\src\coreclr\vm\tieredcompilation.cpp @ 763]
08 0ed4f3c4 70b89917 coreclr!TieredCompilationManager::BackgroundWorkerStart+0x1d7 [D:\a\_work\1\s\src\coreclr\vm\tieredcompilation.cpp @ 482]
09 0ed4f480 70b80b9f coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper1+0xb7 [D:\a\_work\1\s\src\coreclr\vm\tieredcompilation.cpp @ 432]
0a 0ed4f4f4 70b80c25 coreclr!ManagedThreadBase_DispatchInner+0x93 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7325]
0b 0ed4f5d8 70b82d7f coreclr!ManagedThreadBase_DispatchMiddle+0x68 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7369]
0c 0ed4f614 70b82e2a coreclr!``ManagedThreadBase_DispatchOuter'::`8'::__Body::Run'::`5'::__Body::Run+0x43 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7528]
0d 0ed4f660 70b80f98 coreclr!`ManagedThreadBase_DispatchOuter'::`8'::__Body::Run+0x5a [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7528]
0e 0ed4f6bc 70b8105e coreclr!ManagedThreadBase_DispatchOuter+0x7e [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7548]
0f 0ed4f73c 70b80860 coreclr!ManagedThreadBase_FullTransition+0x9c [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7573]
10 0ed4f750 70b8980d coreclr!ManagedThreadBase::KickOff+0x10 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7608]
11 0ed4f818 779162c4 coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper0+0x10d [D:\a\_work\1\s\src\coreclr\vm\tieredcompilation.cpp @ 414]
12 0ed4f82c 77a41b69 kernel32!BaseThreadInitThunk+0x24
13 0ed4f874 77a41b34 ntdll!__RtlUserThreadStart+0x2f
14 0ed4f884 00000000 ntdll!_RtlUserThreadStart+0x1b

[jkotas]

@jkotas do you recall either of these getting fixed?

• m_alignpad == 0 is a classic GC hole type of crash. I do not recall a fix going in in last week that would be fixing a GC hole.
• `Assertion failed 'arg.IsTemp()' is failure due to a problem in the PR (that was not merged yet).

There is also a Windows dump (no Unix dumps for some reason)

I do not see this dump. Also, the stacktrace does not look like anything else mentioned in this issue so far.

no Unix dumps for some reason

The source generator tests are running orders of magnitude more code than other libraries tests, so they will tend to hit intermittent runtime stress bugs more frequently.

The next action for this one should be to figure out why we are not getting dumps. We are not going to make progress of intermittent crashes like this without dumps.

[AndyAyersMS]

@AndyAyersMS is this something already fixed?

I don't know -- from the stack this seems to be related to the TC mechanism.

Looks like there is a windows dump for this merged PR: #67184 under https://dev.azure.com/dnceng/public/_build/results?buildId=1701571&view=ms.vss-test-web.build-test-results-tab

  Starting:    LibraryImportGenerator.Unit.Tests (parallel test collections = on, max threads = 2)
    LibraryImportGenerator.UnitTests.Compiles.ValidateSnippetsWithMarshalType [SKIP]
      No current scenarios to test.
Fatal error. Internal CLR error. (0x80131506)

[AndyAyersMS]

Updated query to see windows failures (-1234 is a cancelled job so excluded those)

WorkItems
| where FriendlyName == "LibraryImportGenerator.Unit.Tests"
//| where Queued > ago(1d)
| where Status == "BadExit"
| where ExitCode != -1234
| join Jobs on JobId
| project
  Queued,
  FriendlyName, ExitCode,
  ConsoleUri,
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  BuildId = tostring(parse_json(Properties).BuildId),
  QueueName, Source
| where Pipeline == "runtime"