Test failure tracing\\eventpipe\\pauseonstart\\pauseonstart\\pauseonstart.cmd

[VincentBu]

Run: runtime-coreclr outerloop 20210618.10

Failed test:

CoreCLR windows x86 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- tracing\\eventpipe\\pauseonstart\\pauseonstart\\pauseonstart.cmd

Error message:

Return code:      1
Raw output file:      C:\h\w\B1050997\w\A68508EA\uploads\Reports\tracing.eventpipe\pauseonstart\pauseonstart\pauseonstart.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\B1050997\p\corerun.exe"  pauseonstart.dll
0.0s: ::== Running test: TEST_RuntimeResumesExecutionWithCommand
0.1s: Server name is 'DOTNET_TRACE_TESTS_40e5ocet.e4a'
0.9s: running sub-process: C:\h\w\B1050997\p\corerun.exe C:\h\w\B1050997\w\A68508EA\e\tracing\eventpipe\pauseonstart\pauseonstart\pauseonstart.dll 0
2.7s: subprocess started: True
2.7s: subprocess PID: 7708
4.6s: { Magic=System.Byte[]; ClrInstanceId=b355e344-977d-4ad4-9912-0ddff2c8f280; ProcessId=7708; Unused=0; }
4.6s: Sent: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=1; Reserved=0 }; }
4.7s: received: { Header={ Magic=System.Byte[]; Size=24; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x00 0x00 0x00 0x00  ]}
4.7s: Sending 'exit' to subprocess stdin
5.8s: ================= Subprocess Exited =================
5.8s: SubProcess exited - Exit code: 0
5.8s: ----------------------------------------
5.8s: Subprocess stdout:
4.8s: Subprocess started!  Waiting for input...
4.8s: Received 'exit'.  Exiting...
5.8s: Subprocess stderr:
5.8s: ----------------------------------------
5.9s: Test passed: True
5.9s:
5.9s: ::== Running test: TEST_TracesHaveRelevantEvents
6.0s: Server name is 'DOTNET_TRACE_TESTS_g4xdact4.0zp'
6.0s: running sub-process: C:\h\w\B1050997\p\corerun.exe C:\h\w\B1050997\w\A68508EA\e\tracing\eventpipe\pauseonstart\pauseonstart\pauseonstart.dll 0
6.0s: subprocess started: True
6.0s: subprocess PID: 7436
6.0s: Standard Diagnostics Server connection not created yet -> try again in 100 ms
6.2s: { Magic=System.Byte[]; ClrInstanceId=2eaf57d3-57a4-4111-b9dd-b6fc312daf82; ProcessId=7436; Unused=0; }
6.2s: Starting EventPipeSession over standard connection
6.4s: Started EventPipeSession over standard connection with session id: 0x3412608
6.5s: Send ResumeRuntime Diagnostics IPC Command
6.5s: Sent: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=1; Reserved=0 }; }
6.5s: received: { Header={ Magic=System.Byte[]; Size=24; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x00 0x00 0x00 0x00  ]}
8.5s: Stopping EventPipeSession over standard connection
9.4s: Stopped EventPipeSession over standard connection
9.4s: Sending 'exit' to subprocess stdin
9.5s: ================= Subprocess Exited =================
9.5s: SubProcess exited - Exit code: 0
9.5s: ----------------------------------------
9.5s: Subprocess stdout:
1.5s: Subprocess started!  Waiting for input...
3.5s: Received 'exit'.  Exiting...
9.5s: Subprocess stderr:
9.5s: ----------------------------------------
10.5s: isStartupEventPresent: True
10.6s: Test passed: True
10.6s:
10.6s: ::== Running test: TEST_MultipleSessionsCanBeStartedWhilepaused
10.6s: Server name is 'DOTNET_TRACE_TESTS_xwv5m345.qzm'
10.6s: running sub-process: C:\h\w\B1050997\p\corerun.exe C:\h\w\B1050997\w\A68508EA\e\tracing\eventpipe\pauseonstart\pauseonstart\pauseonstart.dll 0
10.6s: subprocess started: True
10.6s: subprocess PID: 3760
10.6s: Standard Diagnostics Server connection not created yet -> try again in 100 ms
10.8s: Standard Diagnostics Server connection not created yet -> try again in 100 ms
11.1s: { Magic=System.Byte[]; ClrInstanceId=16260ea4-6d84-403f-b37d-01f47bf81d6c; ProcessId=3760; Unused=0; }
11.1s: Starting EventPipeSession over standard connection
11.1s: Started EventPipeSession over standard connection with session id: 0x2c923f8
11.1s: Starting EventPipeSession over standard connection
11.1s: Started EventPipeSession over standard connection with session id: 0x2c922f0
11.1s: Starting EventPipeSession over standard connection
11.1s: Started EventPipeSession over standard connection with session id: 0x2d62aa0
11.1s: Send


Stack trace
   at tracing_eventpipe._pauseonstart_pauseonstart_pauseonstart_._pauseonstart_pauseonstart_pauseonstart_cmd() in tracing.eventpipe.XUnitWrapper.dll:token 0x6000016+0x284

[tommcdon]

@josalem

[josalem]

Here is the actual error message:

 18.1s: Subprocess stdout: 
  1.4s: Subprocess started!  Waiting for input...
 18.1s: Subprocess stderr: 
  6.2s: Assert failure(PID 3760 [0x00000eb0], Thread: 4660 [0x1234]): Consistency check failed: AV in clr at this callstack:
  6.2s: ------
  6.2s: CORECLR! AllocMemTracker::~AllocMemTracker + 0xFE (0x70b35ed3)
  6.2s: CORECLR! IBCLoggerAwareAllocMemTracker::~IBCLoggerAwareAllocMemTracker + 0xDC (0x708ae031)
  6.2s: CORECLR! ClassLoader::DoIncrementalLoad + 0x437 (0x70676dba)
  6.2s: CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x894 (0x7067f870)
  6.2s: CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x19A (0x7067eab4)
  6.2s: CORECLR! ClassLoader::LoadConstructedTypeThrowing + 0x63D (0x7067b684)
  6.2s: CORECLR! ClassLoader::LoadCanonicalGenericInstantiation + 0x347 (0x706f2ded)
  6.2s: CORECLR! ClassLoader::CreateTypeHandleForNonCanonicalGenericInstantiation + 0x978 (0x706f0f67)
  6.2s: CORECLR! ClassLoader::CreateTypeHandleForTypeKey + 0x30E (0x70676022)
  6.2s: CORECLR! ClassLoader::DoIncrementalLoad + 0x376 (0x70676cf9)
  6.2s: CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x894 (0x7067f870)
  6.2s: CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x19A (0x7067eab4)
  6.2s: CORECLR! ClassLoader::LoadConstructedTypeThrowing + 0x63D (0x7067b684)
  6.2s: CORECLR! ClassLoader::LoadGenericInstantiationThrowing + 0x4DF (0x7067c02d)
  6.2s: CORECLR! SigPointer::GetTypeHandleThrowing + 0xE1F (0x707ad27a)
  6.2s: CORECLR! ZapSig::DecodeType + 0xC9 (0x7080a600)
  6.2s: CORECLR! LoadDynamicInfoEntry + 0x67B (0x7070d03b)
  6.2s: CORECLR! Module::FixupNativeEntry + 0x1A0 (0x706466e0)
  6.2s: CORECLR! Module::FixupDelayListAux<Module *,int (__thiscall Module::*)(CORCOMPILE_IMPORT_SECTION *,unsigned long,unsigned long *)> + 0x34A (0x7080b7e9)
  6.2s: -----
  6.2s: .AV on tid=0x1234 (4660), cxr=04A9BF58, exr=04A9BF08
  6.2s: FAILED: false
  6.2s: CORECLR! CHECK::Trigger + 0x310 (0x70b3387c)
  6.2s: CORECLR! CLRVectoredExceptionHandlerPhase3 + 0x252 (0x706c4a2f)
  6.2s: CORECLR! CLRVectoredExceptionHandlerPhase2 + 0x4E (0x706c46aa)
  6.2s: CORECLR! CLRVectoredExceptionHandler + 0x13C (0x706c4650)
  6.2s: CORECLR! CLRVectoredExceptionHandlerShim + 0x172 (0x706c4c72)
  6.2s: NTDLL! LdrSetAppCompatDllRedirectionCallback + 0x1C21A (0x776af94a)
  6.2s: NTDLL! RtlUnwind + 0x1BA (0x7767011a)
  6.2s: NTDLL! KiUserExceptionDispatcher + 0xF (0x77680dbf)
  6.2s: CORECLR! IBCLoggerAwareAllocMemTracker::~IBCLoggerAwareAllocMemTracker + 0xDC (0x708ae031)
  6.2s: CORECLR! ClassLoader::DoIncrementalLoad + 0x437 (0x70676dba)
  6.2s:     File: D:\workspace\_work\1\s\src\coreclr\vm\excep.cpp Line: 7502
  6.2s:     Image: C:\h\w\B1050997\p\corerun.exe

This looks to be some kind of AV in the class loader?

[tommcdon]

This looks to be some kind of AV in the class loader?

Adding @janvorli

[janvorli]

This looks like some kind of memory corruption. The crash happens in the following loop:

runtime/src/coreclr/utilcode/loaderheap.cpp

Lines 2168 to 2174 in 213600c

	AllocMemTrackerBlock *pBlock = m_pFirstBlock;
	while (pBlock != &m_FirstBlock)
	{
	AllocMemTrackerBlock *pNext = pBlock->m_pNext;
	delete pBlock;
	pBlock = pNext;
	}

The pBlock at the time of crash is NULL, but it is never supposed to be NULL, the last element in the list should have m_pNext pointing to the m_FirstBlock. The m_pNext is always assigned the m_pFirstBlock when a new block is added and the m_pFirstBlock is set to point to the newly added block. So there is no legit way of setting that to NULL.

Unfortunately it is not possible to see what the linked list looked like before the crash, as the loop we are crashing in destroys the blocks one by one, so the previous blocks leading to the one with the NULL m_pNext are already gone.

[josalem]

Interesting. Thanks, Jan.

Looks like the Helix run has dumps for this failure, so I'll try to look when I can get access to a Windows machine.

As far as I can tell, this has only happened in this run (according to the AzDO history).

[VincentBu]

Failed again in runtime-coreclr outerloop 20210628.5

Failed test:

CoreCLR windows x86 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- tracing\\eventpipe\\pauseonstart\\pauseonstart\\pauseonstart.cmd

Error message:

Return code:      1
Raw output file:      C:\h\w\A1A108BA\w\B19109D8\uploads\Reports\tracing.eventpipe\pauseonstart\pauseonstart\pauseonstart.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\A1A108BA\p\corerun.exe"  pauseonstart.dll
0.0s: ::== Running test: TEST_RuntimeResumesExecutionWithCommand
0.1s: Server name is 'DOTNET_TRACE_TESTS_sgyju0hw.jrx'
0.9s: running sub-process: C:\h\w\A1A108BA\p\corerun.exe C:\h\w\A1A108BA\w\B19109D8\e\tracing\eventpipe\pauseonstart\pauseonstart\pauseonstart.dll 0
1.7s: subprocess started: True
1.7s: subprocess PID: 3464
3.8s: { Magic=System.Byte[]; ClrInstanceId=5fcc55d8-e2c4-41a5-9a8d-05bcc370f502; ProcessId=3464; Unused=0; }
3.8s: Sent: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=1; Reserved=0 }; }
3.8s: received: { Header={ Magic=System.Byte[]; Size=24; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x00 0x00 0x00 0x00  ]}
3.8s: Sending 'exit' to subprocess stdin
4.8s: ================= Subprocess Exited =================
4.8s: SubProcess exited - Exit code: 0
4.8s: ----------------------------------------
4.8s: Subprocess stdout:
3.8s: Subprocess started!  Waiting for input...
3.8s: Received 'exit'.  Exiting...
4.8s: Subprocess stderr:
4.8s: ----------------------------------------
4.9s: Test passed: True
4.9s:
4.9s: ::== Running test: TEST_TracesHaveRelevantEvents
5.0s: Server name is 'DOTNET_TRACE_TESTS_wfcjbjtb.yk1'
5.0s: running sub-process: C:\h\w\A1A108BA\p\corerun.exe C:\h\w\A1A108BA\w\B19109D8\e\tracing\eventpipe\pauseonstart\pauseonstart\pauseonstart.dll 0
5.0s: subprocess started: True
5.0s: subprocess PID: 6184
5.0s: Standard Diagnostics Server connection not created yet -> try again in 100 ms
5.2s: { Magic=System.Byte[]; ClrInstanceId=2715c04d-d5a3-432b-9d96-65b5d521f8ef; ProcessId=6184; Unused=0; }
5.2s: Starting EventPipeSession over standard connection
5.3s: Started EventPipeSession over standard connection with session id: 0x3102390
5.4s: Send ResumeRuntime Diagnostics IPC Command
5.4s: Sent: { Header={ Magic=System.Byte[]; Size=20; CommandSet=4; CommandId=1; Reserved=0 }; }
5.4s: received: { Header={ Magic=System.Byte[]; Size=24; CommandSet=255; CommandId=0; Reserved=0 }; Payload=[ 0x00 0x00 0x00 0x00  ]}
7.4s: Stopping EventPipeSession over standard connection
8.4s: Stopped EventPipeSession over standard connection
8.4s: Sending 'exit' to subprocess stdin
8.4s: ================= Subprocess Exited =================
8.4s: SubProcess exited - Exit code: 0
8.4s: ----------------------------------------
8.4s: Subprocess stdout:
1.4s: Subprocess started!  Waiting for input...
3.4s: Received 'exit'.  Exiting...
8.4s: Subprocess stderr:
8.4s: ----------------------------------------
9.4s: isStartupEventPresent: True
9.4s: Test passed: True
9.4s:
9.4s: ::== Running test: TEST_MultipleSessionsCanBeStartedWhilepaused
9.4s: Server name is 'DOTNET_TRACE_TESTS_jbahk3ll.pki'
9.4s: running sub-process: C:\h\w\A1A108BA\p\corerun.exe C:\h\w\A1A108BA\w\B19109D8\e\tracing\eventpipe\pauseonstart\pauseonstart\pauseonstart.dll 0
9.4s: subprocess started: True
9.4s: subprocess PID: 7720
9.5s: Standard Diagnostics Server connection not created yet -> try again in 100 ms
9.7s: { Magic=System.Byte[]; ClrInstanceId=dc42f3f8-4df8-4b46-97ba-609e9227518a; ProcessId=7720; Unused=0; }
9.7s: Starting EventPipeSession over standard connection
9.7s: Started EventPipeSession over standard connection with session id: 0x3492128
9.7s: Starting EventPipeSession over standard connection
9.7s: Started EventPipeSession over standard connection with session id: 0x34922e0
9.7s: Starting EventPipeSession over standard connection
9.7s: Started EventPipeSession over standard connection with session id: 0x3563c88
9.7s: Send ResumeRuntime Diagnostics IPC Command
9.7s: Sent: { Header={ Magic=System.Byte[]; S


Stack trace
   at tracing_eventpipe._pauseonstart_pauseonstart_pauseonstart_._pauseonstart_pauseonstart_pauseonstart_cmd() in tracing.eventpipe.XUnitWrapper.dll:token 0x6000016+0x284