Repository navigation
Closed
Description
Hit in #31991 on Libraries Test Run release coreclr Windows_NT x86 Release:
Stacktrace of the crash from https://helix.dot.net/api/2019-06-17/jobs/55fc485f-ed44-4531-8009-fb6441da25d1/workitems/System.Net.HttpListener.Tests/files/dotnet.exe.1880.dmp :
0c 0739f48c 775efad1 ntdll!RtlpReportHeapFailure+0x21 [d:\w7rtm\minkernel\ntos\rtl\heaplog.c @ 159]
0d 0739f4c0 7759d97c ntdll!RtlpLogHeapFailure+0xa1 [d:\w7rtm\minkernel\ntos\rtl\heaplog.c @ 679]
0e 0739f4f0 76434fde ntdll!RtlFreeHeap+0x64 [d:\w7rtm\minkernel\ntos\rtl\heap.c @ 1887]
0f 0739f538 71e3b3bc KERNELBASE!GlobalFree+0x27 [d:\w7rtm\minkernel\kernelbase\gmem.c @ 153]
10 0739f580 71f6b3ea Interop+Kernel32.LocalFree(IntPtr)+0x28
11 0739f58c 04d23508 System_Private_CoreLib!System.Runtime.InteropServices.Marshal.FreeHGlobal(IntPtr)+0x1a
12 0739f59c 056af618 System_Net_HttpListener!System.Net.RequestContextBase.Dispose(Boolean)+0x28
13 0739f5ac 056af5aa System_Net_HttpListener!System.Net.AsyncRequestContext.Dispose(Boolean)+0x50
14 0739f5b8 056af578 System_Net_HttpListener!System.Net.RequestContextBase.Dispose()+0x1a
15 0739f5c4 044ace49 System_Net_HttpListener!System.Net.RequestContextBase.Close()+0x10
16 0739f5d4 044acd32 System_Net_HttpListener!System.Net.HttpListenerRequest.Close()+0x41
17 0739f61c 05bed744 System_Net_HttpListener!System.Net.HttpListenerContext.Close()+0xaa
18 0739f64c 056a3416 System_Net_HttpListener!System.Net.HttpListenerResponse.NonBlockingCloseCallback(System.IAsyncResult)+0x9c
19 0739f6a0 056a328a System_Net_HttpListener!System.Net.LazyAsyncResult.Complete(IntPtr)+0x16e
1a 0739f6e0 05bed68e System_Net_HttpListener!System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)+0x11a
1b 0739f710 05bed351 System_Net_HttpListener!System.Net.HttpResponseStreamAsyncResult.IOCompleted(System.Net.HttpResponseStreamAsyncResult, UInt32, UInt32)+0x1ae
1c 0739f738 71f0fdfa System_Net_HttpListener!System.Net.HttpResponseStreamAsyncResult.Callback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x111
1d 0739f750 71f10553 System_Private_CoreLib!System.Threading.ThreadPoolBoundHandleOverlapped.CompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x3a
1e 0739f76c 72460a6f System_Private_CoreLib!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x37
Activity
ManickaP commentedon Feb 19, 2020
Does this error from helix mean that the dump has been already deleted?
BTW, I'm signed into helix.
ManickaP commentedon Feb 21, 2020
Notes: might be related to #30284 and fix for this issue: #25497
ManickaP commentedon Feb 26, 2020
The dump I've available has a heap corruption in a different place, but might be caused by the same problem. I don't know if there's something more I could get from the dump or if any of this info is anyhow helpful.
I've also tried to run the
HttpListenertest under WinDbg with PageHeap on and wasn't able to reproduce the issue.Details:
jkotas commentedon Feb 26, 2020
Was it Windows 7 like the one at the top, or newer Windows version?
ManickaP commentedon Feb 26, 2020
The Helix Queue is: windows.7.amd64.open, also there were no other crashes in this build (according to Kusto) and it was on master.
Details: https://helix.dot.net/api/2019-06-17/jobs/9fdff4d5-5f76-43c1-bef7-ab86188e7a53/details
jkotas commentedon Feb 26, 2020
Have you tried with page heap on Windows 7? It may be Windows7 specific bug.
ManickaP commentedon Feb 26, 2020
I haven't, though I will, good idea 👍
ManickaP commentedon Mar 3, 2020
I was able to successfully reproduce the issue on Win7 VM with PageHeap. I was able to isolate the offending test and reproduce the same exception stack:
The same repro doesn't fail on my Win 10 machine (running the test in a loop for many hours).
ManickaP commentedon Mar 3, 2020
The failing test is
HttpListenerResponseTests.CloseResponseEntity_ChunkedSentHeaders_DoesNotModifyContentLength, it fails only forwillBlock == false:The fail is caused by a race condition between
Disposeat the end of the using block callingHttpListenerContext.Close()andNonBlockingCloseCallback(passed to_responseStream.BeginWritein case of non-blockingHttpListenerResponse.Close(byte[])) calling the same method from thread-pool thread. ThisHttpListenerRequest.Close()method calls underneath_memoryBlob.Close()which ends up in calling theMarshal.FreeHGlobal. I don't actually understand how this can work on Win 10, the different timing of theNonBlockingCloseCallbackinvocation is the only thing I can think of.Anyway, I was able to fix this by ensuring that only one thread can dispose/free the
memoryBlob(viaInterlocked.Exchange) and the repro as well as the original test have been running successfully for over an hour. I'll put up a PR soon.Fixed heap corruption in HttpListener