segfault when calling MarshalNative::GCHandleInternalGet in System.Net.Requests.Tests in rolling CI

[jakobbotsch]

Pipeline: https://dev.azure.com/dnceng/public/_build/results?buildId=1761915&view=results
Job: Libraries Test Run checked coreclr Linux x64 Release
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-c09b54c3fa68465384/System.Net.Requests.Tests/1/console.51c5c155.log?helixlogtype=result

/datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e /datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 348 of 367 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.FtpWebRequestTest.Ftp_AppendFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_LargeFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_CreateAndDelete [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFileSubDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_MakeAndRemoveDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
./RunTests.sh: line 168: 10507 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e
----- end Tue May 10 09:05:08 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited

The dump shows segfault in Object::ValidateInner as part of MarshalNative::GCHandleInternalGet:

* thread #1, name = 'dotnet', stop reason = signal SIGSEGV
  * frame #0: 0x00007fc745dff6ed libcoreclr.so`Object::ValidateInner(int, int, int) [inlined] Object::GetGCSafeMethodTable(this=0x0000000000000007) const at object.h:446:59
    frame #1: 0x00007fc745dff6ed libcoreclr.so`Object::ValidateInner(this=0x0000000000000007, bDeep=YES, bVerifyNextHeader=YES, bVerifySyncBlock=YES) at object.cpp:528
    frame #2: 0x00007fc745e0135d libcoreclr.so`OBJECTREF::OBJECTREF(Object*) [inlined] Object::Validate(this=0x0000000000000007, bDeep=YES, bVerifyNextHeader=YES, bVerifySyncBlock=YES) at object.cpp:508:9
    frame #3: 0x00007fc745e012b0 libcoreclr.so`OBJECTREF::OBJECTREF(this=0x00007f85fecf9760, pObject=0x0000000000000007) at object.cpp:1131
    frame #4: 0x00007fc745f70130 libcoreclr.so`MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) [inlined] ObjectFromHandle(handle=0x00007fc7476e6a40) at gchandleutilities.h:42:24
    frame #5: 0x00007fc745f70104 libcoreclr.so`MarshalNative::GCHandleInternalGet(handle=0x00007fc7476e6a40) at marshalnative.cpp:534
    frame #6: 0x00007fc6cf648f26
    frame #7: 0x00007fc6c8310356 libssl.so.1.1`SSL_set_fd + 86

Top frames of dumpstack:

(lldb) dumpstack
OS Thread Id: 0x296f (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libcoreclr.so!Object::ValidateInner(int, int, int) + 0x1ad [/__w/1/s/src/coreclr/vm/object.h:446]
Child-SP         RetAddr          Caller, Callee
00007F85FECF9710 00007fc745e0135d libcoreclr.so!OBJECTREF::OBJECTREF(Object*) + 0x11d [/__w/1/s/src/coreclr/vm/object.cpp:1132], calling libcoreclr.so!Object::ValidateInner(int, int, int) [/__w/1/s/src/coreclr/vm/object.cpp:513]
00007F85FECF9740 00007fc745f70130 libcoreclr.so!MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) + 0x50 [/__w/1/s/src/coreclr/vm/gchandleutilities.h:44], calling libcoreclr.so!OBJECTREF::OBJECTREF(Object*) [/__w/1/s/src/coreclr/vm/object.cpp:1117]
00007F85FECF9780 00007fc6cf648f26 (MethodDesc 00007fc6cf5c4370 + 0x66 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling 00007fc745f700e0 (stub for System.Runtime.InteropServices.GCHandle.InternalGet(IntPtr))
00007F85FECF97B8 00007fc6cf648ef8 (MethodDesc 00007fc6cf5c4370 + 0x38 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling libcoreclr.so!JIT_PInvokeBegin [/__w/1/s/src/coreclr/pal/inc/unixasmmacrosamd64.inc:896]
00007F85FECF9820 00007fc6c8310356 libssl.so.1.1!SSL_set_fd + 0x56
00007F85FECF9850 00007fc6c8329131 libssl.so.1.1!___lldb_unnamed_symbol512$$libssl.so.1.1 + 0x3e1, calling libssl.so.1.1!SSL_get_rfd + 0x10
00007F85FECF98E0 00007fc6c832a9e5 libssl.so.1.1!___lldb_unnamed_symbol521$$libssl.so.1.1 + 0x6b5, calling libssl.so.1.1!___lldb_unnamed_symbol511$$libssl.so.1.1 + 0x170
00007F85FECF9990 00007fc7463675f7 libcoreclr.so!GetCurrentThreadId + 0x77 [/__w/1/s/src/coreclr/pal/src/include/pal/thread.hpp:781], calling libcoreclr.so!__tls_get_addr
00007F85FECF99F0 00007fc6c82fce70 libssl.so.1.1!___lldb_unnamed_symbol114$$libssl.so.1.1 + 0x130
00007F85FECF9A40 00007fc745d78788 libcoreclr.so!Frame::Pop(Thread*) + 0xf8 [/__w/1/s/src/coreclr/vm/frames.cpp:447], calling libcoreclr.so!Thread::SetFrame(Frame*) [/__w/1/s/src/coreclr/vm/threads.cpp:211]
00007F85FECF9A90 00007fc6c8303995 libssl.so.1.1!___lldb_unnamed_symbol152$$libssl.so.1.1 + 0xb45
00007F85FECF9AA0 00007fc6c8303950 libssl.so.1.1!___lldb_unnamed_symbol152$$libssl.so.1.1 + 0xb00, calling libssl.so.1.1!EC_KEY_get_conv_form
00007F85FECF9AE0 00007fc6c830e252 libssl.so.1.1!ERR_load_SSL_strings + 0x22
00007F85FECF9B00 00007f861c275cc9 libcrypto.so.1.1!___lldb_unnamed_symbol1922$$libcrypto.so.1.1 + 0xf9, calling libcrypto.so.1.1 + 0xffffffff
00007F85FECF9B10 00007f861c1ee2e3 libcrypto.so.1.1!ERR_add_error_data + 0x13, calling libcrypto.so.1.1!___lldb_unnamed_symbol1922$$libcrypto.so.1.1 + 0x130
00007F85FECF9B40 00007fc6c830e373 libssl.so.1.1!___lldb_unnamed_symbol287$$libssl.so.1.1 + 0xb3, calling libssl.so.1.1!SSL_CONF_CTX_set_ssl_ctx + 0x50
00007F85FECF9B60 00007fc6c858ca32 libSystem.Security.Cryptography.Native.OpenSsl.so!CryptoNative_SslRead + 0x32 [/__w/1/s/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c:458]
00007F85FECF9B90 00007fc6cf6408fc (MethodDesc 00007fc6cf8629b0 + 0x7c ILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9BD0 00007fc6cf6408fc (MethodDesc 00007fc6cf8629b0 + 0x7c ILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9C20 00007fc6cf644886 (MethodDesc 00007fc6cf5c4de8 + 0xa6 Interop+Ssl.SslRead(Microsoft.Win32.SafeHandles.SafeSslHandle, Byte ByRef, Int32, SslErrorCode ByRef)), calling 00007fc6cf3a4678 (stub for Interop+Ssl.<SslRead>g____PInvoke__|22_0(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9CA0 00007fc6cf644557 (MethodDesc 00007fc6cf5c4230 + 0xb7 Interop+OpenSsl.Decrypt(Microsoft.Win32.SafeHandles.SafeSslHandle, System.Span`1<Byte>, SslErrorCode ByRef)), calling 00007fc6cf614300
00007F85FECF9D40 00007fc6cf6441b1 (MethodDesc 00007fc6cf5a5d70 + 0x91 System.Net.Security.SslStreamPal.DecryptMessage(System.Net.Security.SafeDeleteSslContext, System.Span`1<Byte>, Int32 ByRef, Int32 ByRef)), calling 00007fc6cf6142b8

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Pipeline: https://dev.azure.com/dnceng/public/_build/results?buildId=1761915&view=results
Job: Libraries Test Run checked coreclr Linux x64 Release
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-c09b54c3fa68465384/System.Net.Requests.Tests/1/console.51c5c155.log?helixlogtype=result

/datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e /datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 348 of 367 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.FtpWebRequestTest.Ftp_AppendFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_LargeFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_CreateAndDelete [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFileSubDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_MakeAndRemoveDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
./RunTests.sh: line 168: 10507 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e
----- end Tue May 10 09:05:08 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited

The dump shows segfault in Object::ValidateInner as part of MarshalNative::GCHandleInternalGet:

* thread #1, name = 'dotnet', stop reason = signal SIGSEGV
  * frame #0: 0x00007fc745dff6ed libcoreclr.so`Object::ValidateInner(int, int, int) [inlined] Object::GetGCSafeMethodTable(this=0x0000000000000007) const at object.h:446:59
    frame #1: 0x00007fc745dff6ed libcoreclr.so`Object::ValidateInner(this=0x0000000000000007, bDeep=YES, bVerifyNextHeader=YES, bVerifySyncBlock=YES) at object.cpp:528
    frame #2: 0x00007fc745e0135d libcoreclr.so`OBJECTREF::OBJECTREF(Object*) [inlined] Object::Validate(this=0x0000000000000007, bDeep=YES, bVerifyNextHeader=YES, bVerifySyncBlock=YES) at object.cpp:508:9
    frame #3: 0x00007fc745e012b0 libcoreclr.so`OBJECTREF::OBJECTREF(this=0x00007f85fecf9760, pObject=0x0000000000000007) at object.cpp:1131
    frame #4: 0x00007fc745f70130 libcoreclr.so`MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) [inlined] ObjectFromHandle(handle=0x00007fc7476e6a40) at gchandleutilities.h:42:24
    frame #5: 0x00007fc745f70104 libcoreclr.so`MarshalNative::GCHandleInternalGet(handle=0x00007fc7476e6a40) at marshalnative.cpp:534
    frame #6: 0x00007fc6cf648f26
    frame #7: 0x00007fc6c8310356 libssl.so.1.1`SSL_set_fd + 86

Top frames of dumpstack:

(lldb) dumpstack
OS Thread Id: 0x296f (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libcoreclr.so!Object::ValidateInner(int, int, int) + 0x1ad [/__w/1/s/src/coreclr/vm/object.h:446]
Child-SP         RetAddr          Caller, Callee
00007F85FECF9710 00007fc745e0135d libcoreclr.so!OBJECTREF::OBJECTREF(Object*) + 0x11d [/__w/1/s/src/coreclr/vm/object.cpp:1132], calling libcoreclr.so!Object::ValidateInner(int, int, int) [/__w/1/s/src/coreclr/vm/object.cpp:513]
00007F85FECF9740 00007fc745f70130 libcoreclr.so!MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) + 0x50 [/__w/1/s/src/coreclr/vm/gchandleutilities.h:44], calling libcoreclr.so!OBJECTREF::OBJECTREF(Object*) [/__w/1/s/src/coreclr/vm/object.cpp:1117]
00007F85FECF9780 00007fc6cf648f26 (MethodDesc 00007fc6cf5c4370 + 0x66 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling 00007fc745f700e0 (stub for System.Runtime.InteropServices.GCHandle.InternalGet(IntPtr))
00007F85FECF97B8 00007fc6cf648ef8 (MethodDesc 00007fc6cf5c4370 + 0x38 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling libcoreclr.so!JIT_PInvokeBegin [/__w/1/s/src/coreclr/pal/inc/unixasmmacrosamd64.inc:896]
00007F85FECF9820 00007fc6c8310356 libssl.so.1.1!SSL_set_fd + 0x56
00007F85FECF9850 00007fc6c8329131 libssl.so.1.1!___lldb_unnamed_symbol512$$libssl.so.1.1 + 0x3e1, calling libssl.so.1.1!SSL_get_rfd + 0x10
00007F85FECF98E0 00007fc6c832a9e5 libssl.so.1.1!___lldb_unnamed_symbol521$$libssl.so.1.1 + 0x6b5, calling libssl.so.1.1!___lldb_unnamed_symbol511$$libssl.so.1.1 + 0x170
00007F85FECF9990 00007fc7463675f7 libcoreclr.so!GetCurrentThreadId + 0x77 [/__w/1/s/src/coreclr/pal/src/include/pal/thread.hpp:781], calling libcoreclr.so!__tls_get_addr
00007F85FECF99F0 00007fc6c82fce70 libssl.so.1.1!___lldb_unnamed_symbol114$$libssl.so.1.1 + 0x130
00007F85FECF9A40 00007fc745d78788 libcoreclr.so!Frame::Pop(Thread*) + 0xf8 [/__w/1/s/src/coreclr/vm/frames.cpp:447], calling libcoreclr.so!Thread::SetFrame(Frame*) [/__w/1/s/src/coreclr/vm/threads.cpp:211]
00007F85FECF9A90 00007fc6c8303995 libssl.so.1.1!___lldb_unnamed_symbol152$$libssl.so.1.1 + 0xb45
00007F85FECF9AA0 00007fc6c8303950 libssl.so.1.1!___lldb_unnamed_symbol152$$libssl.so.1.1 + 0xb00, calling libssl.so.1.1!EC_KEY_get_conv_form
00007F85FECF9AE0 00007fc6c830e252 libssl.so.1.1!ERR_load_SSL_strings + 0x22
00007F85FECF9B00 00007f861c275cc9 libcrypto.so.1.1!___lldb_unnamed_symbol1922$$libcrypto.so.1.1 + 0xf9, calling libcrypto.so.1.1 + 0xffffffff
00007F85FECF9B10 00007f861c1ee2e3 libcrypto.so.1.1!ERR_add_error_data + 0x13, calling libcrypto.so.1.1!___lldb_unnamed_symbol1922$$libcrypto.so.1.1 + 0x130
00007F85FECF9B40 00007fc6c830e373 libssl.so.1.1!___lldb_unnamed_symbol287$$libssl.so.1.1 + 0xb3, calling libssl.so.1.1!SSL_CONF_CTX_set_ssl_ctx + 0x50
00007F85FECF9B60 00007fc6c858ca32 libSystem.Security.Cryptography.Native.OpenSsl.so!CryptoNative_SslRead + 0x32 [/__w/1/s/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c:458]
00007F85FECF9B90 00007fc6cf6408fc (MethodDesc 00007fc6cf8629b0 + 0x7c ILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9BD0 00007fc6cf6408fc (MethodDesc 00007fc6cf8629b0 + 0x7c ILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9C20 00007fc6cf644886 (MethodDesc 00007fc6cf5c4de8 + 0xa6 Interop+Ssl.SslRead(Microsoft.Win32.SafeHandles.SafeSslHandle, Byte ByRef, Int32, SslErrorCode ByRef)), calling 00007fc6cf3a4678 (stub for Interop+Ssl.<SslRead>g____PInvoke__|22_0(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9CA0 00007fc6cf644557 (MethodDesc 00007fc6cf5c4230 + 0xb7 Interop+OpenSsl.Decrypt(Microsoft.Win32.SafeHandles.SafeSslHandle, System.Span`1<Byte>, SslErrorCode ByRef)), calling 00007fc6cf614300
00007F85FECF9D40 00007fc6cf6441b1 (MethodDesc 00007fc6cf5a5d70 + 0x91 System.Net.Security.SslStreamPal.DecryptMessage(System.Net.Security.SafeDeleteSslContext, System.Span`1<Byte>, Int32 ByRef, Int32 ByRef)), calling 00007fc6cf6142b8

Author:	jakobbotsch
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[jakobbotsch]

This looks like the same as #68037, so looks like that error is persistent.

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Pipeline: https://dev.azure.com/dnceng/public/_build/results?buildId=1761915&view=results
Job: Libraries Test Run checked coreclr Linux x64 Release
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-c09b54c3fa68465384/System.Net.Requests.Tests/1/console.51c5c155.log?helixlogtype=result

/datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e /datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 348 of 367 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.FtpWebRequestTest.Ftp_AppendFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_LargeFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_CreateAndDelete [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFileSubDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_MakeAndRemoveDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
./RunTests.sh: line 168: 10507 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/AA5E0919/w/BBDA0A1B/e
----- end Tue May 10 09:05:08 UTC 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited

The dump shows segfault in Object::ValidateInner as part of MarshalNative::GCHandleInternalGet:

* thread #1, name = 'dotnet', stop reason = signal SIGSEGV
  * frame #0: 0x00007fc745dff6ed libcoreclr.so`Object::ValidateInner(int, int, int) [inlined] Object::GetGCSafeMethodTable(this=0x0000000000000007) const at object.h:446:59
    frame #1: 0x00007fc745dff6ed libcoreclr.so`Object::ValidateInner(this=0x0000000000000007, bDeep=YES, bVerifyNextHeader=YES, bVerifySyncBlock=YES) at object.cpp:528
    frame #2: 0x00007fc745e0135d libcoreclr.so`OBJECTREF::OBJECTREF(Object*) [inlined] Object::Validate(this=0x0000000000000007, bDeep=YES, bVerifyNextHeader=YES, bVerifySyncBlock=YES) at object.cpp:508:9
    frame #3: 0x00007fc745e012b0 libcoreclr.so`OBJECTREF::OBJECTREF(this=0x00007f85fecf9760, pObject=0x0000000000000007) at object.cpp:1131
    frame #4: 0x00007fc745f70130 libcoreclr.so`MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) [inlined] ObjectFromHandle(handle=0x00007fc7476e6a40) at gchandleutilities.h:42:24
    frame #5: 0x00007fc745f70104 libcoreclr.so`MarshalNative::GCHandleInternalGet(handle=0x00007fc7476e6a40) at marshalnative.cpp:534
    frame #6: 0x00007fc6cf648f26
    frame #7: 0x00007fc6c8310356 libssl.so.1.1`SSL_set_fd + 86

Top frames of dumpstack:

(lldb) dumpstack
OS Thread Id: 0x296f (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libcoreclr.so!Object::ValidateInner(int, int, int) + 0x1ad [/__w/1/s/src/coreclr/vm/object.h:446]
Child-SP         RetAddr          Caller, Callee
00007F85FECF9710 00007fc745e0135d libcoreclr.so!OBJECTREF::OBJECTREF(Object*) + 0x11d [/__w/1/s/src/coreclr/vm/object.cpp:1132], calling libcoreclr.so!Object::ValidateInner(int, int, int) [/__w/1/s/src/coreclr/vm/object.cpp:513]
00007F85FECF9740 00007fc745f70130 libcoreclr.so!MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) + 0x50 [/__w/1/s/src/coreclr/vm/gchandleutilities.h:44], calling libcoreclr.so!OBJECTREF::OBJECTREF(Object*) [/__w/1/s/src/coreclr/vm/object.cpp:1117]
00007F85FECF9780 00007fc6cf648f26 (MethodDesc 00007fc6cf5c4370 + 0x66 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling 00007fc745f700e0 (stub for System.Runtime.InteropServices.GCHandle.InternalGet(IntPtr))
00007F85FECF97B8 00007fc6cf648ef8 (MethodDesc 00007fc6cf5c4370 + 0x38 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling libcoreclr.so!JIT_PInvokeBegin [/__w/1/s/src/coreclr/pal/inc/unixasmmacrosamd64.inc:896]
00007F85FECF9820 00007fc6c8310356 libssl.so.1.1!SSL_set_fd + 0x56
00007F85FECF9850 00007fc6c8329131 libssl.so.1.1!___lldb_unnamed_symbol512$$libssl.so.1.1 + 0x3e1, calling libssl.so.1.1!SSL_get_rfd + 0x10
00007F85FECF98E0 00007fc6c832a9e5 libssl.so.1.1!___lldb_unnamed_symbol521$$libssl.so.1.1 + 0x6b5, calling libssl.so.1.1!___lldb_unnamed_symbol511$$libssl.so.1.1 + 0x170
00007F85FECF9990 00007fc7463675f7 libcoreclr.so!GetCurrentThreadId + 0x77 [/__w/1/s/src/coreclr/pal/src/include/pal/thread.hpp:781], calling libcoreclr.so!__tls_get_addr
00007F85FECF99F0 00007fc6c82fce70 libssl.so.1.1!___lldb_unnamed_symbol114$$libssl.so.1.1 + 0x130
00007F85FECF9A40 00007fc745d78788 libcoreclr.so!Frame::Pop(Thread*) + 0xf8 [/__w/1/s/src/coreclr/vm/frames.cpp:447], calling libcoreclr.so!Thread::SetFrame(Frame*) [/__w/1/s/src/coreclr/vm/threads.cpp:211]
00007F85FECF9A90 00007fc6c8303995 libssl.so.1.1!___lldb_unnamed_symbol152$$libssl.so.1.1 + 0xb45
00007F85FECF9AA0 00007fc6c8303950 libssl.so.1.1!___lldb_unnamed_symbol152$$libssl.so.1.1 + 0xb00, calling libssl.so.1.1!EC_KEY_get_conv_form
00007F85FECF9AE0 00007fc6c830e252 libssl.so.1.1!ERR_load_SSL_strings + 0x22
00007F85FECF9B00 00007f861c275cc9 libcrypto.so.1.1!___lldb_unnamed_symbol1922$$libcrypto.so.1.1 + 0xf9, calling libcrypto.so.1.1 + 0xffffffff
00007F85FECF9B10 00007f861c1ee2e3 libcrypto.so.1.1!ERR_add_error_data + 0x13, calling libcrypto.so.1.1!___lldb_unnamed_symbol1922$$libcrypto.so.1.1 + 0x130
00007F85FECF9B40 00007fc6c830e373 libssl.so.1.1!___lldb_unnamed_symbol287$$libssl.so.1.1 + 0xb3, calling libssl.so.1.1!SSL_CONF_CTX_set_ssl_ctx + 0x50
00007F85FECF9B60 00007fc6c858ca32 libSystem.Security.Cryptography.Native.OpenSsl.so!CryptoNative_SslRead + 0x32 [/__w/1/s/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c:458]
00007F85FECF9B90 00007fc6cf6408fc (MethodDesc 00007fc6cf8629b0 + 0x7c ILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9BD0 00007fc6cf6408fc (MethodDesc 00007fc6cf8629b0 + 0x7c ILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9C20 00007fc6cf644886 (MethodDesc 00007fc6cf5c4de8 + 0xa6 Interop+Ssl.SslRead(Microsoft.Win32.SafeHandles.SafeSslHandle, Byte ByRef, Int32, SslErrorCode ByRef)), calling 00007fc6cf3a4678 (stub for Interop+Ssl.<SslRead>g____PInvoke__|22_0(IntPtr, Byte*, Int32, SslErrorCode*))
00007F85FECF9CA0 00007fc6cf644557 (MethodDesc 00007fc6cf5c4230 + 0xb7 Interop+OpenSsl.Decrypt(Microsoft.Win32.SafeHandles.SafeSslHandle, System.Span`1<Byte>, SslErrorCode ByRef)), calling 00007fc6cf614300
00007F85FECF9D40 00007fc6cf6441b1 (MethodDesc 00007fc6cf5a5d70 + 0x91 System.Net.Security.SslStreamPal.DecryptMessage(System.Net.Security.SafeDeleteSslContext, System.Span`1<Byte>, Int32 ByRef, Int32 ByRef)), calling 00007fc6cf6142b8

Author:	jakobbotsch
Assignees:	-
Labels:	area-System.Net, area-System.Net.Security, blocking-clean-ci, untriaged
Milestone:	-

[danmoseley]

Is it always ubuntu.1804? If so is it #48411 ? Essentially we know we have a potential segfault on 1804 and can't fix it.

If so we could disable tests tests for 1804, I guess. Assuming we have 20.04 in the matrix as well. cc @bartonjs

[rzikm]

Yes, it is always Ubuntu 18.04, and my fair guess it is related to the problem in the issue you mentioned. However, in this case, the crash happens some time before the main thread exits IMO, since the job output does not contain the summary line (total tests run, tests skipped, etc.)

[jkotas]

However, in this case, the crash happens some time before the main thread exits

Then it means that it is a different problem from #48411.

[bartonjs]

(lldb) dumpstack
OS Thread Id: 0x296f (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libcoreclr.so!Object::ValidateInner(int, int, int) + 0x1ad [/__w/1/s/src/coreclr/vm/object.h:446]
Child-SP         RetAddr          Caller, Callee
00007F85FECF9710 00007fc745e0135d libcoreclr.so!OBJECTREF::OBJECTREF(Object*) + 0x11d [/__w/1/s/src/coreclr/vm/object.cpp:1132], calling libcoreclr.so!Object::ValidateInner(int, int, int) [/__w/1/s/src/coreclr/vm/object.cpp:513]
00007F85FECF9740 00007fc745f70130 libcoreclr.so!MarshalNative::GCHandleInternalGet(OBJECTHANDLE__*) + 0x50 [/__w/1/s/src/coreclr/vm/gchandleutilities.h:44], calling libcoreclr.so!OBJECTREF::OBJECTREF(Object*) [/__w/1/s/src/coreclr/vm/object.cpp:1117]
00007F85FECF9780 00007fc6cf648f26 (MethodDesc 00007fc6cf5c4370 + 0x66 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling 00007fc745f700e0 (stub for System.Runtime.InteropServices.GCHandle.InternalGet(IntPtr))
00007F85FECF97B8 00007fc6cf648ef8 (MethodDesc 00007fc6cf5c4370 + 0x38 Interop+OpenSsl.NewSessionCallback(IntPtr, IntPtr)), calling libcoreclr.so!JIT_PInvokeBegin [/__w/1/s/src/coreclr/pal/inc/unixasmmacrosamd64.inc:896]
00007F85FECF9820 00007fc6c8310356 libssl.so.1.1!SSL_set_fd + 0x56

This failed in

runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs

Lines 700 to 726 in 6ca8c9b

	[UnmanagedCallersOnly]
	// Invoked from OpenSSL when new session is created.
	// We attached GCHandle to the SSL so we can find back SafeSslContextHandle holding the cache.
	// New session has refCount of 1.
	// If this function returns 0, OpenSSL will drop the refCount and discard the session.
	// If we return 1, the ownership is transfered to us and we will need to call SessionFree().
	private static unsafe int NewSessionCallback(IntPtr ssl, IntPtr session)
	{
	Debug.Assert(ssl != IntPtr.Zero);
	Debug.Assert(session != IntPtr.Zero);
	IntPtr ptr = Ssl.SslGetData(ssl);
	Debug.Assert(ptr != IntPtr.Zero);
	GCHandle gch = GCHandle.FromIntPtr(ptr);
	SafeSslContextHandle? ctxHandle = gch.Target as SafeSslContextHandle;
	// There is no relation between SafeSslContextHandle and SafeSslHandle so the handle
	// may be released while the ssl session is still active.
	if (ctxHandle != null && ctxHandle.TryAddSession(Ssl.SslGetServerName(ssl), session))
	{
	// offered session was stored in our cache.
	return 1;
	}
	// OpenSSL will destroy session.
	return 0;
	}

, which feels like it means #64369 made some bad assumptions.

[jkotas]

This looks like the bad assumption:

runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs

Lines 716 to 717 in 6ca8c9b

	// There is no relation between SafeSslContextHandle and SafeSslHandle so the handle
	// may be released while the ssl session is still active.

GCHandles are unmanaged resource. It is not ok to access them once they have been freed.