mono `minijit` runtime tests failing with `coreclr_initialize failed - Error: 0x80004005`

[radical]

Failures 4/5-9/8 (incl. PRs and 7.0 branches):

Day	Run	OS
9/5	Rolling run 5191 (only in Runfo)	mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open
9/5	Rolling run 4698 (only in Runfo)	mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open
8/30	Rolling run	Ubuntu.1804.Arm64.Open - Mono Interpreter
8/30	PR #74831	OSX.1200.Amd64.Open - Mono Interpreter
8/24	PR #74459	OSX 12 amd64 - Moni minijit
8/8	PR #73416	OSX 12 amd64 - Mono Interpreter
8/4	PR #73230	OSX 12 amd64 - Mono Interpreter
8/2	PR #73157	OSX 12 amd64 - Moni minijit
7/29	Rolling run	Ubuntu.1804 arm64 - Mono llvm full aot
7/12-7/27	8x unique PRs	OSX 12 amd64 and Ubuntu.1804 arm64
6/1-6/30	6x unique PRs	OSX 12 amd64 and Ubuntu.1804 arm64
5/1-5/31	2x unique PRs	OSX 12 amd64
4/1-4/30	6x unique PRs	OSX 12 amd64 and Ubuntu.1804 arm64

let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true);

Hit on #72094 .
Build. From one of the logs:

    JIT/Math/Functions/Functions_ro/Functions_ro.sh [FAIL]
      BEGIN: coreclr_initialize failed - Error: 0x80004005
      END: coreclr_initialize failed - Error: 0x80004005
      
      Return code:      1
      Raw output file:      /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
      Raw output:
      BEGIN EXECUTION
      /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
      Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
      Properties:
          TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
          APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
          NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
          System.Reflection.Metadata.MetadataUpdater.IsSupported = false
      Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
      Arguments (0): 
      Expected: 100
      Actual: 255
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/tmp/helix/working/B5FB0986/p
      > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_Math._Functions_Functions_ro_Functions_ro_._Functions_Functions_ro_Functions_ro_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        BEGIN: coreclr_initialize failed - Error: 0x80004005
        END: coreclr_initialize failed - Error: 0x80004005
        
        Return code:      1
        Raw output file:      /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
        Raw output:
        BEGIN EXECUTION
        /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
        Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
        Properties:
            TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
            APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
            NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
            System.Reflection.Metadata.MetadataUpdater.IsSupported = false
        Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
        Arguments (0): 
        Expected: 100
        Actual: 255
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/tmp/helix/working/B5FB0986/p
        > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh

cc @lewing

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[lewing]

marking future until we see it again

[karelz]

@lewing it happens more often - see updated top post with Kusto query. Overall about 2x per week - marking it as 'blocking-clean-ci'.
Marking as untriaged again.

Also, it is causing massive failures on the runs, which makes it hard to analyze results -- see for example Rolling build from 7/29 - 1912276 has 200 failures.

[fanyang-mono]

The complete list of test failures with this error:
https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E14+definition%3Aruntime+message%3Acoreclr_initialize+jobName%3AMono

Build from 8/8
https://dev.azure.com/dnceng/public/_build/results?buildId=1929242&view=logs&j=76620f41-b685-5972-e5ab-3722b69632c4&t=e5054d15-9cd7-5687-4154-57982228d697
The failed lane was mono monointerpreter Pri0 Runtime Tests Run OSX x64 release

Build from 8/4 RP #73230
https://dev.azure.com/dnceng/public/_build/results?buildId=1922628&view=logs&j=76620f41-b685-5972-e5ab-3722b69632c4&t=e5054d15-9cd7-5687-4154-57982228d697
The failed lane was mono monointerpreter Pri0 Runtime Tests Run OSX x64 release

Build from 7/29 rolling build https://dev.azure.com/dnceng/public/_build/results?buildId=1912276&view=logs&j=d631b51d-8d68-5da3-5a13-95f6a763c7bb&t=e848a3c9-1589-513e-d5da-7aef4a24d916
The failed lane was mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release

@karelz I couldn't find the build containing this error from 8/2 PR #73157. Could you please comment it here?

[fanyang-mono]

I tried to run Interop/PInvoke/SetLastError/SetLastErrorTest/SetLastErrorTest.sh with interpreter locally multiple times, but wasn't able to reproduce this failure.

[fanyang-mono]

According to https://support.microsoft.com/en-us/topic/you-may-receive-error-code-0x80004005-or-other-error-codes-when-you-try-to-start-a-windows-xp-based-computer-a15f5b2f-642d-24ac-4912-1570a6bcedec. Error: 0x80004005 seems to be a windows related issue. However, the failures are happening on non-win platforms.

@lambdageek Do you have any thoughts on this?