[6.0] CI failure in OSX: Unable to load shared library 'MovedNativeLib' in DllImportPathTest.sh

[carlossanlop]

Found this failure in a release/6.0 backport PR.

• PR: [release/6.0] fix for generation info in GCMemoryInfo  #80417
• Queue: CoreCLR Pri0 Runtime Tests Run OSX x64 checked
• Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=131615&view=logs&j=9eaa4d7e-225e-55b8-2a77-e1ac7e76d448&t=f70528f6-0b93-5c49-dfc7-de8b842ee9d6
• Log files:
  • https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-80417-merge-3415b8869aa64da6b7/Interop/1/console.14d230ef.log?helixlogtype=result
  • https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-80417-merge-4f833e2ea7554e3f94/Interop/1/console.c47afc9b.log?helixlogtype=result
• Callstacks:

Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 6.0.11)
  Discovering: Interop.DllImportAttribute.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  Interop.DllImportAttribute.XUnitWrapper (found 2 test cases)
  Starting:    Interop.DllImportAttribute.XUnitWrapper (parallel test collections = on, max threads = 6)
    Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh [FAIL]
      
      Return code:      1
      Raw output file:      /tmp/helix/working/A4F20917/w/A31C08D1/uploads/Reports/Interop.DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.output.txt
      Raw output:
      BEGIN EXECUTION
      /tmp/helix/working/A4F20917/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false DllImportPathTest.dll ''
      System.DllNotFoundException: Unable to load shared library 'MovedNativeLib' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libMovedNativeLib, 0x0001): tried: 'libMovedNativeLib' (no such file), '/usr/local/lib/libMovedNativeLib' (no such file), '/usr/lib/libMovedNativeLib' (no such file), '/private/tmp/helix/working/A4F20917/w/A31C08D1/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/libMovedNativeLib' (no such file)
         at Test.GetZero_PathEnv()
         at Test.Main(String[] args)
      Expected: 100
      Actual: 101
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/tmp/helix/working/A4F20917/p
      > /private/tmp/helix/working/A4F20917/w/A31C08D1/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at Interop_DllImportAttribute._DllImportPath_DllImportPathTest_DllImportPathTest_._DllImportPath_DllImportPathTest_DllImportPathTest_sh()
      Output:
        
        Return code:      1
        Raw output file:      /tmp/helix/working/A4F20917/w/A31C08D1/uploads/Reports/Interop.DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.output.txt
        Raw output:
        BEGIN EXECUTION
        /tmp/helix/working/A4F20917/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false DllImportPathTest.dll ''
        System.DllNotFoundException: Unable to load shared library 'MovedNativeLib' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libMovedNativeLib, 0x0001): tried: 'libMovedNativeLib' (no such file), '/usr/local/lib/libMovedNativeLib' (no such file), '/usr/lib/libMovedNativeLib' (no such file), '/private/tmp/helix/working/A4F20917/w/A31C08D1/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/libMovedNativeLib' (no such file)
           at Test.GetZero_PathEnv()
           at Test.Main(String[] args)
        Expected: 100
        Actual: 101
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/tmp/helix/working/A4F20917/p
        > /private/tmp/helix/working/A4F20917/w/A31C08D1/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh

{
  "ErrorMessage": "System.DllNotFoundException: Unable to load shared library 'MovedNativeLib' or one of its dependencies.",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
165842	dotnet/runtime	Interop.WorkItemExecution	#81858
164770	dotnet/runtime	Interop.WorkItemExecution
164488	dotnet/runtime	Interop.WorkItemExecution
163905	dotnet/runtime	Interop.WorkItemExecution
163740	dotnet/runtime	Interop.WorkItemExecution
163452	dotnet/runtime	Interop.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	6	6

[akoeplinger]

We've apparently also seen this back when trying to move to OSX 12: #59085 (comment)

[buyaa-n]

Same failure https://github.com/dotnet/runtime/pull/81291/checks?check_run_id=10939895129

[elinor-fung]

We've apparently also seen this back when trying to move to OSX 12

I believe this was from macOS purging environment variables like LD_LIBRARY_PATH/DYLD_LIBRARY_PATH, which the test relies on. We ended up disabling the one failing case on macOS when we actually did the move - #64565 - so I guess we want that in 6.0 too.

[carlossanlop]

This is still happening in PRs in the release/6.0 branch:

• Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=163519
• Build error leg or test failing: Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh
• Pull request: [release/6.0] Use BOOL (vs. bool) in event pipe qcall signatures #81238
• Queue: CoreCLR Pri0 Runtime Tests Run OSX x64 checked
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=163519&view=logs&j=9eaa4d7e-225e-55b8-2a77-e1ac7e76d448&t=f70528f6-0b93-5c49-dfc7-de8b842ee9d6
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-81238-merge-270f0863b8e24baab1/Interop/1/console.534b23e9.log?helixlogtype=result

Callstack

Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 6.0.13)
  Discovering: Interop.DllImportAttribute.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  Interop.DllImportAttribute.XUnitWrapper (found 2 test cases)
  Starting:    Interop.DllImportAttribute.XUnitWrapper (parallel test collections = on, max threads = 6)
    Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh [FAIL]
      
      Return code:      1
      Raw output file:      /tmp/helix/working/A72A0967/w/A6FB08DB/uploads/Reports/Interop.DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.output.txt
      Raw output:
      BEGIN EXECUTION
      /tmp/helix/working/A72A0967/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false DllImportPathTest.dll ''
      System.DllNotFoundException: Unable to load shared library 'MovedNativeLib' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libMovedNativeLib, 0x0001): tried: 'libMovedNativeLib' (no such file), '/usr/local/lib/libMovedNativeLib' (no such file), '/usr/lib/libMovedNativeLib' (no such file), '/private/tmp/helix/working/A72A0967/w/A6FB08DB/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/libMovedNativeLib' (no such file)
         at Test.GetZero_PathEnv()
         at Test.TestNativeLibraryProbingOnPathEnv()
         at Test.Main(String[] args)
      Expected: 100
      Actual: 101
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/tmp/helix/working/A72A0967/p
      > /private/tmp/helix/working/A72A0967/w/A6FB08DB/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at Interop_DllImportAttribute._DllImportPath_DllImportPathTest_DllImportPathTest_._DllImportPath_DllImportPathTest_DllImportPathTest_sh()
      Output:
        
        Return code:      1
        Raw output file:      /tmp/helix/working/A72A0967/w/A6FB08DB/uploads/Reports/Interop.DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.output.txt
        Raw output:
        BEGIN EXECUTION
        /tmp/helix/working/A72A0967/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false DllImportPathTest.dll ''
        System.DllNotFoundException: Unable to load shared library 'MovedNativeLib' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libMovedNativeLib, 0x0001): tried: 'libMovedNativeLib' (no such file), '/usr/local/lib/libMovedNativeLib' (no such file), '/usr/lib/libMovedNativeLib' (no such file), '/private/tmp/helix/working/A72A0967/w/A6FB08DB/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/libMovedNativeLib' (no such file)
           at Test.GetZero_PathEnv()
           at Test.TestNativeLibraryProbingOnPathEnv()
           at Test.Main(String[] args)
        Expected: 100
        Actual: 101
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/tmp/helix/working/A72A0967/p
        > /private/tmp/helix/working/A72A0967/w/A6FB08DB/e/Interop/DllImportAttribute/DllImportPath/DllImportPathTest/DllImportPathTest.sh

[elinor-fung]

#81856 backports the change to disable the test case.