Mutex.TryOpenExisting intermittently throws IOException

[rokonec]

Description

After introducing .NET 7 rc1 SDK into Runtime CI we have started seeing intermittent exceptions System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-{45_random-chars}' in Runtime and Arcade on Linux CI agents or docker based Linux builds.

Reproduction Steps

I was trying hard to create minimal repro - without success.
I believe easiest repro would be to rerun some of our CI's where it was seen:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=31675&view=logs&j=3fe1f0d5-61d6-5e8f-eead-4d3bcfb9dfc3&t=380e8ab7-dd79-5cad-d265-1eca160e9b82&s=526c4a30-42a9-575e-a58b-243c7c515350

https://dev.azure.com/dnceng-public/public/_build/results?buildId=41146&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552

Expected behavior

new Mutex(initiallyOwned: true, name: "Global\UniqueName", out bool createdNew) and Mutex.TryOpenExisting shall never intermitently throw IOException.

Actual behavior

new Mutex(initiallyOwned: true, name: "Global\UniqueName", out bool createdNew) and Mutex.TryOpenExisting sometimes throws:

System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-{45_random-chars}'
     at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
     at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
     at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
     at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
     at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
     at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

Regression?

Unknown

Known Workarounds

Unknown.

Configuration

I have seen this mostly on:

• OSX_x64
• Linux_musl_x64
• Linux_x64

Other information

No response

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

After introducing .NET 7 rc1 SDK into Runtime CI we have started seeing intermittent exceptions System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-{45_random-chars}' in Runtime and Arcade on Linux CI agents or docker based Linux builds.

Reproduction Steps

I was trying hard to create minimal repro - without success.
I believe easiest repro would be to rerun some of our CI's where it was seen:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=31675&view=logs&j=3fe1f0d5-61d6-5e8f-eead-4d3bcfb9dfc3&t=380e8ab7-dd79-5cad-d265-1eca160e9b82&s=526c4a30-42a9-575e-a58b-243c7c515350

https://dev.azure.com/dnceng-public/public/_build/results?buildId=41146&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552

Expected behavior

new Mutex(initiallyOwned: true, name: "Global\UniqueName", out bool createdNew) and Mutex.TryOpenExisting shall never intermitently throw IOException.

Actual behavior

new Mutex(initiallyOwned: true, name: "Global\UniqueName", out bool createdNew) and Mutex.TryOpenExisting sometimes throws:

System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-{45_random-chars}'
     at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
     at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
     at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
     at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
     at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
     at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

Regression?

Unknown

Known Workarounds

Unknown.

Configuration

I have seen this mostly on:

• OSX_x64
• Linux_musl_x64
• Linux_x64

Other information

No response

Author:	rokonec
Assignees:	-
Labels:	area-System.Threading
Milestone:	-

[rokonec]

@kouvel You have been recommended me to tag here by @janvorli. Please check it out.

[kouvel]

The TryOpenExisting failure seems to have occurred in Mono wasm builds. I'm guessing the create failure is also coming from Mono, based on the error message and similarity to the other stack. I didn't find any core dumps, also not sure where the error is being thrown from, looking at the code.

[jkotas]

@kouvel This was thrown from here: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Threading/Mutex.Windows.cs#L35 .

Note that the "Connection timed out" message may be bogus. This code is mixing and matching Windows and Unix error codes.

The error comes from msbuild. msbuild always runs on CoreCLR. This is not Mono problem.

[kouvel]

Ok makes sense. I wasn't aware of anything on these paths that would cause a timeout, but may be possible depending on the setup. I'll try to see what errors may be leading to this and what may be causing it.

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

After introducing .NET 7 rc1 SDK into Runtime CI we have started seeing intermittent exceptions System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-{45_random-chars}' in Runtime and Arcade on Linux CI agents or docker based Linux builds.

Reproduction Steps

I was trying hard to create minimal repro - without success.
I believe easiest repro would be to rerun some of our CI's where it was seen:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=31675&view=logs&j=3fe1f0d5-61d6-5e8f-eead-4d3bcfb9dfc3&t=380e8ab7-dd79-5cad-d265-1eca160e9b82&s=526c4a30-42a9-575e-a58b-243c7c515350

https://dev.azure.com/dnceng-public/public/_build/results?buildId=41146&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552

Expected behavior

new Mutex(initiallyOwned: true, name: "Global\UniqueName", out bool createdNew) and Mutex.TryOpenExisting shall never intermitently throw IOException.

Actual behavior

new Mutex(initiallyOwned: true, name: "Global\UniqueName", out bool createdNew) and Mutex.TryOpenExisting sometimes throws:

System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-{45_random-chars}'
     at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
     at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
     at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
     at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
     at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
     at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

Regression?

Unknown

Known Workarounds

Unknown.

Configuration

I have seen this mostly on:

• OSX_x64
• Linux_musl_x64
• Linux_x64

Other information

No response

Author:	rokonec
Assignees:	-
Labels:	area-System.Threading, untriaged
Milestone:	-