Selected apphost is not a valid PE file on s390x

[omajid]

Using a .NET 8 RC 1 SDK compiled on s390x with #90292 applied locally, dotnet tool install -g dotnet-ef fails:

    $ dotnet tool install -g dotnet-ef
    Unhandled exception: Microsoft.NET.HostModel.AppHost.AppHostNotPEFileException: Selected apphost is not a valid PE file. Subsystem offset out of file range.
       at Microsoft.NET.HostModel.AppHost.PEUtils.GetWindowsGraphicalUserInterfaceBit(MemoryMappedViewAccessor accessor)
       at Microsoft.NET.HostModel.AppHost.PEUtils.GetWindowsGraphicalUserInterfaceBit(String filePath)
       at Microsoft.DotNet.ShellShim.AppHostShellShimMaker.CreateApphostShellShim(FilePath entryPoint, FilePath shimPath)
       at Microsoft.DotNet.ShellShim.ShellShimRepository.<>c__DisplayClass5_0.<CreateShim>b__0()
       at Microsoft.DotNet.Cli.TransactionalAction.<>c__DisplayClass5_0.<Run>b__0()
       at Microsoft.DotNet.Cli.TransactionalAction.Run[Object](Func`1 action, Action commit, Action rollback)
       at Microsoft.DotNet.Cli.TransactionalAction.Run(Action action, Action commit, Action rollback)
       at Microsoft.DotNet.ShellShim.ShellShimRepository.CreateShim(FilePath targetExecutablePath, ToolCommandName commandName, IReadOnlyList`1 packagedShims)
       at Microsoft.DotNet.Tools.Tool.Install.ToolInstallGlobalOrToolPathCommand.Execute()
       at Microsoft.DotNet.Tools.Tool.Install.ToolInstallCommand.Execute()
       at Microsoft.DotNet.Cli.ToolInstallCommandParser.<>c.<ConstructCommand>b__17_0(ParseResult parseResult)
       at System.CommandLine.Invocation.AnonymousCliAction.Invoke(ParseResult parseResult)
       at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
       at System.CommandLine.ParseResult.Invoke()
       at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

The stack trace suggests this is the culprit:

runtime/src/installer/managed/Microsoft.NET.HostModel/AppHost/PEUtils.cs

Lines 114 to 142 in 32c3355

	internal static unsafe ushort GetWindowsGraphicalUserInterfaceBit(MemoryMappedViewAccessor accessor)
	{
	byte* pointer = null;
	try
	{
	accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);
	byte* bytes = pointer + accessor.PointerOffset;
	// https://en.wikipedia.org/wiki/Portable_Executable
	uint peHeaderOffset = ((uint*)(bytes + PEOffsets.DosStub.PESignatureOffset))[0];
	if (accessor.Capacity < peHeaderOffset + PEOffsets.PEHeader.Subsystem + sizeof(ushort))
	{
	throw new AppHostNotPEFileException("Subsystem offset out of file range.");
	}
	ushort* subsystem = ((ushort*)(bytes + peHeaderOffset + PEOffsets.PEHeader.Subsystem));
	return subsystem[0];
	}
	finally
	{
	if (pointer != null)
	{
	accessor.SafeMemoryMappedViewHandle.ReleasePointer();
	}
	}
	}

Probably an endianness issue?

[omajid]

cc @uweigand @Vishwanatha-HD

Tagging subscribers to this area: @vitek-karas, @agocke
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Using a .NET 8 RC 1 SDK compiled on s390x with #90292 applied locally, dotnet tool install -g dotnet-ef fails:

    $ dotnet tool install -g dotnet-ef
    Unhandled exception: Microsoft.NET.HostModel.AppHost.AppHostNotPEFileException: Selected apphost is not a valid PE file. Subsystem offset out of file range.
       at Microsoft.NET.HostModel.AppHost.PEUtils.GetWindowsGraphicalUserInterfaceBit(MemoryMappedViewAccessor accessor)
       at Microsoft.NET.HostModel.AppHost.PEUtils.GetWindowsGraphicalUserInterfaceBit(String filePath)
       at Microsoft.DotNet.ShellShim.AppHostShellShimMaker.CreateApphostShellShim(FilePath entryPoint, FilePath shimPath)
       at Microsoft.DotNet.ShellShim.ShellShimRepository.<>c__DisplayClass5_0.<CreateShim>b__0()
       at Microsoft.DotNet.Cli.TransactionalAction.<>c__DisplayClass5_0.<Run>b__0()
       at Microsoft.DotNet.Cli.TransactionalAction.Run[Object](Func`1 action, Action commit, Action rollback)
       at Microsoft.DotNet.Cli.TransactionalAction.Run(Action action, Action commit, Action rollback)
       at Microsoft.DotNet.ShellShim.ShellShimRepository.CreateShim(FilePath targetExecutablePath, ToolCommandName commandName, IReadOnlyList`1 packagedShims)
       at Microsoft.DotNet.Tools.Tool.Install.ToolInstallGlobalOrToolPathCommand.Execute()
       at Microsoft.DotNet.Tools.Tool.Install.ToolInstallCommand.Execute()
       at Microsoft.DotNet.Cli.ToolInstallCommandParser.<>c.<ConstructCommand>b__17_0(ParseResult parseResult)
       at System.CommandLine.Invocation.AnonymousCliAction.Invoke(ParseResult parseResult)
       at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
       at System.CommandLine.ParseResult.Invoke()
       at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

The stack trace suggests this is the culprit:

runtime/src/installer/managed/Microsoft.NET.HostModel/AppHost/PEUtils.cs

Lines 114 to 142 in 32c3355

	internal static unsafe ushort GetWindowsGraphicalUserInterfaceBit(MemoryMappedViewAccessor accessor)
	{
	byte* pointer = null;
	try
	{
	accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);
	byte* bytes = pointer + accessor.PointerOffset;
	// https://en.wikipedia.org/wiki/Portable_Executable
	uint peHeaderOffset = ((uint*)(bytes + PEOffsets.DosStub.PESignatureOffset))[0];
	if (accessor.Capacity < peHeaderOffset + PEOffsets.PEHeader.Subsystem + sizeof(ushort))
	{
	throw new AppHostNotPEFileException("Subsystem offset out of file range.");
	}
	ushort* subsystem = ((ushort*)(bytes + peHeaderOffset + PEOffsets.PEHeader.Subsystem));
	return subsystem[0];
	}
	finally
	{
	if (pointer != null)
	{
	accessor.SafeMemoryMappedViewHandle.ReleasePointer();
	}
	}
	}

Probably an endianness issue?

Author:	omajid
Assignees:	-
Labels:	area-HostModel, untriaged, arch-s390x, needs-area-label
Milestone:	-

[agocke]

Cc @elinor-fung . Looks like endianness mismatch in the new resource writing code.

[elinor-fung]

Taking a look. It's just in GetWindowsGraphicalUserInterfaceBit, not at the point of trying to write resources yet - I don't think that was changed (constants were just moved to a shared place).

[agocke]

Yeah, that uint* cast looks pretty suspicious -- that'll be reversed on big-endian.

[elinor-fung]

Oh, I think this is because AppHostShimMaker had an implicit assumption that ResourceUpdater.IsSupportedOS() is tied to Windows.
https://github.com/dotnet/sdk/blob/d4f2153fdbdbd5ee6b793c03fc908254fa6b253c/src/Cli/dotnet/ShellShim/AppHostShimMaker.cs#L44-L47

Before, GetWindowsGraphicalUserInterfaceBit was never getting called on non-Windows, but now, ResourceUpdater.IsSupportedOS() is also true on non-Windows.