From 60b820948e8fb99ddd077854cb505fe3631d776e Mon Sep 17 00:00:00 2001 From: Larry Ewing Date: Mon, 28 Sep 2026 13:39:09 -0500 Subject: [PATCH 1/2] [wasi] Publish CoreCLR composite ReadyToRun apps Add CoreCLR-WASI composite ReadyToRun publishing for in-tree WASI apps: crossgen2 composite compilation from the WASI app builder, a host-sized image reservation, the ComposeWasiReadyToRun MSBuild task that composes the self-installing composite into the linked host component, and the wasihost external assembly probe that serves the composite and per-assembly stubs. In-tree publishing acquires pinned wasm-tools and Binaryen through eng/AcquireWasiR2RTools.targets. Runtime-test and library-test CI lanes follow in a separate PR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 919fdde7-f547-4bfb-86ad-7ac02c626980 --- docs/design/mono/webcil.md | 23 + docs/workflow/building/coreclr/README.md | 1 + docs/workflow/building/coreclr/wasi-r2r.md | 47 ++ eng/AcquireWasiR2RTools.targets | 52 ++ eng/wasm/PruneWasmToolCache.proj | 1 + eng/wasm/WasmToolCache.props | 16 +- src/coreclr/hosts/corerun/wasi_r2r_probe.hpp | 214 ++++++++ src/mono/wasi/build/WasiApp.CoreCLR.targets | 196 ++++++- src/mono/wasi/build/WasiApp.InTree.props | 9 + src/mono/wasi/build/WasiApp.InTree.targets | 7 + .../wasi/build/WasiApp.ReadyToRun.targets | 20 + src/native/corehost/wasihost/wasihost.cpp | 48 ++ .../Microsoft.NET.CrossGen.targets | 11 +- .../PrepareForReadyToRunCompilation.cs | 7 + .../WasmModuleInfoReader.cs | 502 ++++++++++++++++++ .../WasmModuleReader.cs | 1 + .../WasmAppBuilder/ComposeWasiReadyToRun.cs | 212 ++++++++ 17 files changed, 1349 insertions(+), 18 deletions(-) create mode 100644 docs/workflow/building/coreclr/wasi-r2r.md create mode 100644 eng/AcquireWasiR2RTools.targets create mode 100644 src/coreclr/hosts/corerun/wasi_r2r_probe.hpp create mode 100644 src/mono/wasi/build/WasiApp.ReadyToRun.targets create mode 100644 src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleInfoReader.cs create mode 100644 src/tasks/WasmAppBuilder/ComposeWasiReadyToRun.cs diff --git a/docs/design/mono/webcil.md b/docs/design/mono/webcil.md index e02ab862b79a3b..b8d1a295962b6c 100644 --- a/docs/design/mono/webcil.md +++ b/docs/design/mono/webcil.md @@ -208,6 +208,29 @@ the relocation scheme in its own code, reducing the volume of code needed in eac images into linear memory, as well as for allowing for efficient storage of 128 bit vector constants within the binary.) +##### WASI host composition + +WASI hosts use offline composition rather than instantiating R2R modules at runtime. +The in-tree C# composer unbundles the host component, creates a shim exporting the image and +table bases, merges the host, shim, and composite with `wasm-merge`, folds the globals with +`wasm-opt --simplify-globals`, and replaces the component's core module. Both Binaryen steps +preserve the `name` section with `-g`. + +The host exports its reserved buffer address, buffer capacity, and composite table base through +`wasi_r2r_image_base`, `wasi_r2r_image_cap`, and `wasi_r2r_table_base`. Before merging, the composer +checks that the active payload fits the buffer and that the composite's table entries end before +the host's own active element segment. These checks must happen before instantiation: a runtime +check cannot prevent an active segment from overwriting an undersized reservation. + +The shim's start function calls `patchWebcilHeader`; the active segments themselves install the +payload and function table. The host's external assembly probe serves `composite-r2r.wasm` from +the embedded buffer and maps raw per-assembly WebCIL forwarding stubs from `comp/.dll`. +The composer extracts those payloads from their self-installing Wasm wrappers at build time. +Placing a composite on disk without composing it into the host does not satisfy this contract. + +Publishing sizes the host reservations from the generated app/framework composite. +See [the WASI R2R workflow](../../workflow/building/coreclr/wasi-r2r.md) for usage and diagnostics. + ### Webcil payload The webcil payload contains the ECMA-335 metadata, IL and resources comprising a .NET assembly. diff --git a/docs/workflow/building/coreclr/README.md b/docs/workflow/building/coreclr/README.md index a9ddf55587034b..4500213ad0ac69 100644 --- a/docs/workflow/building/coreclr/README.md +++ b/docs/workflow/building/coreclr/README.md @@ -96,6 +96,7 @@ Detailed instructions on how to do cross-compilation can be found in the cross-b For specialized platforms, detailed instructions are available in the following guides: - **WebAssembly:** [Building CoreCLR for WebAssembly](/docs/workflow/building/coreclr/wasm.md) - Experimental support for building, running, and debugging CoreCLR on WebAssembly. +- **WASI ReadyToRun:** [Publishing composite R2R images](wasi-r2r.md) - Experimental CoreCLR-WASI composition workflow. ## Other Features diff --git a/docs/workflow/building/coreclr/wasi-r2r.md b/docs/workflow/building/coreclr/wasi-r2r.md new file mode 100644 index 00000000000000..f968cf63639149 --- /dev/null +++ b/docs/workflow/building/coreclr/wasi-r2r.md @@ -0,0 +1,47 @@ +# CoreCLR-WASI composite ReadyToRun + +This is an experimental in-tree publishing workflow. The shipping WASI SDK +does not yet select the CoreCLR app builder. + +WASI requires the compiled composite to be composed into the host component before execution. +Copying `composite-r2r.wasm` beside an unmodified host is not sufficient. Composition is +implemented by the in-tree `ComposeWasiReadyToRun` MSBuild task. +The [WebCIL design document](../../../design/mono/webcil.md#wasi-host-composition) describes +the image layout and host contract. + +## Prerequisites + +Build prerequisites are described in the [CoreCLR build guide](README.md). +Composition additionally requires `wasm-tools` and Binaryen's `wasm-merge` and `wasm-opt`. +Running the result requires wasmtime with WebAssembly exception support. +In-tree publishing acquires pinned tool versions into the shared wasm tool cache through +`eng/AcquireWasiR2RTools.targets`. + +Framework-sized composites can require several GiB of memory during composition. Allow for the +host and composite working sets when sizing build containers. + +## Publishing + +Build the runtime, libraries, and packs, then publish an in-tree WASI project: + +```bash +./build.sh -s clr+libs+packs -os wasi -arch wasm -c Release +./dotnet.sh publish -c Release -p:TargetOS=wasi \ + -p:RuntimeFlavor=CoreCLR -p:PublishReadyToRun=true +``` + +The app builder enables composite R2R, compiles the app/framework closure, sizes the host's image +buffer and table reservation, links the host, and invokes the composer. It deploys the composed +host and the per-assembly stubs under the app bundle's `managed/` directory. +Non-composite R2R and `WasmSingleFileBundle` are not supported by this path. + +## Composer diagnostics + +The build targets generate Crossgen2 response files; a hand-maintained response-file template +is not required. The host supplies the image and table bases, and the C# composer rejects an +undersized buffer, overlapping table reservation, or a non-self-installing composite. + +A valid composed module and a successful run alone do not prove R2R was used. Enable the guest's +`DOTNET_ReadyToRunLogFile` and look for `Ready to Run initialized successfully` for the app +assemblies. This confirms image loading; proving a particular method executes compiled code +requires a breakpoint or trace in that method's wasm body. diff --git a/eng/AcquireWasiR2RTools.targets b/eng/AcquireWasiR2RTools.targets new file mode 100644 index 00000000000000..53186f81ca8e5f --- /dev/null +++ b/eng/AcquireWasiR2RTools.targets @@ -0,0 +1,52 @@ + + + + <_WasiR2RToolOS Condition="'$(HostOS)' == 'linux'">linux + <_WasiR2RToolOS Condition="'$(HostOS)' == 'osx'">macos + <_WasiR2RToolArch Condition="'$(BuildArchitecture)' == 'x64'">x86_64 + <_WasiR2RToolArch Condition="'$(BuildArchitecture)' == 'arm64'">aarch64 + <_BinaryenArch>$(_WasiR2RToolArch) + <_BinaryenArch Condition="'$(HostOS)' == 'osx' and '$(BuildArchitecture)' == 'arm64'">arm64 + <_BinaryenArchiveName>binaryen-version_$(BinaryenVersion)-$(_BinaryenArch)-$(_WasiR2RToolOS) + <_WasmToolsArchiveName>wasm-tools-$(WasmToolsVersion)-$(_WasiR2RToolArch)-$(_WasiR2RToolOS) + + + + + + + + + + + + diff --git a/eng/wasm/PruneWasmToolCache.proj b/eng/wasm/PruneWasmToolCache.proj index 0febac1c46ef72..8347bf5d1552f6 100644 --- a/eng/wasm/PruneWasmToolCache.proj +++ b/eng/wasm/PruneWasmToolCache.proj @@ -29,6 +29,7 @@ <_WasmToolInUseStamp Include="$(EmscriptenSdkStampFile);$(WasiSdkStampFile);$(WasmtimeStampFile)" /> + <_WasmToolInUseStamp Include="$(BinaryenStampFile);$(WasmToolsStampFile)" /> <_WasmToolInUseStamp Include="$(ChromeStampFile);$(ChromeDriverStampFile)" Condition="'$(ChromeVersion)' != ''" /> <_WasmToolInUseStamp Include="$(V8StampFile)" Condition="'$(V8Version)' != ''" /> <_WasmToolInUseStamp Include="$(FirefoxStampFile);$(GeckoDriverStampFile)" Condition="'$(FirefoxRevision)' != ''" /> diff --git a/eng/wasm/WasmToolCache.props b/eng/wasm/WasmToolCache.props index 92021e914890bf..49a39f2bbe9561 100644 --- a/eng/wasm/WasmToolCache.props +++ b/eng/wasm/WasmToolCache.props @@ -1,6 +1,6 @@ @@ -61,6 +61,12 @@ $([System.IO.File]::ReadAllText('$(_WasiSdkVersionFile)').Trim()) + + + 130 + 1.253.0 + + @@ -73,5 +79,11 @@ $([MSBuild]::NormalizeDirectory('$(WasmToolCacheDir)', 'wasmtime', '$(WasmtimeVersion)-$(WasmToolHostRid)')) $([MSBuild]::NormalizePath('$(WasmToolCacheDir)', 'wasmtime', '$(WasmtimeVersion)-$(WasmToolHostRid).complete')) + + $([MSBuild]::NormalizeDirectory('$(WasmToolCacheDir)', 'binaryen', '$(BinaryenVersion)-$(WasmToolHostRid)')) + $([MSBuild]::NormalizePath('$(WasmToolCacheDir)', 'binaryen', '$(BinaryenVersion)-$(WasmToolHostRid).complete')) + + $([MSBuild]::NormalizeDirectory('$(WasmToolCacheDir)', 'wasm-tools', '$(WasmToolsVersion)-$(WasmToolHostRid)')) + $([MSBuild]::NormalizePath('$(WasmToolCacheDir)', 'wasm-tools', '$(WasmToolsVersion)-$(WasmToolHostRid).complete')) diff --git a/src/coreclr/hosts/corerun/wasi_r2r_probe.hpp b/src/coreclr/hosts/corerun/wasi_r2r_probe.hpp new file mode 100644 index 00000000000000..a72d77b85ce55d --- /dev/null +++ b/src/coreclr/hosts/corerun/wasi_r2r_probe.hpp @@ -0,0 +1,214 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +// Statically-composed WASI R2R external-assembly probe for the per-app-linked wasihost corehost +// (libWasiHost.a). The probe is a host_runtime_contract::external_assembly_probe callback: the runtime +// calls out to it to obtain the composite R2R webcil image and the per-assembly stubs. +// +// The in-tree WASI R2R composer populates the image when publishing. The host must be linked with the +// flags that supply a composite's imports -- see WasiEnableCompositeR2R in WasiApp.CoreCLR.targets. +// Without them this probe compiles but can never be satisfied. +// +// Requires corerun.hpp to be included first (for pal::try_map_file_readonly). Include exactly once per +// host binary. + +#ifndef WASI_R2R_PROBE_HPP +#define WASI_R2R_PROBE_HPP + +#ifdef TARGET_WASI + +#include +#include +#include +#include +#include + +namespace wasi_r2r +{ +// A crossgen2-produced R2R webcil image is merged into the host module post-link (its native +// functions land in the shared indirect function table and its webcil payload/metadata is written +// into g_wasi_r2r_image by the offline merge's active data segment at this buffer's address == the +// composite image's imageBase). The runtime then finds the R2R webcil via this probe, exactly the way +// the browser host does via BrowserHost_ExternalAssemblyProbe. +// +// The buffer is defined outside this header: a ReadyToRun publish builds a strong definition sized +// exactly to the composite, while the host archive carries a 64-byte weak fallback for non-R2R apps. +extern "C" uint8_t g_wasi_r2r_image[]; +extern "C" uint32_t g_wasi_r2r_image_cap; + +// The table index at which the composite's functions are installed. Under the reservation model the +// host is linked with `-Wl,--table-base=`, which moves the host's own address-taken functions up +// to start at N+1 and leaves slots 1..N free, so the composite always sits at base 1 regardless of +// its size. This MUST match the `__table_base` global supplied to the merge (see docs/design/mono/webcil.md); +// the two are a coupled constant, which the composer validates against the host's exported value. +#ifndef WASI_R2R_TABLE_BASE +#define WASI_R2R_TABLE_BASE (1u) +#endif +#define WEBCIL_HEADER_V1_SIZE (32u) +#define WEBCIL_SECTION_HEADER_SIZE (16u) +#define WEBCIL_VERSION_MAJOR_OFFSET (4u) + +// The composite native image's bundle-relative file name (the ownerCompositeExecutable named by each +// per-assembly stub). The runtime asks for this via NativeImage::Open -> external_assembly_probe. +#ifndef WASI_R2R_COMPOSITE_NAME +#define WASI_R2R_COMPOSITE_NAME "composite-r2r.wasm" +#endif + +static bool WasiIsWebcilV1(const uint8_t* p, size_t len) +{ + if (len < WEBCIL_HEADER_V1_SIZE) + return false; + + uint16_t versionMajor; + memcpy(&versionMajor, p + WEBCIL_VERSION_MAJOR_OFFSET, sizeof(versionMajor)); + return versionMajor == 1; +} + +// Compute the exact WbIL payload size from its self-describing header - no baked constant needed. +// WebcilHeader_1 (32 bytes): Id[4] 'WbIL', VersionMajor u16, VersionMinor u16, CoffSections u16, +// Reserved0 u16, PeCliHeaderRva u32, PeCliHeaderSize u32, PeDebugRva u32, PeDebugSize u32, TableBase u32. +// Followed by CoffSections * WebcilSectionHeader{VirtualSize, VirtualAddress, SizeOfRawData, PointerToRawData}. +// The payload extent is the maximum (PointerToRawData + SizeOfRawData) across all sections. +// +// Every field here comes from an image this host did not produce, so bounds and overflow are checked +// rather than assumed: a wrapped sum would yield a SMALL extent that passes the cap check below and +// hands the runtime a truncated image. +static int64_t WasiWebcilPayloadSize(const uint8_t* p, size_t len) +{ + if (!WasiIsWebcilV1(p, len)) + return 0; + + if (p[0] != 'W' || p[1] != 'b' || p[2] != 'I' || p[3] != 'L') + return 0; + + uint16_t coffSections; + memcpy(&coffSections, p + 8, sizeof(coffSections)); + + // Section headers must fit entirely within the buffer. + if ((len - WEBCIL_HEADER_V1_SIZE) / WEBCIL_SECTION_HEADER_SIZE < coffSections) + return 0; + + const uint8_t* sec = p + WEBCIL_HEADER_V1_SIZE; + uint32_t maxEnd = 0; + for (uint16_t i = 0; i < coffSections; i++) + { + uint32_t sizeOfRawData; + uint32_t pointerToRawData; + memcpy(&sizeOfRawData, sec + 8, sizeof(sizeOfRawData)); + memcpy(&pointerToRawData, sec + 12, sizeof(pointerToRawData)); + + // Reject rather than wrap: UINT32_MAX - a < b <=> a + b would overflow. + if (UINT32_MAX - pointerToRawData < sizeOfRawData) + return 0; + + uint32_t end = pointerToRawData + sizeOfRawData; + if (end > maxEnd) + maxEnd = end; + sec += WEBCIL_SECTION_HEADER_SIZE; + } + return (int64_t)maxEnd; +} + +// Map a raw WebCIL component forwarding stub extracted by the build-time composer. On success the +// file mapping is deliberately retained and returned directly to the runtime. +static bool WasiMapStubPayload(const char* webcilPath, void** data_start, int64_t* size) +{ + void* filedata = nullptr; int64_t filesize = 0; + if (!pal::try_map_file_readonly(webcilPath, &filedata, &filesize)) + return false; + + if (filesize <= 0) + return false; + + int64_t payloadSize = WasiWebcilPayloadSize(static_cast(filedata), static_cast(filesize)); + if (payloadSize <= 0 || payloadSize != filesize) + { + munmap(filedata, static_cast(filesize)); + return false; + } + + *data_start = filedata; + *size = filesize; + return true; +} + +// The external-assembly R2R probe: serves the composite webcil from the baked buffer and each managed +// assembly's per-assembly stub from "/comp/.dll" on disk, searching the supplied dirs (each +// expected to carry a trailing path delimiter). Returns false for anything it does not provide, letting +// the caller fall back to its normal assembly load. +static bool WasiStaticR2RProbe(const char* name, const char* const* dirs, size_t ndirs, void** data_start, int64_t* size) +{ + // The composite native image itself: return the merged composite payload at imageBase. Its size is + // read from the self-describing WbIL header (no baked constant), and validated against the buffer cap. + if (strcmp(name, WASI_R2R_COMPOSITE_NAME) == 0) + { + int64_t payloadSize = WasiWebcilPayloadSize(&g_wasi_r2r_image[0], g_wasi_r2r_image_cap); + if (payloadSize <= 0 || static_cast(payloadSize) > g_wasi_r2r_image_cap) + return false; // buffer not populated, or composite payload exceeds the cap + + // NOTE: the cap test above cannot protect this buffer -- the engine installs the segment before any + // host code runs, so an over-cap payload has already overwritten whatever follows by the time we look. + // The enforceable check is at build time; the C# composer compares the payload size against the cap. + + *data_start = &g_wasi_r2r_image[0]; + *size = payloadSize; + return true; + } + + // A managed assembly: return its per-assembly stub payload (extracted from .wasm on disk). + // The stub carries the assembly metadata + the R2R header naming the composite, which drives the + // runtime to then request WASI_R2R_COMPOSITE_NAME above. + size_t nlen = strlen(name); + if (nlen > 4 && strcmp(name + nlen - 4, ".dll") == 0) + { + char stub[512]; + for (size_t i = 0; i < ndirs; i++) + { + const char* dir = dirs[i]; + if (dir == nullptr) continue; + // Build "/comp/.dll"; the build-time composer extracts raw WebCIL from the Wasm + // wrapper so the runtime host does not need its own Wasm parser. + int written = snprintf(stub, sizeof(stub), "%scomp/%s", dir, name); + if (written < 0 || static_cast(written) >= sizeof(stub)) + continue; + if (WasiMapStubPayload(stub, data_start, size)) + { + return true; + } + } + } + return false; +} + +} // namespace wasi_r2r + +// Exported so the offline merge step can discover the buffer's address and wire it to the R2R image's +// __memory_base global. Defined outside the namespace with C linkage so the export name is exactly +// "wasi_r2r_image_base" (the merge step targets this symbol). +// +// Keeping the exported accessor here ensures the linker roots the buffer and gives the composition +// step a stable anchor. +extern "C" __attribute__((export_name("wasi_r2r_image_base"))) uint32_t wasi_r2r_image_base(void) +{ + return (uint32_t)(uintptr_t)&wasi_r2r::g_wasi_r2r_image[0]; +} + +// The staging buffer's capacity and the table slot the composite installs at, exported for the same +// reason as the base: the composition must not carry its own copy of either. The host owns these +// values; the WASI R2R composer reads them out of the linked binary and validates the composite +// against them, so a mismatch is a build-time error instead of a wrong-function dispatch at runtime. +// The capacity accessor is weak because a ReadyToRun publish supplies a strong definition alongside +// the sized buffer. +extern "C" __attribute__((weak, export_name("wasi_r2r_image_cap"))) uint32_t wasi_r2r_image_cap(void) +{ + return wasi_r2r::g_wasi_r2r_image_cap; +} + +extern "C" __attribute__((export_name("wasi_r2r_table_base"))) uint32_t wasi_r2r_table_base(void) +{ + return (uint32_t)WASI_R2R_TABLE_BASE; +} + +#endif // TARGET_WASI + +#endif // WASI_R2R_PROBE_HPP diff --git a/src/mono/wasi/build/WasiApp.CoreCLR.targets b/src/mono/wasi/build/WasiApp.CoreCLR.targets index f1e6a2806e0dda..0c61a7cea23567 100644 --- a/src/mono/wasi/build/WasiApp.CoreCLR.targets +++ b/src/mono/wasi/build/WasiApp.CoreCLR.targets @@ -54,6 +54,9 @@ + CoreCLR @@ -65,6 +68,95 @@ false + + wasm + true + $(PublishReadyToRunCrossgen2ExtraArgs);--opt-cross-module:*;--codegenopt:JitWasmNyiToR2RUnsupported=1;--codegenopt:JitWasmSimdNyiToR2RUnsupported=1 + true + + + + + + + + + + + + $(MicrosoftNetCoreAppRuntimePackRidNativeDir) + $(MicrosoftNetCoreAppRuntimePackRidNativeDir) + $(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)System.Private.CoreLib.dll + $(MicrosoftNetCoreAppRuntimePackRidNativeDir)System.Private.CoreLib.dll + + + + + + + + + + + <_WasiReadyToRunFrameworkInput Include="$(WasiCoreCLRSystemPrivateCoreLibPath)" /> + <_WasiReadyToRunFrameworkInput Include="$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)*.dll" + Exclude="$(WasiCoreCLRSystemPrivateCoreLibPath)" /> + + <_WasiReadyToRunFrameworkInput Remove="@(ResolvedFileToPublish)" + MatchOnMetadata="Filename;Extension" /> + + + + + + + <_WasiR2ROutputDir>$([MSBuild]::NormalizeDirectory('$(_ReadyToRunOutputPath)')) + <_WasiR2RCompositePath>$(_WasiR2ROutputDir)composite-r2r.wasm + <_WasiR2RImageBufferSource>$([MSBuild]::NormalizePath('$(_WasmIntermediateOutputPath)', 'corerun-relink', 'r2r-image-buffer.cpp')) + + + + + + + + + $([MSBuild]::Add($(_WasiR2RFunctionCount), 1)) + + + + + + + + @@ -90,15 +182,17 @@ (linked below), which finds them via CORE_ROOT. See https://github.com/dotnet/runtime/issues/130129. --> - + - <_WasiCoreCLRCoreLib Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)System.Private.CoreLib.dll" /> + <_WasiCoreCLRCoreLib Include="$(WasiCoreCLRSystemPrivateCoreLibPath)" /> - <_WasiCoreCLRFrameworkFiles Include="$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)\*.dll" /> + <_WasiCoreCLRFrameworkFiles Include="$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)\*.dll" + Exclude="$(WasiCoreCLRSystemPrivateCoreLibPath)" /> <_WasiCoreCLRFrameworkToCopy Include="@(_WasiCoreCLRFrameworkFiles)" Condition="!Exists('$(WasmAppDir)managed\%(_WasiCoreCLRFrameworkFiles.FileName)%(_WasiCoreCLRFrameworkFiles.Extension)')" /> @@ -131,7 +225,7 @@ <_WasiCallHelperSource Include="$(_WasiReversePInvokeTablePath)" ObjectFile="$(_WasiRelinkObjDir)callhelpers-reverse.o" /> <_WasiCallHelperSource Include="$(_WasiInterpToNativeTablePath)" ObjectFile="$(_WasiRelinkObjDir)callhelpers-interp-to-managed.o" /> + <_WasiCallHelperSource Include="$(_WasiR2RImageBufferSource)" + ObjectFile="$(_WasiRelinkObjDir)r2r-image-buffer.o" + Condition="'$(PublishReadyToRun)' == 'true'" /> - <_WasiHostLibsPre Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libminipal.a" /> - <_WasiHostLibsPre Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libcoreclr_static.a" /> + <_WasiHostLibsPre Include="$(WasiCoreCLRNativeRuntimeDir)libminipal.a" /> + <_WasiHostLibsPre Include="$(WasiCoreCLRNativeRuntimeDir)libcoreclr_static.a" /> <_WasiHostLibsPre Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libSystem.IO.Compression.Native.a" /> <_WasiHostLibsPre Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libSystem.Native.a" /> <_WasiHostLibsPre Condition="'$(InvariantTimezone)' == 'true'" Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libSystem.Native.TimeZoneData.Invariant.a" /> <_WasiHostLibsPre Condition="'$(InvariantTimezone)' != 'true'" Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libSystem.Native.TimeZoneData.a" /> <_WasiHostLibsPre Include="%(_WasiCallHelperSource.ObjectFile)" /> - <_WasiHostLibsPre Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libnativeresourcestring.a" /> + <_WasiHostLibsPre Include="$(WasiCoreCLRNativeRuntimeDir)libnativeresourcestring.a" /> - <_WasiHostLibsPost Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libcoreclrminipal.a" /> - <_WasiHostLibsPost Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libcoreclrpal.a" /> - <_WasiHostLibsPost Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libminipal.a" /> + <_WasiHostLibsPost Include="$(WasiCoreCLRNativeRuntimeDir)libcoreclrminipal.a" /> + <_WasiHostLibsPost Include="$(WasiCoreCLRNativeRuntimeDir)libcoreclrpal.a" /> + <_WasiHostLibsPost Include="$(WasiCoreCLRNativeRuntimeDir)libminipal.a" /> <_WasiHostLibsPost Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libz.a" /> <_WasiHostLibsPost Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libzstd.a" Condition="Exists('$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libzstd.a')" /> - <_WasiRelinkOutput>$(WasmAppDir)managed\corerun + <_WasiRelinkOutput>$(_WasiRelinkObjDir)corerun + <_WasiPublishedHost>$(WasmAppDir)managed\corerun + + false + + 65537 @@ -350,6 +456,16 @@ <_WasiRelinkLinkFlags Include="$(_WasiRelinkOptFlag)" /> <_WasiRelinkLinkFlags Include="-DNDEBUG" Condition="'$(Configuration)' != 'Debug'" /> <_WasiRelinkLinkFlags Include="-Wl,--gc-sections" /> + + <_WasiRelinkLinkFlags Include="-Wl,--export=__async_continuation" /> + + <_WasiRelinkLinkFlags Include="-Wl,--table-base=$(WasiCompositeR2RTableBase)" Condition="'$(WasiEnableCompositeR2R)' == 'true'" /> + <_WasiRelinkLinkFlags Include="-Wl,--export-table" Condition="'$(WasiEnableCompositeR2R)' == 'true'" /> + <_WasiRelinkLinkFlags Include="-Wl,--export=__stack_pointer" Condition="'$(WasiEnableCompositeR2R)' == 'true'" /> + <_WasiRelinkLinkFlags Include="-Wl,--export=__coreclr_wasm_rtlrestorecontext_tag" Condition="'$(WasiEnableCompositeR2R)' == 'true'" /> <_WasiRelinkLinkFlags Include="-fwasm-exceptions" /> <_WasiRelinkLinkFlags Include="-mllvm" /> <_WasiRelinkLinkFlags Include="-wasm-use-legacy-eh=false" /> @@ -364,7 +480,7 @@ <_WasiRelinkLinkFlags Include="-Wl,--component-type,"$(_WasiHttpWorldWit.Replace('\','/'))"" /> <_WasiRelinkLinkFlags Include="-Wl,--whole-archive" /> - <_WasiRelinkLinkFlags Include=""$(MicrosoftNetCoreAppRuntimePackRidNativeDir)libWasiHost.a"" /> + <_WasiRelinkLinkFlags Include=""$(WasiCoreCLRHostNativeDir)libWasiHost.a"" /> <_WasiRelinkLinkFlags Include="-Wl,--no-whole-archive" /> <_WasiRelinkLinkFlags Include="@(_WasiHostLibsPre->'"%(Identity)"')" /> <_WasiRelinkLinkFlags Include="-lstdc++" /> @@ -376,10 +492,64 @@ + + + + + <_WasiR2RComposeOutputDir>$([MSBuild]::NormalizeDirectory('$(_WasmIntermediateOutputPath)', 'r2r-compose')) + <_WasiR2RStubDir>$([MSBuild]::NormalizeDirectory('$(WasmAppDir)', 'managed', 'comp')) + + + + + <_WasiR2RComponentStub Include="@(_ReadyToRunCompositeBuildInput->'$(_WasiR2ROutputDir)%(FileName).wasm')" /> + <_WasiR2RComponentStub Include="@(_ReadyToRunCompositeUnrootedBuildInput->'$(_WasiR2ROutputDir)%(FileName).wasm')" /> + + + + + + + + + + + + $(WasmAppDir)run-wasmtime.sh + <_ScriptContent Condition="'$(WasmSingleFileBundle)' == 'true'">wasmtime run $([System.IO.Path]::GetFileNameWithoutExtension($(WasmMainAssemblyFileName))).wasm $* + <_ScriptContent Condition="'$(WasmSingleFileBundle)' != 'true'">cd managed && wasmtime run -W exceptions=y -S http --dir . corerun $(WasmMainAssemblyFileName) $* + + + + + + + + + diff --git a/src/mono/wasi/build/WasiApp.InTree.props b/src/mono/wasi/build/WasiApp.InTree.props index ae0f77312bfe23..77780493efd1c2 100644 --- a/src/mono/wasi/build/WasiApp.InTree.props +++ b/src/mono/wasi/build/WasiApp.InTree.props @@ -8,11 +8,20 @@ WasiApp.CoreCLR.targets). --> false + $([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'wasi-wasm.$(Configuration)', 'sharedFramework')) + $([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'coreclr', 'wasi.wasm.$(Configuration)', 'sharedFramework')) + $([MSBuild]::NormalizePath('$(ArtifactsBinDir)', 'coreclr', 'wasi.wasm.$(Configuration)', 'IL', 'System.Private.CoreLib.dll')) false + + $(AfterMicrosoftNETSdkTargets);$(Crossgen2SdkOverrideTargetsPath) + $(AfterMicrosoftNETSdkTargets);$(MSBuildThisFileDirectory)WasiApp.ReadyToRun.targets + true diff --git a/src/mono/wasi/build/WasiApp.InTree.targets b/src/mono/wasi/build/WasiApp.InTree.targets index 09bda24fef051e..b7aa5f10076813 100644 --- a/src/mono/wasi/build/WasiApp.InTree.targets +++ b/src/mono/wasi/build/WasiApp.InTree.targets @@ -7,6 +7,7 @@ + @@ -23,6 +24,12 @@ + + + diff --git a/src/mono/wasi/build/WasiApp.ReadyToRun.targets b/src/mono/wasi/build/WasiApp.ReadyToRun.targets new file mode 100644 index 00000000000000..55fd76a0ad77a0 --- /dev/null +++ b/src/mono/wasi/build/WasiApp.ReadyToRun.targets @@ -0,0 +1,20 @@ + + + + + $([MSBuild]::NormalizePath('$(Crossgen2InBuildDir)', 'crossgen2$(ExeSuffix)')) + + + + + + + + diff --git a/src/native/corehost/wasihost/wasihost.cpp b/src/native/corehost/wasihost/wasihost.cpp index 9d34a920480a9d..b1b4068b26a43d 100644 --- a/src/native/corehost/wasihost/wasihost.cpp +++ b/src/native/corehost/wasihost/wasihost.cpp @@ -17,6 +17,21 @@ // Shared pal (path handling, CORE_ROOT/TPA helpers); header-only, so no corerun object is linked. #include "corerun.hpp" +// WASI R2R external-assembly probe, so the per-app host serves statically-composed R2R images instead +// of silently interpreting everything. Requires corerun.hpp above (pal::try_map_file_readonly). +#include "wasi_r2r_probe.hpp" + +namespace wasi_r2r +{ +// A ReadyToRun publish supplies strong definitions sized from its composite. Keep non-R2R app links +// working with a minimal placeholder. +extern "C" +{ + alignas(16) __attribute__((weak)) uint8_t g_wasi_r2r_image[64] = {}; + __attribute__((weak)) uint32_t g_wasi_r2r_image_cap = sizeof(g_wasi_r2r_image); +} +} + #include using pal::char_t; @@ -70,11 +85,38 @@ extern "C" __attribute__((weak)) int32_t GlobalizationNative_LoadICUData(const c static std::vector s_property_keys; static std::vector s_property_values; +// R2R external-assembly probe search dirs, captured before coreclr_initialize so the probe callback +// (invoked later by the runtime) can reach them. +static string_t s_r2r_app_path; +static string_t s_r2r_core_root; +static string_t s_r2r_core_libs; + static void log_error_info(const char* line) { std::fprintf(stderr, "%s\n", line); } +// Serves statically-composed R2R images (the composite plus per-assembly stubs) to the runtime, using +// the shared WASI probe. Returns false for everything else, so non-R2R assemblies load normally via the +// TPA list. +static bool HOST_CONTRACT_CALLTYPE external_assembly_probe( + const char* path, + void** data_start, + int64_t* size) +{ + const char* name = path; + const char* slash = ::strrchr(name, '/'); + if (slash != nullptr) + name = slash + 1; + + const char* const r2r_dirs[] = { + s_r2r_app_path.empty() ? nullptr : s_r2r_app_path.c_str(), + s_r2r_core_libs.empty() ? nullptr : s_r2r_core_libs.c_str(), + (s_r2r_core_root.empty() || s_r2r_core_root == s_r2r_app_path) ? nullptr : s_r2r_core_root.c_str() + }; + return wasi_r2r::WasiStaticR2RProbe(name, r2r_dirs, sizeof(r2r_dirs) / sizeof(r2r_dirs[0]), data_start, size); +} + // Include only the first instance of each simple assembly name (CoreCLR may otherwise prefer a // later ni over an earlier il). static string_t build_tpa(const string_t& core_root, const string_t& core_libraries) @@ -172,6 +214,11 @@ int main(int argc, char* argv[]) core_root = app_path; pal::ensure_trailing_delimiter(core_root); + // Capture the R2R probe search dirs (trailing-delimited) for the external_assembly_probe callback. + s_r2r_app_path = app_path; + s_r2r_core_root = core_root; + s_r2r_core_libs = core_libs; + string_t exe_path = pal::get_exe_path(); string_t tpa_list = build_tpa(core_root, core_libs); @@ -193,6 +240,7 @@ int main(int argc, char* argv[]) static host_runtime_contract host_contract = { sizeof(host_runtime_contract), nullptr }; host_contract.get_runtime_property = &get_runtime_property; host_contract.pinvoke_override = &callhelpers_pinvoke_override; + host_contract.external_assembly_probe = &external_assembly_probe; { std::stringstream ss; ss << "0x" << std::hex << (size_t)(&host_contract); diff --git a/src/tasks/Crossgen2Tasks/Microsoft.NET.CrossGen.targets b/src/tasks/Crossgen2Tasks/Microsoft.NET.CrossGen.targets index 4ebf5ebb634321..e2a311cad2e8e0 100644 --- a/src/tasks/Crossgen2Tasks/Microsoft.NET.CrossGen.targets +++ b/src/tasks/Crossgen2Tasks/Microsoft.NET.CrossGen.targets @@ -385,9 +385,14 @@ Copyright (c) .NET Foundation. All rights reserved. and do not replace them. IL PDBs are still required for debugging. Native PDBs emitted by the R2R compiler are only used for profiling purposes. --> - - - + + + + diff --git a/src/tasks/Crossgen2Tasks/PrepareForReadyToRunCompilation.cs b/src/tasks/Crossgen2Tasks/PrepareForReadyToRunCompilation.cs index 09bdade3adcf4a..43ca1856a04ec3 100644 --- a/src/tasks/Crossgen2Tasks/PrepareForReadyToRunCompilation.cs +++ b/src/tasks/Crossgen2Tasks/PrepareForReadyToRunCompilation.cs @@ -346,6 +346,13 @@ private TaskItem CreateReadyToRunFileToPublish( out string compilerOutputRelativePath, out string compilerOutputPath) { + if (isCompositeImage && Crossgen2ContainerFormat == "wasm" && + Crossgen2Tool?.GetMetadata(MetadataKeys.TargetOS) == "wasi") + { + // The WASI composition pipeline consumes this fixed composite image name. + relativePath = "composite-r2r.wasm"; + } + // Crossgen2 emits WebAssembly directly, while Mach-O composite output is an object file // that must be linked into the dylib published by the SDK. (string compilerExtension, string publishExtension) = Crossgen2ContainerFormat switch diff --git a/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleInfoReader.cs b/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleInfoReader.cs new file mode 100644 index 00000000000000..20b8a30228defa --- /dev/null +++ b/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleInfoReader.cs @@ -0,0 +1,502 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; + +namespace Microsoft.NET.WebAssembly.Webcil; + +internal sealed class WasmModuleInfo +{ + public uint DefinedFunctionCount { get; internal set; } + public uint ImportedFunctionCount { get; internal set; } + public IReadOnlyDictionary Exports => _exports; + public IReadOnlyList DefinedI32Functions => _definedI32Functions; + public IReadOnlyList<(bool Active, int? Offset, long PayloadOffset, uint Size)> DataSegments => _dataSegments; + public IReadOnlyList<(bool Active, int? Offset)> ElementSegments => _elementSegments; + + internal readonly Dictionary _exports = new(StringComparer.Ordinal); + internal readonly List _definedI32Functions = new(); + internal readonly List<(bool Active, int? Offset, long PayloadOffset, uint Size)> _dataSegments = new(); + internal readonly List<(bool Active, int? Offset)> _elementSegments = new(); + + public int? GetExportedI32Function(string name) + { + if (!_exports.TryGetValue(name, out (byte Kind, uint Index) export) || export.Kind != 0) + return null; + + if (export.Index < ImportedFunctionCount) + return null; + + uint definedIndex = export.Index - ImportedFunctionCount; + return definedIndex < _definedI32Functions.Count ? _definedI32Functions[(int)definedIndex] : null; + } +} + +internal sealed class WasmModuleInfoReader : WasmModuleReader +{ + private const byte FunctionImport = 0; + private const byte GlobalImport = 3; + private const byte I32Const = 0x41; + private const byte GlobalGet = 0x23; + private const byte End = 0x0b; + + private readonly WasmModuleInfo _info = new(); + + public WasmModuleInfoReader(Stream stream) : base(stream) + { + } + + public WasmModuleInfo Read() + { + if (!Visit()) + throw new BadImageFormatException("The input is not a valid WebAssembly module."); + return _info; + } + + protected override bool VisitSection(Section section, out bool shouldStop) + { + shouldStop = false; + try + { + switch (section) + { + case Section.Import: + ReadImports(); + break; + case Section.Function: + _info.DefinedFunctionCount = ReadULEB128(); + break; + case Section.Export: + ReadExports(); + break; + case Section.Element: + ReadElements(); + break; + case Section.Code: + ReadCode(); + break; + case Section.Data: + ReadDataSegments(); + shouldStop = true; + break; + } + } + catch (EndOfStreamException ex) + { + throw new BadImageFormatException($"Unexpected end of WebAssembly {section} section.", ex); + } + return true; + } + + private void ReadImports() + { + uint count = ReadULEB128(); + for (uint i = 0; i < count; i++) + { + ReadName(); + ReadName(); + byte kind = ReadByte(); + switch (kind) + { + case FunctionImport: + ReadULEB128(); + _info.ImportedFunctionCount++; + break; + case 1: + ReadByte(); + SkipLimits(); + break; + case 2: + SkipLimits(); + break; + case GlobalImport: + ReadByte(); + ReadByte(); + break; + case 4: + ReadByte(); + ReadULEB128(); + break; + default: + throw new BadImageFormatException($"Unsupported WebAssembly import kind {kind}."); + } + } + } + + private void ReadExports() + { + uint count = ReadULEB128(); + for (uint i = 0; i < count; i++) + { + string name = ReadName(); + byte kind = ReadByte(); + uint index = ReadULEB128(); + _info._exports[name] = (kind, index); + } + } + + private void ReadElements() + { + uint count = ReadULEB128(); + for (uint i = 0; i < count; i++) + { + uint flags = ReadULEB128(); + if (flags > 7) + throw new BadImageFormatException($"Unsupported WebAssembly element segment flags {flags}."); + + bool active = flags is 0 or 2 or 4 or 6; + int? offset = null; + if (flags is 2 or 6) + ReadULEB128(); // table index + if (active) + offset = ReadConstExpression(); + + if (flags is 1 or 2 or 3) + ReadByte(); // elemkind + else if (flags >= 4) + ReadByte(); // reftype + + uint elementCount = ReadULEB128(); + for (uint element = 0; element < elementCount; element++) + { + if (flags < 4) + { + ReadULEB128(); + } + else + { + byte opcode = ReadByte(); + SkipInstructionOperand(opcode); + RequireEnd(); + } + } + _info._elementSegments.Add((active, offset)); + } + } + + private void ReadDataSegments() + { + uint count = ReadULEB128(); + for (uint i = 0; i < count; i++) + { + uint flags = ReadULEB128(); + bool active = flags is 0 or 2; + int? offset = null; + if (flags == 2) + ReadULEB128(); // memory index + if (active) + offset = ReadConstExpression(); + else if (flags != 1) + throw new BadImageFormatException($"Unsupported WebAssembly data segment flags {flags}."); + + uint size = ReadULEB128(); + long payloadOffset = BaseStream.Position; + if (size > BaseStream.Length - payloadOffset) + throw new BadImageFormatException("WebAssembly data segment extends past the end of the file."); + BaseStream.Seek(size, SeekOrigin.Current); + _info._dataSegments.Add((active, offset, payloadOffset, size)); + } + } + + private void ReadCode() + { + uint count = ReadULEB128(); + for (uint i = 0; i < count; i++) + { + uint bodySize = ReadULEB128(); + long bodyEnd = checked(BaseStream.Position + bodySize); + if (bodyEnd > BaseStream.Length) + throw new BadImageFormatException("WebAssembly function body extends past the end of the file."); + uint localDeclarationCount = ReadULEB128(); + for (uint local = 0; local < localDeclarationCount; local++) + { + ReadULEB128(); + ReadByte(); + } + + int? value = null; + if (BaseStream.Position < bodyEnd && ReadByte() == I32Const) + { + int candidate = ReadSLEB32(); + if (BaseStream.Position < bodyEnd && ReadByte() == End && BaseStream.Position == bodyEnd) + value = candidate; + } + _info._definedI32Functions.Add(value); + BaseStream.Position = bodyEnd; + } + } + + private int? ReadConstExpression() + { + byte opcode = ReadByte(); + int? value = opcode == I32Const ? ReadSLEB32() : null; + if (opcode != I32Const) + SkipInstructionOperand(opcode); + RequireEnd(); + return value; + } + + private void SkipInstructionOperand(byte opcode) + { + switch (opcode) + { + case I32Const: + ReadSLEB32(); + break; + case GlobalGet: + case 0xd2: // ref.func + ReadULEB128(); + break; + case 0xd0: // ref.null + ReadByte(); + break; + default: + throw new BadImageFormatException($"Unsupported WebAssembly constant-expression opcode 0x{opcode:x2}."); + } + } + + private void SkipLimits() + { + uint flags = ReadULEB128(); + ReadULEB128(); + if ((flags & 1) != 0) + ReadULEB128(); + } + + private string ReadName() + { + uint length = ReadULEB128(); + byte[] bytes = ReadBytes(checked((int)length)); + return Encoding.UTF8.GetString(bytes); + } + + private int ReadSLEB32() + { + int value = 0; + int shift = 0; + byte current; + do + { + current = ReadByte(); + value |= (current & 0x7f) << shift; + shift += 7; + if (shift > 35) + throw new OverflowException(); + } while ((current & 0x80) != 0); + + if (shift < 32 && (current & 0x40) != 0) + value |= -1 << shift; + return value; + } + + private void RequireEnd() + { + if (ReadByte() != End) + throw new BadImageFormatException("WebAssembly constant expression is missing its end opcode."); + } + + private byte ReadByte() + { + int value = BaseStream.ReadByte(); + if (value < 0) + throw new EndOfStreamException(); + return (byte)value; + } + + private byte[] ReadBytes(int count) + { + byte[] bytes = new byte[count]; + int offset = 0; + while (offset < count) + { + int read = BaseStream.Read(bytes, offset, count - offset); + if (read == 0) + throw new EndOfStreamException(); + offset += read; + } + return bytes; + } +} + +internal static class WasmComponentFile +{ + public static void ReplaceFirstCoreModule(string componentPath, string modulePath, string outputPath) + { + byte[] component = File.ReadAllBytes(componentPath); + byte[] module = File.ReadAllBytes(modulePath); + if (component.Length < 8 || + component[0] != 0 || component[1] != 0x61 || component[2] != 0x73 || component[3] != 0x6d || + component[4] != 0x0d || component[5] != 0 || component[6] != 1 || component[7] != 0) + throw new BadImageFormatException($"{componentPath} is not a WebAssembly component."); + + using FileStream output = File.Create(outputPath); + output.Write(component, 0, 8); + int position = 8; + bool replaced = false; + while (position < component.Length) + { + int sectionStart = position; + byte sectionId = component[position++]; + uint sectionSize = ReadULEB128(component, ref position); + int payloadStart = position; + int sectionEnd = checked(payloadStart + (int)sectionSize); + if (sectionEnd > component.Length) + throw new BadImageFormatException("WebAssembly component section extends past the end of the file."); + + if (sectionId == 1 && !replaced) + { + output.WriteByte(sectionId); + WriteULEB128(output, checked((uint)module.Length)); + output.Write(module, 0, module.Length); + replaced = true; + } + else + { + output.Write(component, sectionStart, sectionEnd - sectionStart); + } + position = sectionEnd; + } + + if (!replaced) + throw new BadImageFormatException($"{componentPath} has no core-module section."); + } + + private static uint ReadULEB128(byte[] data, ref int offset) + { + uint value = 0; + int shift = 0; + while (offset < data.Length) + { + byte current = data[offset++]; + value |= (uint)(current & 0x7f) << shift; + if ((current & 0x80) == 0) + return value; + shift += 7; + if (shift >= 35) + throw new OverflowException(); + } + throw new EndOfStreamException(); + } + + private static void WriteULEB128(Stream stream, uint value) + { + do + { + byte current = (byte)(value & 0x7f); + value >>= 7; + if (value != 0) + current |= 0x80; + stream.WriteByte(current); + } while (value != 0); + } +} + +public static class WasiR2RComposition +{ + // Every self-installing image defines getWebcilSize and patchWebcilHeader; a component forwarding + // stub defines nothing else. + private const uint ComponentStubFunctionCount = 2; + + public static void InspectComposite(string path, out int functionCount, out int payloadSize) + { + WasmModuleInfo module = ReadModule(path); + functionCount = checked((int)module.DefinedFunctionCount); + payloadSize = checked((int)GetSelfInstallingPayloadSegment(module, path).Size); + } + + public static void InspectHost( + string path, + out int imageBase, + out int imageCapacity, + out int tableBase, + out int reservedTableStart) + { + WasmModuleInfo module = ReadModule(path); + imageBase = GetRequiredI32Export(module, "wasi_r2r_image_base"); + imageCapacity = GetRequiredI32Export(module, "wasi_r2r_image_cap"); + tableBase = GetRequiredI32Export(module, "wasi_r2r_table_base"); + reservedTableStart = int.MaxValue; + foreach ((bool Active, int? Offset) segment in module.ElementSegments) + { + if (segment.Active && segment.Offset.HasValue) + reservedTableStart = Math.Min(reservedTableStart, segment.Offset.Value); + } + if (reservedTableStart == int.MaxValue) + throw new BadImageFormatException($"{path} has no active element segment."); + } + + public static void ExtractComponentStubPayload(string inputPath, string outputPath) + { + using FileStream input = File.OpenRead(inputPath); + WasmModuleInfo module = new WasmModuleInfoReader(input).Read(); + (bool Active, int? Offset, long PayloadOffset, uint Size) segment = GetSelfInstallingPayloadSegment(module, inputPath); + if (module.DefinedFunctionCount != ComponentStubFunctionCount) + throw new BadImageFormatException($"{inputPath} is a code-carrying image, not a component forwarding stub."); + + input.Position = segment.PayloadOffset; + byte[] magic = ReadExactly(input, 4); + if (magic[0] != 0x57 || magic[1] != 0x62 || magic[2] != 0x49 || magic[3] != 0x4c) + throw new BadImageFormatException($"{inputPath} does not contain a WebCIL payload."); + + input.Position = segment.PayloadOffset; + using FileStream output = File.Create(outputPath); + CopyExactly(input, output, segment.Size); + } + + public static void ReplaceFirstCoreModule(string componentPath, string modulePath, string outputPath) => + WasmComponentFile.ReplaceFirstCoreModule(componentPath, modulePath, outputPath); + + private static WasmModuleInfo ReadModule(string path) + { + using FileStream stream = File.OpenRead(path); + return new WasmModuleInfoReader(stream).Read(); + } + + private static (bool Active, int? Offset, long PayloadOffset, uint Size) GetSelfInstallingPayloadSegment(WasmModuleInfo module, string path) + { + (bool Active, int? Offset, long PayloadOffset, uint Size)[] active = + new List<(bool Active, int? Offset, long PayloadOffset, uint Size)>(module.DataSegments) + .FindAll(segment => segment.Active) + .ToArray(); + if (active.Length != 1) + throw new BadImageFormatException($"Expected exactly one active WebCIL payload segment in {path}, found {active.Length}."); + if (!module.Exports.TryGetValue("patchWebcilHeader", out (byte Kind, uint Index) patchExport) || patchExport.Kind != 0) + throw new BadImageFormatException($"{path} is not a self-installing WebCIL image: patchWebcilHeader is missing."); + return active[0]; + } + + private static int GetRequiredI32Export(WasmModuleInfo module, string name) => + module.GetExportedI32Function(name) + ?? throw new BadImageFormatException($"The host does not export constant i32 function '{name}'."); + + private static byte[] ReadExactly(Stream input, int count) + { + byte[] bytes = new byte[count]; + int offset = 0; + while (offset < count) + { + int read = input.Read(bytes, offset, count - offset); + if (read == 0) + throw new EndOfStreamException(); + offset += read; + } + return bytes; + } + + private static void CopyExactly(Stream input, Stream output, uint count) + { + byte[] buffer = new byte[81920]; + long remaining = count; + while (remaining > 0) + { + int read = input.Read(buffer, 0, (int)Math.Min(buffer.Length, remaining)); + if (read == 0) + throw new EndOfStreamException(); + output.Write(buffer, 0, read); + remaining -= read; + } + } +} diff --git a/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleReader.cs b/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleReader.cs index d9e668bad65b26..0ae789ac184042 100644 --- a/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleReader.cs +++ b/src/tasks/Microsoft.NET.WebAssembly.Webcil/WasmModuleReader.cs @@ -28,6 +28,7 @@ public enum Section : byte Code, Data, DataCount, + Tag, } private readonly BinaryReader _reader; diff --git a/src/tasks/WasmAppBuilder/ComposeWasiReadyToRun.cs b/src/tasks/WasmAppBuilder/ComposeWasiReadyToRun.cs new file mode 100644 index 00000000000000..aa571add48aa62 --- /dev/null +++ b/src/tasks/WasmAppBuilder/ComposeWasiReadyToRun.cs @@ -0,0 +1,212 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using System.IO; +using System.Linq; +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; +using Microsoft.NET.WebAssembly.Webcil; + +namespace Microsoft.WebAssembly.Build.Tasks; + +public sealed class ComposeWasiReadyToRun : Task +{ + [Required, NotNull] + public string? CompositePath { get; set; } + + public bool InspectOnly { get; set; } + + public string? ComponentPath { get; set; } + public string? OutputDirectory { get; set; } + public string? OutputPath { get; set; } + public string? WasmToolsPath { get; set; } + public string? WasmMergePath { get; set; } + public string? WasmOptPath { get; set; } + public ITaskItem[] ComponentStubs { get; set; } = Array.Empty(); + public string? StubOutputDirectory { get; set; } + + [Output] + public int FunctionCount { get; private set; } + + [Output] + public int PayloadSize { get; private set; } + + [Output] + public ITaskItem[] FileWrites { get; private set; } = Array.Empty(); + + public override bool Execute() + { + try + { + WasiR2RComposition.InspectComposite(CompositePath, out int functionCount, out int payloadSize); + FunctionCount = functionCount; + PayloadSize = payloadSize; + + if (InspectOnly) + return true; + + ValidateCompositionArguments(); + NormalizeCompositionPaths(); + File.Delete(OutputPath!); + Directory.CreateDirectory(OutputDirectory!); + DeleteUnbundledModules(OutputDirectory!); + + string unbundleOutput = Path.Combine(OutputDirectory!, "unbundle-output.wasm"); + Run(WasmToolsPath!, $"component unbundle {Quote(ComponentPath!)} --module-dir {Quote(OutputDirectory!)} -o {Quote(unbundleOutput)}"); + string hostModule = Directory.GetFiles(OutputDirectory!, "*module0*.wasm").OrderBy(path => path, StringComparer.Ordinal).FirstOrDefault() + ?? throw new LogAsErrorException($"Unbundling {ComponentPath} produced no *module0*.wasm file in {OutputDirectory}."); + + WasiR2RComposition.InspectHost( + hostModule, + out int imageBase, + out int imageCapacity, + out int tableBase, + out int reservedTableStart); + int compositeTableEnd = checked(tableBase + FunctionCount); + if (compositeTableEnd > reservedTableStart) + throw new LogAsErrorException( + $"The composite needs table slots {tableBase}..{compositeTableEnd - 1}, " + + $"but the host's functions begin at {reservedTableStart}."); + if (PayloadSize > imageCapacity) + throw new LogAsErrorException( + $"The composite payload is {PayloadSize} bytes but the host staging buffer is only {imageCapacity} bytes."); + + Log.LogMessage(MessageImportance.High, + $"WASI R2R composition: imageBase={imageBase} tableBase={tableBase} " + + $"reservedSlots={reservedTableStart} compositeFuncs={FunctionCount} payload={PayloadSize} cap={imageCapacity}"); + + string shimWatPath = Path.Combine(OutputDirectory!, "shim.wat"); + string shimPath = Path.Combine(OutputDirectory!, "shim.wasm"); + File.WriteAllText(shimWatPath, CreateShimWat(imageBase, tableBase, PayloadSize)); + Run(WasmToolsPath!, $"parse {Quote(shimWatPath)} -o {Quote(shimPath)}"); + Run(WasmToolsPath!, $"validate --features all {Quote(shimPath)}"); + + string mergedPath = Path.Combine(OutputDirectory!, "merged.wasm"); + Run(WasmMergePath!, + $"-g --all-features --enable-gc {Quote(hostModule)} webcil {Quote(shimPath)} webcil " + + $"{Quote(CompositePath)} composite -o {Quote(mergedPath)}"); + + string finalModulePath = Path.Combine(OutputDirectory!, "final.wasm"); + Run(WasmOptPath!, + $"{Quote(mergedPath)} --all-features -g --simplify-globals -o {Quote(finalModulePath)}"); + + WasiR2RComposition.ReplaceFirstCoreModule(ComponentPath!, finalModulePath, OutputPath!); + Run(WasmToolsPath!, $"validate --features all {Quote(OutputPath!)}"); + + List fileWrites = new() { new TaskItem(OutputPath!) }; + if (ComponentStubs.Length > 0) + { + Directory.CreateDirectory(StubOutputDirectory!); + foreach (ITaskItem stub in ComponentStubs) + { + string inputPath = stub.GetMetadata("FullPath"); + string outputPath = Path.Combine( + StubOutputDirectory!, + Path.GetFileNameWithoutExtension(inputPath) + ".dll"); + WasiR2RComposition.ExtractComponentStubPayload(inputPath, outputPath); + TaskItem output = new(outputPath); + fileWrites.Add(output); + } + } + + FileWrites = fileWrites.ToArray(); + return !Log.HasLoggedErrors; + } + catch (LogAsErrorException ex) + { + DeleteFailedOutput(); + Log.LogError(ex.Message); + return false; + } + catch (Exception ex) + { + DeleteFailedOutput(); + Log.LogErrorFromException(ex, showStackTrace: true); + return false; + } + } + + private void ValidateCompositionArguments() + { + RequireFile(ComponentPath, nameof(ComponentPath)); + RequireFile(WasmToolsPath, nameof(WasmToolsPath)); + RequireFile(WasmMergePath, nameof(WasmMergePath)); + RequireFile(WasmOptPath, nameof(WasmOptPath)); + if (string.IsNullOrEmpty(OutputDirectory)) + throw new LogAsErrorException($"{nameof(OutputDirectory)} is required."); + if (string.IsNullOrEmpty(OutputPath)) + throw new LogAsErrorException($"{nameof(OutputPath)} is required."); + if (ComponentStubs.Length > 0 && string.IsNullOrEmpty(StubOutputDirectory)) + throw new LogAsErrorException($"{nameof(StubOutputDirectory)} is required when component stubs are provided."); + } + + private static void RequireFile(string? path, string propertyName) + { + if (string.IsNullOrEmpty(path) || !File.Exists(path)) + throw new LogAsErrorException($"{propertyName} does not name an existing file: '{path}'."); + } + + private void NormalizeCompositionPaths() + { + CompositePath = Path.GetFullPath(CompositePath); + ComponentPath = Path.GetFullPath(ComponentPath!); + OutputDirectory = Path.GetFullPath(OutputDirectory!); + OutputPath = Path.GetFullPath(OutputPath!); + WasmToolsPath = Path.GetFullPath(WasmToolsPath!); + WasmMergePath = Path.GetFullPath(WasmMergePath!); + WasmOptPath = Path.GetFullPath(WasmOptPath!); + if (StubOutputDirectory is not null) + StubOutputDirectory = Path.GetFullPath(StubOutputDirectory); + } + + private static void DeleteUnbundledModules(string outputDirectory) + { + foreach (string path in Directory.GetFiles(outputDirectory, "*module0*.wasm")) + File.Delete(path); + } + + private static string CreateShimWat(int memoryBase, int tableBase, int payloadSize) => + FormattableString.Invariant($""" + (module + (import "composite" "patchWebcilHeader" (func $patchWebcilHeader (param i32 i32))) + (global (export "__memory_base") i32 (i32.const {memoryBase})) + (global (export "__table_base") i32 (i32.const {tableBase})) + (func $start + i32.const {memoryBase} + i32.const {payloadSize} + call $patchWebcilHeader) + (start $start)) + """); + + private void Run(string tool, string arguments) + { + (int exitCode, string output) = Utils.TryRunProcess( + Log, tool, arguments, workingDir: OutputDirectory, silent: true); + if (exitCode != 0) + throw new LogAsErrorException($"{Path.GetFileName(tool)} failed with exit code {exitCode}:{Environment.NewLine}{output}"); + } + + private void DeleteFailedOutput() + { + if (InspectOnly || string.IsNullOrEmpty(OutputPath)) + return; + + try + { + File.Delete(OutputPath); + } + catch (IOException ex) + { + Log.LogWarning($"Failed to delete incomplete WASI R2R output '{OutputPath}': {ex.Message}"); + } + catch (UnauthorizedAccessException ex) + { + Log.LogWarning($"Failed to delete incomplete WASI R2R output '{OutputPath}': {ex.Message}"); + } + } + + private static string Quote(string value) => $"\"{value.Replace("\"", "\\\"")}\""; +} From b007fd28ec099ba2a2df30f58cca1c9109f1335f Mon Sep 17 00:00:00 2001 From: Larry Ewing Date: Mon, 28 Sep 2026 16:32:49 -0500 Subject: [PATCH 2/2] [wasi] Verify R2R tool download hashes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- eng/AcquireWasiR2RTools.targets | 14 ++++++++++++++ eng/wasm/AcquireWasmTool.proj | 17 +++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/eng/AcquireWasiR2RTools.targets b/eng/AcquireWasiR2RTools.targets index 53186f81ca8e5f..92803df79dd617 100644 --- a/eng/AcquireWasiR2RTools.targets +++ b/eng/AcquireWasiR2RTools.targets @@ -13,6 +13,16 @@ <_BinaryenArch Condition="'$(HostOS)' == 'osx' and '$(BuildArchitecture)' == 'arm64'">arm64 <_BinaryenArchiveName>binaryen-version_$(BinaryenVersion)-$(_BinaryenArch)-$(_WasiR2RToolOS) <_WasmToolsArchiveName>wasm-tools-$(WasmToolsVersion)-$(_WasiR2RToolArch)-$(_WasiR2RToolOS) + + <_BinaryenSha256 Condition="'$(WasmToolHostRid)' == 'linux-x64'">0a18362361ad05465118cd8eeb72edaeec89de6894bc283576ef4e07aa3babcc + <_BinaryenSha256 Condition="'$(WasmToolHostRid)' == 'linux-arm64'">e6ae6e09ac40f4e14bc5be6f687c58e2995c84170013975fa641809dd3b480a0 + <_BinaryenSha256 Condition="'$(WasmToolHostRid)' == 'osx-x64'">d3e2d1235b70c93c54b52eabc1625ea960965152218754f1f4eeb0f873c48e03 + <_BinaryenSha256 Condition="'$(WasmToolHostRid)' == 'osx-arm64'">79d3ab9f417d9e215f15f598f523d001a7d9ac1e59367e5c869fbdabd1cba72e + + <_WasmToolsSha256 Condition="'$(WasmToolHostRid)' == 'linux-x64'">4e2898f7ca3bd0536218ed9b7b36ff7b86954c57ae0e6272fde69728cbe01088 + <_WasmToolsSha256 Condition="'$(WasmToolHostRid)' == 'linux-arm64'">4a9a1c6a0885aaf17f433f2927bd61b1fe40069836f750f4e9f9ccd661106c65 + <_WasmToolsSha256 Condition="'$(WasmToolHostRid)' == 'osx-x64'">9e311880fe8fcb7ef55cfeea98dc6235440a0aa0a66373affdda4bc1dde47602 + <_WasmToolsSha256 Condition="'$(WasmToolHostRid)' == 'osx-arm64'">27b5f83d82ae8f2d961b5a87bdc024bca9db851a7b617d14065cec196fed29d7