Repository navigation
[wasm] [spmi] Collect SPMI contexts from libraries via crossgen2 - #135296
Open
adamperlin wants to merge 7 commits into
Open
adamperlin wants to merge 7 commits into
adamperlin wants to merge 7 commits into
Conversation
The browser/wasm crossgen2 SuperPMI collection compiled exactly one assembly, the wasm-built System.Private.CoreLib.dll, which is a very narrow slice of JIT coverage for the wasm target: one Helix partition, one MCH, and no coverage of any shared framework library. Extend it to the full browser-wasm managed assembly set, sourced from the browser-wasm CoreCLR runtime pack: - The browser_wasm_win build job now builds `clr+libs+packs` instead of `clr.corelib`, since the runtime pack layout is produced by the packs.product subset. It uploads only the runtime pack directory rather than all of artifacts/bin. - superpmi_collect_setup.py flattens the runtime pack's managed assemblies (lib/<tfm>/*.dll plus native/System.Private.CoreLib.dll, which is where CoreCLR runtime packs keep corelib) into the set to partition over. - crossgen2 has to resolve references against the target's assemblies rather than the host Core_Root, so add an explicit `-crossgen2_reference_directory` option to superpmi.py collect. The setup script stages the assemblies into the correlation payload and reports the folder name via a pipeline variable that superpmi-collect.proj turns into the new option. Host-target collections are unaffected; the option defaults to Core_Root. - The corelib-specific special cases in superpmi_collect_setup.py are replaced by conditions on cross-targeting. The collection is renamed from `corelib` to `libraries`, matching the other crossgen2 collections. Because it now shares collectionName/collectionType/osGroup/archType/buildConfig with the host-target windows-x64 libraries collection, published artifact names gain a target suffix and the job display name includes the target. The partition size becomes a template parameter (default 25 MB) so the wasm job can use a smaller value and spread its ~30 MB of assemblies over several Helix work items. Note that the rename orphans corelib.crossgen2.windows.x64.checked.mch under <jit_ee_version>/browser/x64/. Replay downloads every MCH under that prefix, so the stale file has to be deleted from the collections container by hand. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c6bbfae-eed7-4545-8f2a-5cfb15d62df4 Collect the wasm crossgen2 SPMI collection in composite mode The wasm collection only produced 82,870 method contexts. Most of the shortfall is structural: crossgen2 compiles each assembly in its own version bubble, so generic instantiations that cross an assembly boundary -- above all instantiations over CoreLib types, which is most of them -- are skipped as cross-bubble and never reach the JIT. Add a `--crossgen2_composite` option to `superpmi.py collect`. In that mode a work item's assemblies are compiled in a single crossgen2 invocation with `--composite`, and every other managed assembly in the reference set is passed as an unrooted input (`-u`). Unrooted inputs join the version bubble without being rooted, so each work item sees the whole framework as one bubble while still only rooting its own partition. Measured locally over the equivalent x64 shared framework (181 assemblies, real crossgen2 + SuperPMI collection): non-composite 93,077 contexts 1000 MB 3m20s composite 123,347 contexts 677 MB 5m31s so a third more contexts, and the merged MCH is smaller. The composite collection replays clean. The collection stays partitioned. Every partition gets the full bubble either way, and the merge step deduplicates the overlap between them, so partitioning costs redundant compilation but keeps the work parallel and keeps a single failure from losing the whole collection. One 5 MB partition measured 95s locally, so five in parallel is about the same wall time as one unpartitioned run. The per-work-item timeout goes to 4:00 for composite since each item now compiles considerably more. Two supporting helpers: - `is_managed_assembly` filters native binaries out of the unrooted set. crossgen2's unrooted-input loop loads modules without tolerating failure, and reference directories such as Core_Root contain native DLLs that match the reference globs (System.IO.Compression.Native.dll). - `get_assembly_simple_name` is used both to exclude rooted assemblies from the unrooted set and to reject a work item containing duplicate assembly simple names up front. crossgen2 keys modules by simple name and aborts on a collision; partition payloads preserve directory structure precisely because duplicate file names are expected for some collections. The non-composite path is unchanged: collecting the same assemblies without the flag produces an identical context count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c6bbfae-eed7-4545-8f2a-5cfb15d62df4 Pick the highest TFM if a runtime pack has more than one Staging the runtime pack assumed exactly one target framework directory under `lib` and hard-failed otherwise, which is fragile if the pack ever grows another one. Select the highest `netX.Y` instead, comparing the version numerically so `net10.0` beats `net9.0`, and sort anything unrecognized below all of them. Staging every target framework directory is not an option: the assemblies would collide on assembly simple name, which crossgen2 rejects outright in composite mode. Missing the `lib` directory entirely is still an error, just a clearer one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c6bbfae-eed7-4545-8f2a-5cfb15d62df4 Cite the measured wasm collection numbers for composite mode The comment quoted the local x64 proxy measurement taken before the collection had run in CI. Replace it with the actual result for this collection: 82,870 -> 109,684 contexts, and 177,230 collected across partitions before merge deduplication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c6bbfae-eed7-4545-8f2a-5cfb15d62df4
Fix collection template expansion and distinguish cross-target Helix telemetry. Quote unrooted crossgen2 inputs and launch without a shell to preserve paths containing spaces. Clarify composite rooting comments and remove the one-time migration note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cab095ec-ed42-4681-bb05-39e3b20b3aa7
github-actions
Bot
added
the
area-CodeGen-coreclr
adamperlin
changed the title
|
Azure Pipelines: Successfully started running 5 pipeline(s). 11 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
It restores the removed JitWasmNyiToR2RUnsupported option, which is now ineffective and contradicts current WASM JIT behavior.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Extends WASM SuperPMI collection from CoreLib to composite crossgen2 compilation of the full runtime pack.
Changes:
- Stages and partitions WASM runtime-pack assemblies.
- Adds composite crossgen2 collection and target-reference support.
- Updates pipeline artifacts, naming, timeouts, and replay configuration.
| File | Description |
|---|---|
src/coreclr/scripts/superpmi.py |
Implements composite crossgen2 collection. |
src/coreclr/scripts/superpmi-replay.proj |
Updates the WASM collection reference. |
src/coreclr/scripts/superpmi-collect.proj |
Passes composite and reference options. |
src/coreclr/scripts/superpmi_collect_setup.py |
Stages target runtime-pack assemblies. |
eng/pipelines/coreclr/templates/superpmi-send-to-helix.yml |
Forwards new collection settings. |
eng/pipelines/coreclr/templates/superpmi-collect-pipeline.yml |
Builds and collects the WASM runtime pack. |
eng/pipelines/coreclr/templates/superpmi-collect-job.yml |
Adds WASM artifact dependencies and parameters. |
eng/pipelines/coreclr/templates/run-superpmi-collect-job.yml |
Wires staging, naming, and Helix execution. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca214a40-e41a-4bf5-a3d8-f57ba9c01cf6
Contributor
Author
|
/azp run runtime-coreclr superpmi-collect-test |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 68180ef1-f87f-43d6-9b86-e2e924240351
JulieLeeMSFT
left a comment
JulieLeeMSFT
left a comment
Member
There was a problem hiding this comment.
Nit: please add what you have validated in the PR description.
| liveLibrariesBuildConfig: Release | ||
| collectionType: pmi | ||
| collectionName: libraries | ||
| - ${{ if false }}: |
Member
There was a problem hiding this comment.
Why is it wrapped with ${{ if false }}? It disables non-wasm collections. The production superpmi-collect.yml extends this template, so merging would stop MCH collection for all other platforms.
Contributor
Author
There was a problem hiding this comment.
Sorry I should have added a PR comment about this. This is temporary in order to test wasm collections in CI. I'll make sure to remove it before merging.
Contributor
Author
|
/azp run runtime-coreclr superpmi-collect-test |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
Author
|
/azp run runtime-coreclr superpmi-collect-test |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a continuation of #132538 which @AndyAyersMS opened; The only real change is addressing review feedback. The overall idea here is to run superpmi collection over composite crossgen2 compilations of library assemblies to significantly increase the number of method contexts for Wasm.
Validation
superpmi.py replay