JIT: Timeout in libraries-pgo tests

[jakobbotsch]

Quite a few timeouts in x86 (and some other runs). Smaller number in other platforms.
Example pipeline run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1179410&view=results
Example x86 console log: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-d5f58b4e6d2846888a/System.Memory.Tests/1/console.a5759d34.log?helixlogtype=result
Example x64 console log: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-7af6aae6ada343fb96/System.Memory.Tests/1/console.31e7ab50.log?helixlogtype=result

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• JIT: libraries PGO tests time out #59667

[AndyAyersMS]

I think we've seen this before: the DOTNET_JitInstrumentIfOptimizing=1 mode is very slow as it is forcing all optimized methods to have instrumentation.

Not sure what to do about this yet. We may get enough coverage with normal testing (eg all R2R methods+inlinees will end up at this tier if called enough), but maybe not.

[JulieLeeMSFT]

Failed in: runtime-coreclr libraries-pgo 20260222.1

Failed tests:

    net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open
    net11.0-windows-Release-x64-instrument_if_optimizing-Windows.10.Amd64.Open
    net11.0-linux-Release-x64-instrument_if_optimizing-AzureLinux.3.Amd64.Open
    - System.Memory.Tests.WorkItemExecution

Error message:

System.Memory.Tests: [Long Running Test] 'System.SpanTests.ReadOnlySpanTests.TestMatchLastIndexOfAny_ThreeByte', Elapsed: 00:02:08
['System.Memory.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

[JulieLeeMSFT]

Summary:
Long-standing timeout under PGO instrumentation in System.Memory.Tests.
StringSearchValuesTests and ReadOnlySpanTests take 2-3 minutes, causing the overall test to time out.
Failing since Sep 2025.

Failed in:

• runtime-coreclr libraries-pgo 20260301.1

Failed tests:

- net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open
- net11.0-windows-Release-x64-instrument_if_optimizing-Windows.10.Amd64.Open
- net11.0-linux-Release-x64-instrument_if_optimizing-AzureLinux.3.Amd64.Open

  - System.Memory.Tests.WorkItemExecution

[kg]

Summary:

Multiple tests time out under instrument_if_optimizing PGO stress mode on Windows.

Failed in (1):

• runtime-coreclr libraries-pgo 20260408.1

Console Log: Console Log

Source: runtime-coreclr libraries-pgo / net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open / System.Linq.Tests

Failed tests:

- net11.0-windows-Release-x64-instrument_if_optimizing-Windows.10.Amd64.Open

- net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open

  - System.Linq.Tests

  - System.Memory.Tests

  - System.Transactions.Local.Tests

Error Message:

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.TakeLastTests.TakeLast', Elapsed: 00:05:10

[Long Running Test] 'System.Linq.Tests.ToArrayTests.ToArrayShouldWorkWithSpecialLengthLazyEnumerables', Elapsed: 00:03:34

[Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:03:49

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.TakeLastTests.TakeLast', Elapsed: 00:07:10

[Long Running Test] 'System.Linq.Tests.ToArrayTests.ToArrayShouldWorkWithSpecialLengthLazyEnumerables', Elapsed: 00:05:34

[Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:05:49

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.TakeLastTests.TakeLast', Elapsed: 00:09:10

[Long Running Test] 'System.Linq.Tests.ToArrayTests.ToArrayShouldWorkWithSpecialLengthLazyEnumerables', Elapsed: 00:07:34

[Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:07:49

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.TakeLastTests.TakeLast', Elapsed: 00:11:11

[Long Running Test] 'System.Linq.Tests.ToArrayTests.ToArrayShouldWorkWithSpecialLengthLazyEnumerables', Elapsed: 00:09:34

[Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:09:49

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:15:02

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:17:02

   System.Linq.Tests: [Long Running Test] 'System.Linq.Tests.SkipLastTests.SkipLast', Elapsed: 00:19:02

['System.Linq.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Stack Trace:

N/A

Analysis:

All three tests time out under the instrument_if_optimizing PGO stress mode. DOTNET_JitInstrumentIfOptimizing=1 forces all optimized methods to have instrumentation, which is extremely slow. NULL exit codes because Helix kills the process when it exceeds timeout. Tracked by #120902, filed by @jakobbotsch.

• Generated by ci-pipeline-monitor/scripts/generate_report.py *

[JulieLeeMSFT]

Summary:

Tests run extremely slowly under PGO instrumentation stress modes (DOTNET_JitInstrumentIfOptimizing=1) and exceed the Helix work item timeout. Affects System.Linq.Tests, System.Memory.Tests, and System.Transactions.Local.Tests on x86 Windows.

Failed in (1):

• runtime-coreclr libraries-pgo 20260408.1

Console Log: Console Log

Source: runtime-coreclr libraries-pgo / net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open / System.Linq.Tests

Failed tests:

- net11.0-windows-Release-x64-instrument_if_optimizing-Windows.10.Amd64.Open

- net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open

  - System.Linq.Tests

  - System.Memory.Tests

  - System.Transactions.Local.Tests

Error Message:

Command timed out, and was killed

Stack Trace:

[Long Running Test] System.Linq.Tests.TakeLastTests.TakeLast, Elapsed: 00:11:11
[Long Running Test] System.Linq.Tests.SkipLastTests.SkipLast, Elapsed: 00:19:02

Analysis:

Known issue with PGO instrumentation causing extreme slowdown. Tests like TakeLast, SkipLast, and TestMatchLastIndexOfAny run for 15-20+ minutes before being killed. Tracked by #120902.

Generated by ci-pipeline-monitor/scripts/generate_report.py

[AndyAyersMS]

I think we can disable this mode for libraries tests (leaving it enabled for coreclr tests). Let me look into doing that.