[release/10.0] JIT: Add visit budget checks to IsMonotonicallyIncreasing and ComputeDoesOverflow

[EgorBo]

Backport of #125156 to release/10.0.

Customer Impact

A single method can take ~10 seconds and ~7.9 GB of JIT arena memory to compile, on the thread that first calls it. Reported in #133147: a containerized ASP.NET Core service was OOM-killed by the cgroup limit on the first request that reached one method, returning BadGateway. The memory is JIT arena, not the managed heap, so it is invisible to GC counters and eeheap, which makes it very hard to diagnose.

The trigger is a method with a flattened control flow graph (dispatcher switch loops, opaque predicates, backward branches inside finally handlers), as produced by commercial obfuscators with control-flow obfuscation enabled. The reporter hit it in production; the obfuscator output is non-deterministic, so it appeared and disappeared between releases with no source change.

The generated code is correct. Only JIT throughput and JIT memory are affected.

Regression

No, this is a long-standing issue, not a regression from a previous release. It reproduces on 10.0.10 and 10.0.11.

Testing

Reproduced with the reduced IL repro from #133147 (a 2415-byte-IL async state machine MoveNext), win-x64, DOTNET_TieredCompilation=0:

JIT	Compile time	Peak working set
Stock 10.0.11	10064 ms	7850 MB
release/10.0 built locally, without this change	9303 ms	7850 MB
release/10.0 with this change	5 ms	26 MB

DOTNET_JitTimeLogCsv on 10.0.11 attributes 40,743,816,978 of 40,767,907,863 total cycles (99.94%) to the Optimize index checks phase, with Total Bytes Allocated of 8.2 GB for a graph of only 194 basic blocks. That isolates the cost to range check analysis rather than to graph size.

Causality was confirmed independently on main, which already carries #125156 and compiles the same repro in 39 ms / 47 MB. Reverting only these two budget checks on main brings the pathology back and amplifies it: the compile did not finish within 13 minutes and reached 45 GB working set / 191 GB committed.

jit-format is clean.

Risk

Low. The change adds the same m_nVisitBudget guard that ComputeRange already uses, so it only takes effect once the existing visit budget is exhausted, which does not happen in ordinary code.

Both early exits return the conservative answer:

• IsMonotonicallyIncreasing returns false, meaning "not proven monotonic".
• ComputeDoesOverflow returns true, meaning "may overflow".

Both block bounds-check elimination rather than enabling it, so no unsound optimization can result. The checks are placed before GetSearchPath()->Set(...) so no stale search-path entries are left behind on early exit.

This has been in main since March 2026 (#125156) with no follow-up issues.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

The change is a minimal, conservative early-exit guard consistent with existing visit-budget usage and does not introduce observable correctness risk (only potentially reduces an optimization when over budget).

Pull request overview

Backport to release/10.0 of a JIT range-check analysis hardening change that prevents pathological compilation-time and JIT arena memory blowups by enforcing the existing visit-budget mechanism in additional recursive helpers.

Changes:

• Add m_nVisitBudget guard/decrement to RangeCheck::IsMonotonicallyIncreasing to bound recursive UD-chain walking.
• Add m_nVisitBudget guard/decrement to RangeCheck::ComputeDoesOverflow to bound recursive overflow checks.
• Ensure both early-exit paths return conservative results and occur before mutating the search-path state.

File summaries

File	Description
src/coreclr/jit/rangecheck.cpp	Adds visit-budget checks to two recursive range-analysis helpers to prevent extreme JIT throughput/memory pathologies on obfuscated control flow.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[EgorBo]

PTAL @AndyAyersMS, backport of #125156