Skip to content

JIT: Treat tailcalled delegate receivers as escaping - #136408

Merged
EgorBo merged 3 commits into
dotnet:mainfrom
EgorBo:fix-136373-delegate-tailcalls
Oct 29, 2026
Merged

EgorBo merged 3 commits into
dotnet:mainfrom
EgorBo:fix-136373-delegate-tailcalls

Conversation

@EgorBo

@EgorBo EgorBo commented Oct 24, 2026

Copy link
Copy Markdown
Member

Fixes #136373.

Treat delegate receivers of explicit tail calls as escaping. Tailcall helpers require the original delegate and can use it after the caller's frame returns, so stack allocation is unsafe.

Adds a small F# regression test that tailcalls a freshly created delegate with enough arguments to require the helpers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7bb954bf-2cce-4297-acb5-e4f5e2cb32b2
@github-actions github-actions Bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Oct 24, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

EgorBo and others added 2 commits October 24, 2026 03:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7bb954bf-2cce-4297-acb5-e4f5e2cb32b2
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7bb954bf-2cce-4297-acb5-e4f5e2cb32b2
@EgorBo

EgorBo commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

PTAL @dotnet/jit-contrib trivial fix, no diffs .
We just conservatively give up on explicit tail calls in escape analysis.

@EgorBo
EgorBo requested a review from a team October 26, 2026 10:47

@adamperlin adamperlin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lgtm from what I understand. Is it worth adding a regression test for the second failure case?

@adam-hebert

Copy link
Copy Markdown

Thanks for the quick fix! I reported #136373. While narrowing it down we hit a second shape of the same bug, and I'd like to make sure this PR covers it before it merges.

The second shape: no factory in tail position. A plain C# helper (Alg.Let(value, f) => f(value), no tail prefix) gets inlined, guarded devirtualization inlines the lambda behind it, and the Func<W, decimal, string> that the lambda builds ends up stack-allocated in Root.Invoke's frame. Root.Invoke tail-calls that delegate's Invoke through the tailcall helpers (lea rcx, [rbp-0x78] into IL_STUB_StoreTailCallArgs, with the delegate's MethodTable, _target and _methodPtr at [rbp-0x78], [rbp-0x70] and [rbp-0x60]). Because Root.Invoke is itself running under a tail-call dispatcher (Top → Outer → Root, all tail. callvirt), DispatchTailCalls returns at once and the frame is released, and the outer dispatcher then calls through the dangling delegate. It needs DOTNET_FastTailCalls=0 to force the helpers on x64, but it's the same pattern F# code hits when the arguments don't fit a fast tail call.

Results (Windows 11 x64, 2026-10-23):

Runtime Default nowarm (no GDV) outer (one dispatcher fewer) direct (no dispatcher) with DOTNET_JitObjectStackAllocation=0
10.0.13 Fatal error. Internal CLR error. (0x80131506), 5 of 5 runs returns returns returns returns
11.0.0-rc.2.26504.105 crash, 3 of 3 runs returns returns
9.0.21 returns

All with DOTNET_FastTailCalls=0; without it, it returns. 11 RC2 was run with DOTNET_ROLL_FORWARD=Major.

Repro: two projects, 96 lines (click to expand)

Helpers/Helpers.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net9.0;net10.0</TargetFrameworks>
    <Optimize>true</Optimize>
    <DebugType>none</DebugType>
  </PropertyGroup>
</Project>

Helpers/Alg.cs

using System;

public static class Alg
{
    // A plain C# helper: an ordinary call to the delegate, no tail prefix, so the JIT can inline it.
    public static TResult Let<T, TResult>(T value, Func<T, TResult> f) => f(value);
}

App/App.fsproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net9.0;net10.0</TargetFrameworks>
    <Optimize>true</Optimize>
    <Tailcalls>true</Tailcalls>
    <DebugType>none</DebugType>
    <RollForward>LatestPatch</RollForward>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="Program.fs" />
    <ProjectReference Include="../Helpers/Helpers.csproj" />
  </ItemGroup>
</Project>

App/Program.fs

// Second failure mode: a delegate stack-allocated behind a guarded devirtualization, used after its frame is gone.
// Run with DOTNET_FastTailCalls=0, which sends every explicit tail call through the tailcall helpers. Top.Call's tail
// call into Outer.Call, and Outer.Call's into Root.Invoke, then make Root.Invoke run under a tail-call dispatcher.
//   (no argument)  warm Alg.Let up first (so its delegate call gets a profile and can be GDV'd), then call Top.Call
//   nowarm         call Top.Call without warming Alg.Let up (the delegate comes back from a real call, on the heap)
//   outer          warm up, then call Outer.Call (one dispatcher level fewer)
//   direct         warm up, then call Root.Invoke directly (no dispatcher above it)
open System
open System.Runtime.CompilerServices
open System.Threading

[<Sealed>]
type W(full: decimal) =
    member _.Full = full

let levelOf (reversed: (decimal * string)[]) (w: W) (win: decimal) =
    let basis = win / w.Full
    match reversed |> Array.tryFind (fun (minimum, _) -> minimum <= basis) with
    | Some (_, level) -> level
    | None -> "none"

type Factory() =
    // The delegate is created inside the lambda handed to Alg.Let. The lambda captures `entries` (an object), so the
    // JIT can guess its target and inline it once Alg.Let's delegate call has a profile.
    [<MethodImpl(MethodImplOptions.AggressiveInlining)>]
    static member Make(entries: (decimal * string)[]) : Func<W, decimal, string> =
        Alg.Let(entries |> Array.sortBy fst, Func<(decimal * string)[], Func<W, decimal, string>>(fun sorted ->
            let reversed = Array.rev sorted
            Func<W, decimal, string>(fun w win -> if entries.Length = 0 then "none" else levelOf reversed w win)))

type Root(entries: (decimal * string)[]) =
    // NoInlining only keeps the F# compiler from copying this into Outer; it doesn't restrict what the JIT inlines here.
    [<MethodImpl(MethodImplOptions.NoInlining)>]
    member _.Invoke(w: W, win: int64) : string = Factory.Make(entries).Invoke(w, decimal win)   // tail. callvirt Invoke

type Outer(root: Root) =
    member _.Call(w: W, win: int64) : string = root.Invoke(w, win)                                // tail. callvirt

type Top(outer: Outer) =
    member _.Call(w: W, win: int64) : string = outer.Call(w, win)                                 // tail. callvirt

let say (s: string) = Console.Out.WriteLine s; Console.Out.Flush()

[<EntryPoint>]
let main argv =
    let mode = if argv.Length > 0 then argv[0] else "dispatcher"
    let fastTailCalls = Environment.GetEnvironmentVariable "DOTNET_FastTailCalls"
    say $"runtime {Environment.Version} {Runtime.InteropServices.RuntimeInformation.OSArchitecture}, mode {mode}, DOTNET_FastTailCalls={fastTailCalls}"
    let entries = [| 0M, "Small"; 5M, "Big" |]
    let w = W 100M
    if mode <> "nowarm" then
        // Calls in rounds with pauses, so Alg.Let tiers up (instrumented Tier0, then Tier1) in the background.
        for _ in 1 .. 160 do
            for _ in 1 .. 250 do Factory.Make(entries).Invoke(w, 700M) |> ignore
            Thread.Sleep 50
        say "warmed Alg.Let up"
    let root = Root entries
    let outer = Outer root
    let top = Top outer
    RuntimeHelpers.PrepareMethod(typeof<Top>.GetMethod("Call").MethodHandle)
    RuntimeHelpers.PrepareMethod(typeof<Outer>.GetMethod("Call").MethodHandle)
    RuntimeHelpers.PrepareMethod(typeof<Root>.GetMethod("Invoke").MethodHandle)
    say "calling"
    let mutable result = ""
    for _ in 1 .. 200000 do
        result <- if mode = "direct" then root.Invoke(w, 700L) elif mode = "outer" then outer.Call(w, 700L) else top.Call(w, 700L)
    say $"returned {result}"
    0

Run:

cd App
dotnet build -c Release
set DOTNET_FastTailCalls=0
bin\Release\net10.0\App.exe            -> Fatal error. Internal CLR error. (0x80131506)
bin\Release\net10.0\App.exe nowarm     -> returned Big
bin\Release\net10.0\App.exe outer      -> returned Big
bin\Release\net10.0\App.exe direct     -> returned Big
bin\Release\net9.0\App.exe             -> returned Big

Since the delegate here is the receiver of the tail-called Invoke, I'd expect this PR to cover it, but it reaches that point through a different path (an inlined C# helper plus GDV rather than an inlined factory), so a quick check would be much appreciated.

Backport request. Could this be considered for servicing in release/10.0, and release/11.0 too, since 11 RC2 still crashes? For the servicing template:

  • Customer impact: valid IL that the F# compiler emits by default in Release (--tailcalls+) kills the process with an access violation that can't be caught and leaves no managed stack trace. We hit it in our game-server code (a jackpot reset path, in Release builds, before it reached production) and had to move a release back from .NET 10 to .NET 8 to ship it.
  • Regression: yes. It works on .NET 8 and 9 and fails on every .NET 10 release from 10.0.0 to 10.0.13, on Windows x64 and Linux x64 (details in JIT: explicit tail call to Invoke on a stack-allocated delegate passes the delegate's _target to the tailcall helpers (access violation, .NET 10) #136373).
  • Risk: low, as I understand the change: escape analysis gives up on explicit tail calls, with no SPMI diffs.
  • Workaround: DOTNET_JitObjectStackAllocation=0 (environment only) or --tailcalls-. Both are process- or assembly-wide, and anyone who doesn't know about the bug won't find it from the crash.

Once this is merged, I'm happy to run both repros and our real crash path against a daily build and report back here, if that helps the backport.

@EgorBo

EgorBo commented Oct 29, 2026 •

Copy link
Copy Markdown
Member Author

this shape is covered too: Root.Invoke still explicitly tail-calls the delegate, so the fix prevents its stack allocation regardless of how it was created.

Backport request. Could this be considered for servicing in release/10.0, and release/11.0 too, since 11 RC2 still crashes?

I'll backport it to 11.0 (although, it won't make it to GA as it's too late, only with the first servicing patch), but the bar is high for 10.0, given there is a workaround I am not sure it can meet it.

@EgorBo
EgorBo merged commit a01cb86 into dotnet:main Oct 29, 2026
134 of 136 checks passed
@adam-hebert

Copy link
Copy Markdown

Excellent, I'm glad it covers that case too, and thank you for taking it to 11.0!

On 10.0, I'd like to make the case once, since I think the workaround is weaker than it looks:

  • It's a regression in the LTS. The same code returns on .NET 9 and crashes on 10.0.12 and 10.0.13 with default settings (no environment variables), because delegate stack allocation is new in 10. With .NET 8 and 9 reaching end of support on November 10, 10.0 is the release F# shops are moving to right now, and 11.0 is STS.
  • The workaround can only be set per process. DOTNET_JitObjectStackAllocation=0 has no runtimeconfig equivalent, so it has to be set in every host's environment (container images, IIS app pools, test runners, CI agents). It's missed easily, and it turns off object stack allocation for the whole process, one of 10.0's headline optimizations. TieredPGO=false (which runtimeconfig can set) is only partial: an inlined factory still crashes.
  • Nothing points users to the workaround. The failure is Fatal error. Internal CLR error. (0x80131506): an access violation that can't be caught, with no managed stack. Valid IL that the F# compiler emits by default in Release (--tailcalls+) triggers it, so a team sees a process crash in ordinary closure code and has nothing to search for.
  • Where we hit it: a production game server's code (caught before it shipped), plus a test suite that only crashed in Release. We've kept that server on .NET 8 because of it.

If it would help the servicing bar, I'm glad to verify the fix on a 10.0 daily or servicing build against both repros and our production code path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JIT: explicit tail call to Invoke on a stack-allocated delegate passes the delegate's _target to the tailcall helpers (access violation, .NET 10)

3 participants