InMemoryOrchestrationService: External events not delivered when durable timer is pending in Task.WhenAny

Description

When using DurableTaskTestHost (from Microsoft.DurableTask.InProcessTestHost), external events raised via RaiseEventAsync are not properly delivered to an orchestration that is waiting on Task.WhenAny(timerTask, externalEventTask). The orchestration hangs indefinitely instead of completing when the event arrives before the timer fires.

This makes it impossible to write integration tests for any orchestrator pattern that combines durable timers with external events (e.g., wait for approval with timeout).

Versions affected

Microsoft.DurableTask.InProcessTestHost 0.2.1-preview.1 through 0.2.3-preview.1 (latest as of this report)

Root cause analysis

The issue is in InMemoryInstanceStore.AddMessage() in InMemoryOrchestrationService.cs.

When an orchestrator creates a durable timer, the resulting TimerFiredEvent message has a future scheduledTime. In AddMessage(), this triggers the delayed path:

if (TryGetScheduledTime(message, out TimeSpan delay))
{
    // Not ready for this message yet - delay the enqueue
    _ = Task.Delay(delay).ContinueWith(t => this.AddMessage(message));
    return;  // exits without adding to MessagesJson
}

The sequence that causes the hang:

Orchestrator runs, creates a TimerCreatedEvent, and suspends. ReleaseLock sets state.IsLoaded = false.
The TimerFiredEvent message enters AddMessage() but is deferred via Task.Delay — it returns immediately without adding to MessagesJson.
An external event (EventRaisedEvent) arrives via RaiseEventAsync → AddMessage(). Since state.IsLoaded == false and no scheduled delay, the event is added to MessagesJson and the instance is scheduled to run.
The orchestrator replays, sees the EventRaisedEvent in the new messages, but the TimerFiredEvent hasn't arrived yet. The Durable Task Framework's Task.WhenAny implementation needs both tasks to be representable in the history to resolve. Since the timer hasn't fired, the framework re-suspends the orchestrator.
ReleaseLock is called — state.IsLoaded = false, MessagesJson was cleared (line 572), so the event message is gone.
When the timer finally fires and AddMessage is called again, the orchestrator replays but the external event message was already consumed and cleared. The orchestrator suspends again waiting for an event that was already delivered and discarded.

Reproduction

[Fact]
public async Task ExternalEvent_WithTimer_ShouldComplete()
{
    await using var host = await DurableTaskTestHost.StartAsync(tasks =>
    {
        tasks.AddOrchestrator("TestOrchestrator", async ctx =>
        {
            using var cts = ctx.CreateTimer(TimeSpan.FromMinutes(5), CancellationToken.None);
            var eventTask = ctx.WaitForExternalEvent<string>("MyEvent");
            var timerTask = cts.Task;
            var winner = await Task.WhenAny(eventTask, timerTask);
            return winner == eventTask ? "event" : "timeout";
        });
    });

    string instanceId = "test-instance";
    await host.Client.ScheduleNewOrchestrationInstanceAsync(
        "TestOrchestrator",
        new StartOrchestrationOptions { InstanceId = instanceId });

    await Task.Delay(TimeSpan.FromSeconds(3));
    await host.Client.RaiseEventAsync(instanceId, "MyEvent", "hello");

    using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(30));
    var metadata = await host.Client.WaitForInstanceCompletionAsync(
        instanceId, getInputsAndOutputs: true, cts.Token);

    // FAILS: OperationCanceledException — orchestration never completes
    Assert.Equal(OrchestrationRuntimeStatus.Completed, metadata.RuntimeStatus);
    Assert.Equal("event", metadata.ReadOutputAs<string>());
}

Expected behavior

The orchestration should complete immediately when the external event arrives, with the event task winning the Task.WhenAny.

Actual behavior

The orchestration hangs indefinitely. After the 30-second cancellation token fires, WaitForInstanceCompletionAsync throws OperationCanceledException.

Suggested fix

When an external event arrives for an instance that has pending deferred timer messages, the instance should still be able to process the event and complete. One possible approach: instead of deferring timer messages with Task.Delay + re-enqueue, add them to MessagesJson immediately and let the orchestrator dispatcher check scheduledTime before delivering them to the orchestrator.

Issue Triage

Classification: triage/needs-human-verification

Reasoning:
This issue describes a scenario where external events are lost when a durable timer is pending in Task.WhenAny within the InProcessTestHost. The root cause analysis is detailed, but after reviewing the current code in InMemoryOrchestrationService.cs, the described sequence may not produce a hang: when the orchestrator processes the EventRaisedEvent, it is saved to history via SaveState (even if the orchestrator suspends). On the next execution (when the timer fires), the event should be found in past events during replay, allowing WhenAny to resolve. This needs human verification with a running reproduction to confirm whether the described hang actually occurs on the current codebase (post-PRs #678, #703, #707 which fixed related race conditions).

Automated fix candidate: No

The root cause needs verification before any fix can be designed
If the issue is real, it involves subtle race condition semantics in the in-memory orchestration service that require careful human review

I stumbled upon this issue after attempting to implement an orchestration which handled an event. I didn't find it using the InProcessTestHost, I found it using our Cosmos backed implementation, most written by Claude, which was given the InProcessTestHost as a guide, so I don't suspect this is actually a race condition. One thing that I've noticed is that the inherited method CompleteTaskOrchestrationWorkItemAsync always receives null for the continuedAsNewMessage. I've attached our current implementation (works great as long as you're not using events.) I'd love it if someone on the team could review the implementation as it's a little difficult to figure out the correct things to do purely based off of the interfaces.

Using our implementation is injection friendly. It does need a named Singleton CosmosClient, where you provide the key name in the CosmosOrchestrationServiceOptions. Oh, and the extension method AddActivitiesAndOrchestrations takes an assembly which it will examine and automatically add all orchestrators and activities.

I do think our Cosmos provider is actually very useful when developing as it provides a bit of a middle ground between the in memory implementation and the fully managed azure implementation.

            services.AddCosmosOrchestrationProvider(new CosmosOrchestrationServiceOptions
            {
                CosmosKey = "default",
                DatabaseName = "yourdatabasename",
                ContainerName = "yourdatabasecontainer",
                MaxConcurrentTaskOrchestrationWorkItems = 1,
                MaxConcurrentTaskActivityWorkItems = 1,
                PollingInterval = TimeSpan.FromSeconds(2),
                WorkItemLeaseDuration = TimeSpan.FromSeconds(60),
            });

            services.AddDurableTaskWorker(worker =>
            {
                worker.UseCosmosProvider();

                worker.AddActivitiesAndOrchestrations(
                    typeof(DurableInjectionExtensions).Assembly);
            });

            services.AddDurableTaskClient(builder =>
            {
                builder.UseCosmosProvider();
                builder.RegisterDirectly();
            });

cosmosbackend.zip

With it being announced that the older task framework library is no longer officially supported, this issue really needs to be addressed. Unless I'm mistaken, there are no unit tests for the InProcessTestHost which demonstrate an event being processed.

We need to be able to effectively test our code, and this bug is a show stopper.

send2vinnie (@send2vinnie)

Triage summary — reproduces, but this is expected Durable Task behavior (not an event-delivery bug)

Thanks send2vinnie (@send2vinnie) for the detailed repro and AdmiralBond for the corroboration. We reproduced the hang on current main (which already includes the recent fixes and matches the 0.2.3-preview.1 you tested) and traced it end-to-end from worker/Core logs. The external event is not lost — the scenario hangs because the durable timer is never cancelled.

What actually happens

The repro orchestrator does Task.WhenAny(WaitForExternalEvent, CreateTimer(5 min)) and never cancels the timer. When the event is raised:

The EventRaisedEvent is delivered and the event wins the WhenAny (winner = EVENT) on the replay at ~3s.
But on that replay the worker sends 0 actions (no CompleteOrchestration), because a durable timer it created is still outstanding. Per Durable Task semantics, an orchestration is not marked Completed while any task it created — including a durable timer — is still outstanding (unfired and uncancelled).
So the instance stays Running until the TimerFiredEvent fires ~5 minutes later, then completes with output "event". Under a short timeout this looks like an indefinite hang.

Two controls confirm the mechanism:

Short timer (10s), event at 3s → completes at ~10s (timer-fire time) with output "event" (not "timeout"). If the event were lost, output would be "timeout" — it isn't; completion is merely deferred to the timer.
Same 5-min timer but cancelled on event-win (cts.Cancel()) → completes at ~3s with the event result; the event replay now emits [CompleteOrchestration] instead of 0 actions.

This is documented behavior — Durable timers (Microsoft Learn):

The Durable Task Framework doesn't change an orchestration's status to "Completed" until all outstanding tasks, including durable timer tasks, are either completed or canceled.

Why this isn't specific to InMemoryOrchestrationService

The "hold completion while a timer is outstanding" decision is made in the worker/DurableTask.Core path (TaskOrchestrationContextWrapper.CreateTimer forwards the CancellationToken straight into Core's CreateTimer), which is identical across backends. A real Azure Storage / DTS backend would likewise stay Running until the 5-minute timer fires; the in-memory store's Task.Delay-based scheduling just makes the wait exactly as long as the timer, which is what made it so visible. The suggested change (enqueue the timer immediately, gate on scheduledTime) wouldn't change the outcome: the TimerFiredEvent still wouldn't be delivered until its scheduled time, so the event replay would still emit 0 actions.

Fix (orchestrator code): cancel the timer when the event wins — the canonical when-any timeout pattern:

using var cts = new CancellationTokenSource();
Task<string> eventTask   = context.WaitForExternalEvent<string>("MyEvent");
Task         timeoutTask = context.CreateTimer(TimeSpan.FromMinutes(5), cts.Token);

Task winner = await Task.WhenAny(eventTask, timeoutTask);
if (winner == eventTask)
{
    cts.Cancel();            // lets the orchestration complete promptly
    return await eventTask;
}
return "timeout";

With this, the same scenario completes at ~3s (at event time) with the event's payload.

Follow-up

Good call-out that there was no external-event coverage in InProcessTestHost.Tests. We're adding tests for WaitForExternalEvent (plain delivery + the when-any timeout pattern) plus a short doc note on the cancellation requirement (PR to follow).

Given the above, we plan to close this as by design. If you see completion not occur even after cancelling the timer (or with no pending timer at all), that's a different issue — please let us know and we'll dig in.

test(InProcessTestHost): address review feedback on external-event tests

test(InProcessTestHost): external-event + durable-timer timeout cover…

Thanks for the great info. I'd suggest adding a WaitForExternalEventAsync method that instead of throwing if it times out, should return some sort of Result with a property that lets the caller know whether the event or timer was fired.

Navigation Menu

Repository navigation

InMemoryOrchestrationService: External events not delivered when durable timer is pending in Task.WhenAny #713

Description

Versions affected

Root cause analysis

Reproduction

Expected behavior

Actual behavior

Suggested fix

Issue Triage

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

InMemoryOrchestrationService: External events not delivered when durable timer is pending in Task.WhenAny #713

Description

Description

Versions affected

Root cause analysis

Reproduction

Expected behavior

Actual behavior

Suggested fix

Activity

github-actions commented on Apr 25, 2026

Issue Triage

AdmiralBond commented on May 25, 2026

AdmiralBond commented on Jun 1, 2026

YunchuWang commented on Jul 15, 2026

AdmiralBond commented on Jul 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions