Skip to content

fix(voice): wait for the in-flight handoff in wait_for_idle - #7930

Open
BlueX888 wants to merge 1 commit into
livekit:mainfrom
BlueX888:fix/wait-for-idle-handoff-activity-closed
Open

BlueX888 wants to merge 1 commit into
livekit:mainfrom
BlueX888:fix/wait-for-idle-handoff-activity-closed

Conversation

@BlueX888

Copy link
Copy Markdown
Contributor

fix(voice): wait for the in-flight handoff in wait_for_idle

What breaks

AgentSession.wait_for_idle() raises ActivityClosedError when called during a live handoff, even though the session is fine β€” the current activity is closed but the replacement swap has not happened yet. Callers that catch the error treat the session as shutting down: _ToolExecutor._deliver_reply's session-scoped path catches the spurious error and clears _pending_updates, so an AsyncToolset reply arriving during the handoff window is silently dropped (the tool ran, the reply never reached the user).

Fix

When the current activity is closed but a handoff is in flight, wait_for_idle awaits the in-flight handoff task under asyncio.shield and retries against the incoming activity, so a session-scoped waiter resolves against the new activity instead of failing. self._closing is added to the closing check so the public aclose() path still raises ActivityClosedError (not RuntimeError) when no handoff will arrive.

Deadlock guard: a caller that is itself part of the outgoing activity's drain graph (a tool parked in the old activity, ctx.foreground() after the handoff starts, scheduling/speech tasks, the activity reply/executor tasks) must NOT retarget β€” waiting there cycles (wait β†’ handoff β†’ drain β†’ caller). AgentActivity._is_draining_task identifies exactly those tasks and keeps the pre-fix fast ActivityClosedError; session-scoped toolset tasks and _deliver_reply are not in that set and do retarget. The guard is consulted only for the outgoing activity, so the retarget branch itself is untouched.

Tests

tests/test_wait_for_idle_handoff.py (deterministic asyncio.Event gating, FakeLLM, no sleep-timing):

  • test_wait_for_idle_retargets_during_live_handoff β€” red at base (ActivityClosedError: activity _agent_a is closing).
  • test_session_scoped_async_tool_reply_survives_handoff_window β€” red at base (scheduled call_ids=set()).
  • test_wait_for_idle_fails_fast_from_a_draining_tool β€” a non-cancellable tool parks after ctx.update(), then calls wait_for_idle after the handoff starts; fails at the previous revision with the cycle (TimeoutError), passes with the guard (ActivityClosedError, promptly).

At base: 2 failed / 1 passed (the drain-graph test passes at base because fast-raise was the base behavior). With the fix: 3 passed, repeated 20Γ— without flakes; the neighboring session/handoff/tool suites are green (15/155/218 across the relevant sets). An independent review verified the guard is conservative (nothing outside the drain graph gets the fast-raise), the retarget path is unchanged for session-scoped callers, and one residual edge is documented below.

Known residual (non-blocking): a draining tool that spawns a child task which itself waits cannot be enumerated by the guard and may still cycle; tracked as a follow-up.

The outgoing activity sets _closed at the start of aclose(), but _update_activity only swaps in the incoming activity after aclose() returns, and aclose() awaits the tool-executor drain of non-cancellable tools. A concurrent AgentSession.wait_for_idle() therefore saw a closed activity that was still current and re-raised ActivityClosedError even though the session was alive and the incoming activity was about to start.

Wait for the in-flight handoff task and retry against the incoming activity. Callers that the outgoing activity's own drain is awaiting (a function tool or foreground task inside it) keep the fast ActivityClosedError: waiting for the handoff there would deadlock (wait -> handoff -> drain -> caller). Session closing now also raises ActivityClosedError for the public aclose() path, which sets _closing without a _closing_task.

This also fixes session-scoped AsyncToolset replies: _ToolExecutor._deliver_reply resolves its target through session.wait_for_idle(), caught the error and discarded the buffered update and final output, so the new agent never spoke the result.
@BlueX888
BlueX888 requested a review from a team as a code owner October 18, 2026 17:35

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment on lines +1800 to 1805
current_task = asyncio.current_task()
if current_task is not None and activity._is_draining_task(current_task):
# a caller awaited by the outgoing activity's own drain (a function
# tool or foreground task inside it) must not wait for the handoff
# that is waiting for it β€” keep the fast failure instead
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ”΄ Nested idle waits deadlock agent handoffs

When a draining tool wraps wait_for_idle() in asyncio.wait_for, _is_draining_task misses the new waiter task. The tool waits for its own drain to finish, so the handoff hangs until the timeout or indefinitely.

Learn more

asyncio.wait_for(coro, timeout) schedules coro in a separate task. The outgoing activity's drain awaits non-cancellable tools via _ToolExecutor.drain. The new guard only recognizes the tool's own exe_task, not a child task it is awaiting, so the child waits on the handoff that needs the tool to finish. The new draining-tool test itself wraps the wait this way and expects ActivityClosedError, though this path instead reaches its timeout.

Example: A non-cancellable tool starts under agent A and calls await asyncio.wait_for(ctx.session.wait_for_idle(), 1) after A closes. The child waiter blocks on the A-to-B handoff; that handoff waits for the tool. The tool receives TimeoutError after one second instead of failing promptly; without a timeout, neither finishes.

Recommended fix: Track drain ownership across child tasks, or arrange for wait_for_idle to detect that an ancestor awaiting its task belongs to the outgoing drain graph. Cover both direct calls and asyncio.wait_for or create_task wrappers in a handoff test.

Devin Review


Was this helpful? React with πŸ‘ or πŸ‘Ž to provide feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant