Skip to content

fix(webrtc-sys): capture add_ice_candidate completion state by value - #1392

Merged
stephen-derosa merged 8 commits into
livekit:mainfrom
cliqer:fix/add-ice-candidate-dangling-capture
Sep 8, 2026
Merged

fix(webrtc-sys): capture add_ice_candidate completion state by value#1392
stephen-derosa merged 8 commits into
livekit:mainfrom
cliqer:fix/add-ice-candidate-dangling-capture

Conversation

@cliqer

@cliqer cliqer commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Problem

PeerConnection::add_ice_candidate (webrtc-sys/src/peer_connection.cpp) passes libwebrtc a completion lambda that captures ctx and on_complete by reference:

peer_connection_->AddIceCandidate(
    candidate->release(), [&](const webrtc::RTCError& err) {
      on_complete(std::move(ctx), to_error(err));
    });

PeerConnectionInterface::AddIceCandidate with a callback completes asynchronously on the signaling thread, and libwebrtc defers it behind the operations chain whenever the chain is busy (typically while a SetRemoteDescription is still in flight). In that case the lambda runs after add_ice_candidate has returned, so ctx and on_complete are dangling references into a dead stack frame. The deferred call then invokes freed memory.

Observed in production as a SIGSEGV on the signaling_thread with PC=0 / LR=0 (a null function pointer call) on the first remote ICE candidate that arrived while the answer was being applied. Reproduces reliably when candidates are forwarded immediately after set_remote_description without waiting for it to settle.

Fix

Move ctx into a shared_ptr owned by the lambda and capture on_complete by value, so the callback holds its own state for as long as libwebrtc keeps it. No API change.

We have shipped this as a vendored patch for several months on macOS desktop builds and the crash has not recurred.

@cliqer
cliqer requested a review from ladvoc as a code owner September 2, 2026 12:11

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@CLAassistant

CLAassistant commented Sep 2, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cliqer

cliqer commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

The Test (x86_64-pc-windows-msvc) failure is test_e2ee (livekit integration test), which is unrelated to this change: webrtc-sys builds and every other test passes on Windows.

The same test fails identically on main on the Windows runner, for example run 33550195134 (2026-09-01, 14 passed / 1 failed / test_e2ee) and run 33463303270, while other recent main runs pass, so it looks like a flaky test on that runner rather than a regression from this PR. I do not have permission to rerun the job from a fork.

@stephen-derosa stephen-derosa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cliqer thanks for opening this fix! We have a PR for i believe the same bug here:
#1388. #1388 does guard against a second invocation of completion, but to be frank I dont think this is a reachable code path is reachable. It also adds tests for the explicit dereferencing of freed stack memory.

Would you be open to pulling in the relevant tests from #1388 into this PR?

Im glad you have been proving out this fix for the last few months!

@cliqer
cliqer requested a review from cloudwebrtc as a code owner September 2, 2026 16:34
@cliqer

cliqer commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

@stephen-derosa happy to. Pulled the three libwebrtc regression tests from #1388 into this branch as-is (deferred completion resolves, close during a deferred completion, 25 repeats of the deferred path). Ran them locally on macOS arm64 against the current prebuilt:

  • with this fix: 3 passed, 0 failed
  • with the unpatched add_ice_candidate: the first test fails with add_ice_candidate cancelled and the test process dies on the freed callback state, so they do reproduce the bug

I left out the webrtc-sys seam tests, since they exercise the double-invocation guard of #1388's AddIceCandidateCompletion, which as you say is not a reachable path with libwebrtc; this PR keeps the shim to the shared context capture only. If you would rather land #1388 with its guard and tests, I am fine closing this one, whichever gets the crash out of the next release faster.

devin-ai-integration[bot]

This comment was marked as resolved.

@stephen-derosa stephen-derosa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving since this is a bit cleaner than #1388 . Please wait for @lukasIO review before merging since he has a better understanding than i do on the intricacies here

Comment thread webrtc-sys/src/peer_connection.cpp Outdated
@cliqer

cliqer commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Done: reverted 01466f3 and dropped the three libwebrtc tests, so the PR is back to the shared-context capture only. CI red on Test API is sip_busy hitting the live server (fails on every PR today, unrelated).

@stephen-derosa

Copy link
Copy Markdown
Contributor

@cliqer after you rebase, tests should pass and we can merge in

cliqer and others added 8 commits September 8, 2026 16:26
…alue

AddIceCandidate's completion callback fires asynchronously on the signaling
thread and is deferred behind the operations chain when it is busy (for
example while a SetRemoteDescription is in flight). The lambda captured
`ctx` and `on_complete` by reference, so a deferred completion dereferenced
a stack frame that had already returned and crashed the signaling thread
with a null function pointer call.

Move both into a shared_ptr owned by the lambda so the callback holds its
own state regardless of when libwebrtc runs it.
…operations chain

Ports the three regression tests from livekit#1388: a completion deferred behind a busy operations chain resolves, closing the peer connection during a deferred completion does not read released state, and the deferred path repeated 25 times surfaces no use-after-free.

Without the capture-by-value fix the first test fails with "add_ice_candidate cancelled" and the test process dies on the freed callback state.
libwebrtc keeps the completion in a copyable std::function. Hold the context in a shared optional and exchange it out before calling into Rust, so a second invocation of any copy is ignored instead of moving an already-moved Box.
@cliqer
cliqer force-pushed the fix/add-ice-candidate-dangling-capture branch from 1714a4d to 5a84f00 Compare September 8, 2026 15:32
@cliqer

cliqer commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (4dfca31), now includes #1398 so the sip_busy check should go green. Net diff unchanged: the shared-context capture in peer_connection.cpp plus the changeset.

Thanks @stephen-derosa and @lukasIO for the quick reviews and for sorting the test-server drift. Happy to see this land; it lets us drop the vendored webrtc-sys patch on our side.

@stephen-derosa
stephen-derosa merged commit 0b6ce1a into livekit:main Sep 8, 2026
22 checks passed
@stephen-derosa

Copy link
Copy Markdown
Contributor

@cliqer this will be in the next release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants