Skip to content

bugfix: PeerConnectionInterface::AddIceCandidate - #1388

Closed
stephen-derosa wants to merge 1 commit into
mainfrom
sderosa/ice-callback-ownership
Closed

bugfix: PeerConnectionInterface::AddIceCandidate#1388
stephen-derosa wants to merge 1 commit into
mainfrom
sderosa/ice-callback-ownership

Conversation

@stephen-derosa

@stephen-derosa stephen-derosa commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Overview

PeerConnection::add_ice_candidate handed libwebrtc a [&] lambda that captured ctx (the
rust::Box<PeerContext>) and on_complete by reference — both stack parameters of the shim.
PeerConnectionInterface::AddIceCandidate completes asynchronously, so whenever libwebrtc's
operations chain deferred the completion, the shim returned and destroyed the captures first.
Dropping the context dropped the pending oneshot::Sender, so the Rust future resolved with
add_ice_candidate cancelled; the later callback then read freed memory (SIGSEGV/SIGBUS).

The callback state now lives in AddIceCandidateCompletion, held behind a shared_ptr by
make_add_ice_candidate_callback. It outlives the call, is safe for the copies libwebrtc makes
(AddIceCandidate takes a copyable std::function, and OperationsChain copies it), and an
atomic guard moves the context back to Rust exactly once.

Reproducing failure (before merge)

The bug only shows when libwebrtc's operations chain is non-empty: ChainOperation runs an
operation immediately when the chain is idle, so the callback normally fires synchronously inside
the blocking proxy marshal and the dangling captures are still alive. Stacking a CreateOffer on a
freshly created peer connection (its DTLS certificate is still generating) keeps the chain busy and
defers the completion:

res=Err(RtcError { error_type: Internal, message: "add_ice_candidate cancelled" })
signal: 10, SIGBUS: access to undefined memory

Tests

webrtc-sys — four unit tests over a test-only seam that drives the production callback
factory the way libwebrtc does (build it in a frame that returns, copy it, drop the original, then
invoke): survives-the-frame, error forwarding, exactly-once across repeated invocations,
release-on-drop.

libwebrtc — three integration tests on the existing peer-connection fixture: deferred
completion resolves, close during a deferred completion, and 25 repeated deferred completions. Each
asserts the chain is genuinely occupied first, so they cannot pass vacuously.

Against the old [&] implementation: 3/4 webrtc-sys tests SIGBUS, all 3 libwebrtc tests SIGSEGV.

Command Result
cargo fmt -- --check clean
clang-format on new C++ regions clean
cargo test -p libwebrtc -p webrtc-sys -- --test-threads=1 29 passed, 0 failed
cargo clippy -p webrtc-sys -p libwebrtc --all-targets no warnings in changed files
cargo check --workspace --all-targets no errors
20x repeat under MallocScribble/MallocPreScribble/MallocGuardEdges 0 failures

ASan was not run: -Zsanitizer=address needs nightly and the repo pins stable 1.97.1; the
prebuilt libwebrtc is not instrumented either. macOS malloc guards were used instead.

API breaking changes

None. PeerConnection::add_ice_candidate keeps its signature and its success/error results; the
change is confined to how the native shim owns the completion state.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Changeset incomplete

This PR's changeset is missing version bumps for packages that are affected by the change. The following packages still require a bump:

  • livekit
  • livekit-ffi

Already covered:

  • libwebrtc (patch)
  • webrtc-sys (patch)

A package must be bumped when its own files change, and whenever a package it depends on is bumped (so downstream consumers get a matching release).

Click here to create a changeset for the missing packages

The link pre-populates a changeset file with patch bumps for the missing packages. You can also add them to your existing changeset. Edit the bump types as needed before committing.

If this change doesn't require a version bump, add the internal label to this PR.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

resolved with `add_ice_candidate cancelled`, and the later callback read freed memory,
crashing with `SIGSEGV`/`SIGBUS`.

The completion state now lives in a shared, single-use object that outlives the call, is

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is sort of the main item to highlight in the whole PR

…usly and reacxhes an invalidated reference in edge cases
@stephen-derosa
stephen-derosa force-pushed the sderosa/ice-callback-ownership branch from 232f391 to b925200 Compare September 2, 2026 15:52
cliqer added a commit to cliqer/rust-sdks that referenced this pull request Sep 2, 2026
…operations chain

Ports the three regression tests from livekit#1388: a completion deferred behind a busy operations chain resolves, closing the peer connection during a deferred completion does not read released state, and the deferred path repeated 25 times surfaces no use-after-free.

Without the capture-by-value fix the first test fails with "add_ice_candidate cancelled" and the test process dies on the freed callback state.
@cliqer

cliqer commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Read through this one properly since it overlaps with #1392; the completion ownership and the single-use claim look right to me, and the three libwebrtc tests reproduce the crash on the unpatched shim (verified locally on macOS arm64: add_ice_candidate cancelled followed by the test process dying, then 3/3 green with the fix). Two things blocking or worth a look before merge:

  1. Changeset Check fails. The changeset bumps only webrtc-sys and libwebrtc, but the check requires the dependents as well, so it needs livekit: patch and livekit-ffi: patch too. Same failure fix(webrtc-sys): capture add_ice_candidate completion state by value #1392 hit; that front matter is at https://github.com/livekit/rust-sdks/blob/fix/add-ice-candidate-dangling-capture/.changeset/add_ice_candidate_capture_by_value.md if you want to copy it.

  2. The test seam is compiled into every build. webrtc-sys/build.rs defines LIVEKIT_TEST unconditionally (there is a TODO(theomonnom) on that line saying it should be tests only), so complete_add_ice_candidate_for_test ships in release builds. Harmless dead code, but the #ifdef LIVEKIT_TEST guard reads as test-only when it is not. Fine to leave for a follow-up, just flagging it so the comment does not mislead.

Happy to close #1392 once this lands.

@stephen-derosa

Copy link
Copy Markdown
Contributor Author

consumed by #1392

cliqer added a commit to cliqer/rust-sdks that referenced this pull request Sep 8, 2026
…operations chain

Ports the three regression tests from livekit#1388: a completion deferred behind a busy operations chain resolves, closing the peer connection during a deferred completion does not read released state, and the deferred path repeated 25 times surfaces no use-after-free.

Without the capture-by-value fix the first test fails with "add_ice_candidate cancelled" and the test process dies on the freed callback state.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants