fix(bithuman): publish each picture when its audio starts playing by sgu-bithuman · Pull Request #7492 · livekit/agents

sgu-bithuman · 2026-09-26T18:29:07Z

What

AvatarRunner pushes the avatar's audio into an rtc.AudioSource that holds up to ~100 ms before it plays out, while BithumanGenerator yields each picture as soon as it is rendered. The picture therefore reaches the room ahead of the audio it belongs to, and the avatar's mouth leads its voice by whatever the audio source is holding. With Expression 2 the source stays filled (102-106 ms held in steady state), so the lead is constant.

This PR:

adds AvatarRunner.audio_queued_duration, a read-only property: seconds of audio pushed to the room's audio source and not yet played out;
has BithumanGenerator yield the audio first and hold the picture until that audio starts playing out (now + queued - chunk duration), 25 ms early to absorb the video pacer. AvatarSession wires it up; nothing changes for callers;
drops the held pictures on clear_buffer() (barge-in), and publishes a frame that has no audio immediately, as before.

Measured

Setup: AvatarRunner with this plugin's generator and the bitHuman runtime rendering in real time on an x86 Linux machine, a LiveKit Cloud room, and a Chrome viewer. Offset = picture minus voice; negative means the mouth moves before the voice.

Publisher: time each picture is captured vs the time its audio starts playing out.
Viewer: SyncNet over a screen-and-audio capture of the Chrome viewer, corrected for the capture's own A/V offset. Each value is the mean of n reads with confidence ≥ 3.

	Before	After
Expression 2, publisher	-72 ms	-1 ms
Expression 2, viewer	-121 ms (n=4)	-59 ms (n=5)
Essence 2, publisher	+1 ms	+5 ms
Essence 2, viewer	+48 ms (n=1)	+47 ms (n=4)

Time to first audio did not change (2.86 s for Expression 2, 1.14 s for Essence 2), and there were no frozen frames.

These numbers were taken with the same release rule running one layer below this generator, in the bitHuman Python SDK, with the plugin's generator unchanged on top. This PR moves the rule into the plugin so every bithuman.AvatarSession user gets it. The unit tests below check the release timing of the plugin code itself.

What this does not fix

The publisher side is now aligned. The viewer still sees Expression 2 about 60 ms mouth-early, and Essence 2 about 47 ms mouth-late. So this PR does not bring the viewer within a ±40 ms lip-sync target, and we are not claiming that it does.

The remaining offset is at the receiver. The avatar's audio and video tracks reach the browser with different media stream ids (msid), so the browser treats them as unrelated streams: each plays out from its own jitter buffer, and no audio/video synchronization happens between them. Setting the same stream in TrackPublishOptions for both tracks did not change this, because the subscriber's SDP still carried separate msids. Closing the viewer-side gap probably needs the audio and video from one participant to be grouped into one media stream on the subscriber side, so the browser's own lip-sync applies. We would like maintainers' guidance on whether that is possible.

Tests

tests/test_plugin_bithuman_sync.py (unit, fake runtime and fake audio queue):

test_picture_is_released_when_its_audio_starts_playing
test_without_follow_audio_the_picture_leads_its_audio_by_the_queue (control: the old behaviour)
test_clear_buffer_drops_pictures_waiting_for_their_audio
test_a_picture_that_arrives_without_audio_is_published_as_is

ruff format --check and ruff check are clean. mypy is clean on livekit.agents.voice.avatar. For livekit.plugins.bithuman with our local dependencies, mypy reports two no-any-return errors on video_fps and audio_sample_rate; those lines already exist on main, and this PR does not touch them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01THQV7JTqkEVKG6PZedmQYr

AvatarRunner's audio source holds up to ~100 ms of audio before it plays out, while the bitHuman generator yields each picture as soon as it is rendered. Once the source has filled, the avatar's mouth leads its voice by what the source holds (Expression 2 at the publisher: -72 ms; at a browser viewer: -121 ms). The generator now hands the audio over first and releases the picture when that audio starts playing out (AvatarRunner.audio_queued_duration, a new read-only property), 25 ms early to absorb the video pacer. A barge-in drops the pictures still waiting; frames that arrive without audio are published as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01THQV7JTqkEVKG6PZedmQYr

… its voice by ~0.1 s, and link the upstream fix (livekit/agents#7492) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01THQV7JTqkEVKG6PZedmQYr

devin-ai-integration

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-09-26T18:31:45Z

+                    yield create_audio_frame(audio_chunk)
+                    try:
+                        queued = max(0.0, float(self._audio_queued()))
+                    except Exception:
+                        queued = 0.0
+                    duration = len(audio_chunk.array) / audio_chunk.sample_rate
+                    release = time.monotonic() + max(0.0, queued - duration) - _PICTURE_LEAD_S
+                    self._pending.append((release, create_video_frame(frame.bgr_image)))


🟡 Interrupted pictures reappear after buffer clearing

When clear_buffer() runs during an audio yield, _stream_impl queues that interrupted chunk's picture after the clear. The stale picture then reaches the room during the next turn.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-09-26T18:31:46Z

+                if frame.end_of_speech:
+                    yield AudioSegmentEnd()


🟡 Playback completion precedes final picture

When end_of_speech arrives with queued pictures, _stream_impl sends AudioSegmentEnd before releasing them. AvatarRunner reports playback finished while the final pictures remain unpublished.

Was this helpful? React with 👍 or 👎 to provide feedback.

sgu-bithuman · 2026-10-03T00:59:00Z

Hi @longcw, when you have a moment, could you take a look at this one? You reviewed the earlier bitHuman plugin changes (#7301, #7366, #7406), so you have the most context.

It fixes a visible issue for every bitHuman user on LiveKit with Expression 2: the avatar's mouth moves about 70-120 ms before its voice. The picture is published as soon as it's rendered, while its audio waits in the rtc.AudioSource buffer. The table above has before/after measurements at the publisher and at a Chrome viewer.

To make review easier:

The only core change is a read-only AvatarRunner.audio_queued_duration property in voice/avatar/_runner.py. Everything else is in the bitHuman plugin, plus tests/test_plugin_bithuman_sync.py.
Nothing changes for callers. Barge-in (clear_buffer()) drops held pictures, and frames with no audio still publish immediately.
CI is green: ruff, type-check 3.10/3.13, unit tests, and Devin found no issues.

If it's easier, I can split the core property into its own small PR, or rebase onto current main (it's 23 commits behind, with no conflicts). Happy to adjust anything. Thank you!

longcw

sorry for the late reply. I just got back from a holiday vacation.

One edge case found by claude: the plugin allows bithuman>=0.5.25, and bithuman 0.8.1 emits idle pictures with no audio as soon as speech input runs out. Those skip the hold, so they go out ahead of the last held speech pictures, and the face jumps to idle and back. Maybe raise the floor past 0.8.x, or hold audio-less pictures behind _pending?

longcw · 2026-10-08T09:12:16Z


    def clear_buffer(self) -> None:
+        # the audio these pictures belong to is being cleared too
+        self._pending.clear()


nit: clear_buffer() drops _pending, but AvatarRunner does not clear the AudioSource on interrupt. The last ~100 ms of speech still plays, maybe keep the pending here?

sgu-bithuman requested a review from a team as a code owner September 26, 2026 18:29

sgu-bithuman mentioned this pull request Sep 26, 2026

sdk/livekit: link the upstream picture-to-voice fix (livekit/agents#7492) bithuman-product/public-docs#371

Merged

devin-ai-integration Bot reviewed Sep 26, 2026

View reviewed changes

sidxh mentioned this pull request Sep 26, 2026

fix(bithuman): don't retry 4xx when starting a session, and keep the provider's error #7500

Open

longcw approved these changes Oct 8, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(bithuman): publish each picture when its audio starts playing - #7492

fix(bithuman): publish each picture when its audio starts playing#7492
sgu-bithuman wants to merge 1 commit into
livekit:mainfrom
sgu-bithuman:bithuman-sync-picture-to-voice

sgu-bithuman commented Sep 26, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Sep 26, 2026

Uh oh!

devin-ai-integration Bot Sep 26, 2026

Uh oh!

sgu-bithuman commented Oct 3, 2026

Uh oh!

longcw left a comment

Uh oh!

longcw Oct 8, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sgu-bithuman commented Sep 26, 2026

What

Measured

What this does not fix

Tests

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

sgu-bithuman commented Oct 3, 2026

Uh oh!

longcw left a comment

Choose a reason for hiding this comment

Uh oh!

longcw Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants