Meetings: load the voiceprint model off the main thread at launch - #1932
Merged
Merged
Conversation
MeetingSessionController.init runs on main in applicationDidFinishLaunching, and it called SpeakerEmbedderFactory.makeEmbedder, which loaded ReDimNet2 (MLModel(contentsOf:), plus a GPU compile the first launch after an update) right there. With ReDimNet2 the default, every launch froze the menubar for that long. makeEmbedder now loads nothing. It picks the model, and so the speaker database, from model-file presence and returns Core's new BackgroundLoadedSpeakerSegmentEmbedder, which knows the model's id, size and thresholds up front and loads the real embedder on a utility queue the first time something waits for it: - DiarizationService awaits it during initialize() (the launch warmup), so "meeting models warm" still means the voiceprint is loaded, and again before re-embedding a meeting, so a meeting that races the load gets the right model's vectors. - SpeakerVoiceprintMigration.run awaits it before moving anyone. The gate still closes at launch, so writers stay held through the load. If the load fails, this launch keeps the model's own database and gets no voiceprints (nothing 256-d ever reaches the 192-d file), the migration fails with embedder_unavailable without writing the ledger, and SpeakerEmbedderLoadFailureMemory makes the next launch on the same app build and macOS version use WeSpeaker and speakers.sqlite, the old fallback. A new build or macOS update tries the model again.
Owner
Author
|
Independent review (Fable reviewer, full diff): SHIP. No deadlocks (continuation waits only; blocking wait unreachable from main), meetings always finish (failed load → no vectors, transcript saves), migration gate race safe, happy path identical, failure memory acceptable. Non-blocking notes: one-launch empty Speakers view after a failed load; no wrapper timeout (CoreML errors instead). Integrated via #1934. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MeetingSessionController.initruns on main fromapplicationDidFinishLaunching. It calledSpeakerEmbedderFactory.makeEmbedder, which loaded ReDimNet2 synchronously (MLModel(contentsOf:)viarouter.preload, plus a GPU compile the first launch after an update). ReDimNet2 is the default in this release, so every launch froze the menubar for that long. On slower Macs that also means Sentry app-hang reports.Fix
makeEmbedderloads nothing now. It still picks the model, and so the speaker DB, synchronously, from model-file presence. What it returns is Core's newBackgroundLoadedSpeakerSegmentEmbedder. That type knows the model's id, size and thresholds up front, so the controller buildsDiarizationService,SpeakerDatabase(thresholds:)and the migration target exactly as before. It loads the real embedder on a utility queue the first time something waits for it.MeetingSessionControllerandTranscriptedApponly get a comment change.Who waits:
DiarizationService.initialize()(the existing launch warmup inTranscriptedAppState) awaits the load alongside the diarizer models. SoareMeetingModelsWarmstill means the voiceprint is loaded.DiarizationService.diarizeOffline(pyannote and Nemotron) awaits it again before re-embedding. A meeting that stops before the load finishes gets the right model's vectors against the right DB.SpeakerVoiceprintMigration.runawaits it before moving anyone. The gate still closes at launch ingate.start, so every speaker-DB writer is held through the load plus the move.embedcalled before the load ends blocks its (background) thread until the load ends. It never returns nil just because it was early.When the load fails:
embedreturns nil, andDiarizationServicedrops the native vector, so nothing 256-d reaches the 192-d file. Diarization and transcripts still work.embedder_unavailableand writes no ledger rows, so it retries later.SpeakerEmbedderLoadFailureMemoryrecords the failure against app build + macOS version. The next launch then resolves to WeSpeaker +speakers.sqlitewith WeSpeaker thresholds, which is the old fallback, and the DB path and embedder stay consistent. A new build or macOS update tries ReDimNet2 again.ERes2Net (hidden opt-in) goes through the same path, so it no longer loads on main either.
Tests
New behavior tests:
Tests/TranscriptedCoreTests/SpeakerTests/BackgroundLoadedSpeakerSegmentEmbedderTests.swiftcovers:embedisn't nilSpeakerVoiceprintMigrationGateTestsadds two tests:embedder_unavailable, writes no ledger rows and leaves the source DB untouched, and the next launch moves everyoneTests/SpeakerEmbedderLoadFailureMemoryTests.swift(fast suite) covers fallback tospeakers.sqliteafter a failure on the same build, a retry on a new build or OS, and a success clearing the failure.Mutation-probed: dropping the migration's wait, dropping the declared-model check, and dropping the failure check each turn a test red.
Run locally:
swift test --filter 'BackgroundLoadedSpeakerSegmentEmbedderTests|SpeakerVoiceprintMigrationGateTests|DiarizationReembedTests|SpeakerVoiceprintMigrationTests': 34 passbash run-tests.sh --filter SpeakerEmbedderLoadFailureMemory: passbash build-deps.sh --forceandbash build.sh --no-open: passbash check.shon the uncommitted tree:run-tests.sh, integration smoke, fullswift test, source pins and test shape all passed.Sources/Support, and doc-paths (the new file wasn't tracked yet). Both are fixed in the commit. After the fix I rerancheck-doc-paths.pyandconcurrency-census.sh --check, and both pass.check.shafter that fix, per the coordinator. CI runs the full suite.Not done / risks
voiceprint_model: redimnet2-b4, even though no voiceprints were computed..utility. A meeting imported in the first seconds after launch may wait a little longer for the voiceprint than it waits for the diarizer.speakers.sqlite), before the dictation warmup, now off main. Before this change it ran on main at that same point.🤖 Generated with Claude Code