Add Apple Speech as a transcription engine choice

[claude]

Requested by Justin · project thread

Before: the Model picker offered Parakeet V3, Parakeet V2, and two Whisper models. People who find Apple's own speech engine more accurate for their language had no way to use it.

After: the Model picker has a fifth choice, "Apple Speech (built into macOS)". It runs on the Mac using macOS 26's SpeechAnalyzer/SpeechTranscriber. It works for meetings and imported files (with the Meeting language picker) and for dictation (in the Mac's language). The first time a language is used, macOS downloads its files from Apple and the Model files card shows progress. Parakeet V3 stays the default.

It has never run on a Mac. It merges into main once CI is green and its deep review has nothing open. Merging doesn't ship it: it goes out in a later release after the Mac test below. It's not part of 1.1.62.

Why

User feedback (Chris, emailed 2026-09-23): Apple's on-device engine is the most accurate for his native language, and he asked for SpeechAnalyzer/SpeechTranscriber by name. His main use is importing hour-long recorded lectures and meetings.

Product Impact

• Affects: meetings, dictation
• Lane: meeting reliability
• Why this matters: gives people a third engine without changing anything for users who keep the default.

What changed

• Sources/Speech/AppleSpeechEngine.swift (new): wraps SpeechTranscriber + SpeechAnalyzer. Each diarized segment is converted to the analyzer's preferred format, analyzed, and its final results joined. Language assets install through AssetInventory.assetInstallationRequest, with progress published as .downloading. Finished text runs through the custom dictionary, like the other engines.
• Sources/Speech/AppleSpeechLocalePolicy.swift (new, Foundation-only, fast-tested): maps the app's bare language codes ("es") onto Apple's per-region locales. The Mac's region wins, then the region its script implies (Traditional Chinese → Taiwan), then the language's home region, then a stable first match.
• Language rules: Apple's engine can't detect a language, so Auto means the Mac's first preferred language. An explicit meeting language is honored. When a language isn't supported, the error says so and names what to change ("Apple Speech can't transcribe Finnish yet. Choose another meeting language or another model in Settings."). The Settings row says this before recording too, and the picker only lists languages macOS reports as supported.
• TranscriptionModelChoice.appleSpeech (raw value apple-speech) with its own warmup runtime, so it never shares a lease with Parakeet or Whisper. supportsMeetingLanguageChoice replaces isWhisper for the language setting.
• STTRouter routes every engine switch to the new engine. Dictation uses the existing Whisper-style external-engine path over Parakeet-recorded samples.
• Model files card: Apple-specific download copy ("Downloading from Apple") and the real failure message instead of the generic retry text.
• -framework Speech added to the shared swiftc args.
• Setup downloads the Mac's language first, since that's what dictation uses. A different saved meeting language downloads right after, with progress in the Meeting language row. Once the engine is ready, later language downloads don't flip it back to "not loaded", so dictation never waits on a meeting language.
• SpeechAnalyzer is created with .lingering model retention so Apple keeps the model loaded across a meeting's segments.
• TranscriptedQA accepts transcription_engine: apple_speech_local, and docs/capture-format.md lists it.
• Changing the Meeting language starts that language's download right away, and the Meeting language row shows "Downloading Spanish from Apple… 40%" (or a plain failure line). This progress is separate from the model state, so the engine never looks unloaded because of it.
• Apple caps how many languages an app can reserve. At the cap, the engine releases the least recently used one that isn't in use. A transcription failure makes the next attempt re-check the language with Apple instead of trusting a stale "installed" mark.
• NSSpeechRecognitionUsageDescription added to Info.plist as insurance in case macOS checks it on this path.
• A saved meeting language Apple can't transcribe now gets the "Choose a Whisper model" failure copy instead of generic "needs another pass" copy.
• Per-segment audio conversion runs off the main thread. Chinese, Japanese, Cantonese and Thai results join without spaces. Traditional Chinese Macs get zh_TW, and Mandarin never lands on a Hong Kong (Cantonese) locale.
• No new Sentry/analytics events: the new apple_speech.* EventReporter events aren't on either allowlist, so they stay in local logs only.

How I checked it

• scripts/dev/agent-preflight.sh
• python3 scripts/dev/check-build-source-lists.py (passed)
• bash -n on the edited scripts (passed)
• bash build.sh --no-open and bash run-tests.sh: green in CI (app-build, checks, spm-tests on 00fcf03). The Speech API calls compiled first try.
• Everything after 00fcf03 (two in-thread reviews, the release-wide deep review, and a final review, all findings fixed), merged with main after 1.1.62: CI running.
• Apple API names checked against Apple's published docs (SpeechAnalyzer.Options(priority:modelRetention:), AssetInventory reserve/release/maximumReservedLocales, SpeechTranscriber.isAvailable).
• Added Tests/AppleSpeechLocalePolicyTests.swift and extended the model/language/router policy tests.
• Mac test: not done yet (before the release that carries it, steps below).

Mac test (before the release that carries it)

Do one step at a time. Stop and report at the first thing that looks wrong.

1. Open Transcripted from Applications (this PR's build). Go to Settings and set Model to Apple Speech (built into macOS).
   • Expect: no crash. If an "allow speech recognition" popup appears, click Allow and note it. The Model files card says it's getting ready, or shows a download from Apple.
2. Wait for the Model files card to say ready.
   • Expect: it finishes, or shows a clear message. It shouldn't spin forever.
3. Set Meeting language to Spanish.
   • Expect: Spanish is in the list. Right away, the note under it says "Downloading Spanish from Apple… N%" and then goes back to the normal note when it's done.
4. Import a short Spanish audio or video file (1–5 minutes).
   • Expect: a Spanish transcript that saves and opens.
5. Set Meeting language back to Auto. Record a short meeting (30 seconds, just talk).
   • Expect: a transcript in your Mac's language.
6. Dictate one sentence into any text field.
   • Expect: the text pastes. It may be a bit slower than Parakeet on the first try.
7. Switch Model back to Parakeet V3 and dictate once.
   • Expect: it works exactly like before.

If anything fails, grab ~/Library/Application Support/Transcripted/logs/app.jsonl and search for apple_speech.

Risk Review

• Privacy / local-first behavior reviewed: audio stays on the Mac. Only Apple's language assets are downloaded, by macOS.
• Storage path or migration impact reviewed: new raw value only. Unknown values already fall back to Parakeet, so downgrading is safe.
• Public-facing copy stays concrete and matches current product scope
• Release/update impact reviewed: not part of 1.1.62.
• Agent PRs link the issue/workpad and stay draft until human review
• UI changes include sanitized .agent-review/visuals/ evidence: none, never rendered.
• No private transcripts, audio, tokens, personal paths, or customer data are included

Notes

Known limits, left for measurement: one analyzer per diarized segment (the STT shootout in #1788 measures Apple Speech on an hour-long lecture, and if it's slow, a per-job analyzer is the next step); setup needs the Mac's language, so if that download fails, the engine fails even when the meeting language is installed. Language reservations go through one queue, so a hung Apple reservation call would stall later installs too (accepted; it's what stops two installs from releasing each other's slot).

How: one new engine behind the existing STTRouter switch. Nothing in TranscriptedCore changed. The pipeline already asks the engine to resolve the meeting language once per job, then passes that language to every segment.

Agent handoff

COORD_DONE: BRIEF | this PR | Apple Speech engine choice for meetings/imports/dictation | none | Mac test before its release (steps in PR body) | preflight + source-list check + CI green on 00fcf03 | CI green, then merge into main

🤖 Generated with Claude Code

https://claude.ai/code/session_013GAxhzGyuaTxNTv8tpj9j2

[claude]

build-and-test is red on 1a17e3a because its three Mac jobs (app-build, checks, spm-tests) were cancelled on purpose. Nothing failed. CI on this draft is on hold so the 1.1.62 release PRs get the Mac runners first. The last full run (00fcf03) was green. Once 1.1.62 ships I'll merge main in again and push without [skip ci], which re-runs everything.

---

Generated by Claude Code