Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions docs/desktop-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,9 +150,9 @@ binary is included in desktop or Android releases. The
[control dependency record](desktop-obs-control-resource.md) records the pinned
library, reviewed full license and development-wheel provenance.

Continue on `feat/obs-enrollment-flow` with frontend acceptance and the desktop
connection/controller after the pairing setup increment. Arm/PCM and live
recognition remain later gates. No live-capture app entry point, audio endpoint, plugin binary
Pairing setup remains in draft PR #36 on `feat/obs-enrollment-flow`; the separate
Arm increment continues on `feat/obs-session-arm`. The desktop controller, PCM and
live recognition remain later gates. No live-capture app entry point, audio endpoint, plugin binary
publication or OBS capture integration exists yet.

The linked native pairing Tools flow, exclusive per-user owner and strict
Expand All @@ -178,10 +178,22 @@ pass after the final status-card adjustment. Native TaskDialog activation,
Escape, modal cleanup and zero-mutation checks also pass in an isolated fixture;
all 29 native driver commands pass with `--ui`. Normal/compact/enlarged desktop
renders were inspected. Independent final desktop setup review is clear, with
29 focused and 187 related tests rerun on the final source. Its own CI remains
pending; no existing release changes. Earlier linked checkpoint `15e1c77` passed
29 focused and 187 related tests rerun on the final source. All five exact-source
CI jobs pass at `e71b627` in [34760760046](https://github.com/RioPlay/utterleaf/actions/runs/34760760046).
No existing release changes. Earlier linked checkpoint `15e1c77` passed
all five desktop CI jobs in [34758907354](https://github.com/RioPlay/utterleaf/actions/runs/34758907354).

The separate [Arm integration](plans/active/obs-session-arm.md) is active on
`feat/obs-session-arm`. It adds a fixed authenticated-pipe command, guarded
frontend consent, bounded native I/O and an explicit desktop Arm operation.
The 199-test affected desktop bundle and all 34 native build/test commands pass,
along with the linked build and headless refusal smoke. Independent source review
is clear. Startup/started/stopping states remain busy even when public activity
flags are false; a failed start without STOPPED requires a later actual STOPPED
or OBS restart before fresh Arm. Visible failure/recovery controls and real OBS
acceptance remain pending with the controller/PCM work. This is source work,
not live OBS support or a new release.

Current integrated desktop regression: **1,432 passed, 13 skipped in 39.47 seconds**,
including the reviewed OBS control, native identity, pipe and audio handshake.
The combined focused OBS/pipe/privacy/configuration/boundary/packaging bundle
Expand Down
24 changes: 15 additions & 9 deletions docs/plans/active/obs-audio-implementation.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,25 @@ in source; its physical microphone and release gates remain separate.

## Current increment

The [enrollment plan](obs-native-enrollment.md) now records the integrated private
pairing stores and the typed desktop Issue/Prepare adapter on
`feat/obs-enrollment-flow`. Its focused regression passes 341 tests with no skips,
including 66 enrollment cases. Native owner, Tools and vendor dispatch are still
pending; no app entry point or capture activation is exposed.
The [enrollment plan](obs-native-enrollment.md) records the integrated private
pairing stores, native owner/Tools/vendor flow and typed desktop Issue/Prepare
adapter. The Windows [pairing setup](obs-desktop-pairing-ui.md) at `e71b627` passed
all five desktop CI jobs in run 34760760046 and remains in draft PR #36.
The separate [explicit Arm increment](obs-session-arm.md) on
`feat/obs-session-arm` passes 199 desktop tests, 34 native build/test commands,
the linked build and headless smoke, with independent source review. Its
conservative frontend guard refuses fresh Arm after a failed start without
STOPPED until a later actual STOPPED or OBS restart. Controller guidance and real
frontend recovery remain acceptance gates. No live-capture app entry or audio
activation is exposed.

The bounded audio protocol, consent/session receiver and read-only loopback
WebSocket control client are reviewed. The Windows TCP peer identity gate is now
implemented and integrated, as recorded below. The separate original C module now
has a verified inert build and `libobs` load/unload prerequisite; this does not
establish the native server, vendor registration, PCM callback, Arm, or real OBS
application workflow. Continue with the original native plugin/server and its
authenticated audio endpoint. Root owns the integrated source,
has a verified build and headless-refusal prerequisite, linked native enrollment
and authenticated pipe admission. These do not establish actual OBS frontend
load, the PCM callback or the complete application workflow. Continue the
original mix callback and visible controller. Root owns the integrated source,
tests and this plan after delegated handoff; assign each next component one writer
and a separate reviewer under the repository's ownership rules.
No runtime/UI entry is exposed until the authenticated transport can satisfy the
Expand Down
160 changes: 160 additions & 0 deletions docs/plans/active/obs-session-arm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Explicit OBS session arming

September 13, 2026. Continue on `feat/obs-session-arm`, stacked on reviewed
pairing setup `e71b627` in draft PR #36. That exact setup source passed all five
desktop jobs in [CI 34760760046](https://github.com/RioPlay/utterleaf/actions/runs/34760760046).
This plan implements the next part of the full [live audio contract](obs-audio-design.md).
The published previews remain unchanged.

## Goal and area

After explicit enrollment and pipe authentication, require a separate one-use
Arm transaction. Admit it only while OBS is idle. Only a later ordered frontend
STARTING then STARTED may authorize that session's future audio delivery. Keep
the authenticated connection alive while armed, detect client departure, and
retire it on disarm, stop, cancellation, revocation or exit. Connecting or finding
an existing stream must never grant capture consent.

Area: native authenticated I/O, bounded session command codec, plugin worker and
frontend state coordination; desktop `ObsAudioPipe` explicit Arm operation;
focused native/desktop interop and race tests. The connection controller and PCM
callback/recognition remain required follow-ups, not replacements for this scope.

## Constraints

- Keep enrollment proof and Hello/ACK unchanged. Arm travels only inside the
already authenticated duplex pipe, never vendor JSON. Keep the prepared session
ID and requested additional mix mask until worker teardown.
- Keep unauthenticated and authenticated-but-unarmed deadlines bounded at 15
seconds each. Armed waiting has no arbitrary recording-duration countdown.
Retain one joined worker, one request slot and bounded wire/I/O buffers.
- Use the existing retained process/pipe identity and cancellation/drain logic
before and after native I/O. Never allocate from untrusted wire lengths.
- Serialize Arm's OBS idle observation with frontend state transitions. Do not
block the frontend on a worker that needs frontend work. Queued callbacks must
survive late invocation after EXIT without touching retired heap state or OBS.
- Success acknowledges a committed Arm, not successful capture. Lost replies and
errors are terminal; no automatic retry/reconnect/rearm. Session IDs are routing
metadata and do not independently authenticate anything.
- No OBS installation, consumer profile or microphone changes during fixtures.
Original native GPL source remains separate from Apache desktop code and Android.

## Acceptance and verification

The fixed Arm record is 28 bytes, `<4sBBH16sBBH`: `ULAC`, version 1, kind
1=request or 2=reply, zero header reserved, the prepared 16-byte session ID,
prepared additional mix mask (0–63), status, zero trailing reserved. Request
status is 0; reply status is 1=committed or 2=refused. Only the exact prepared
session/mask may be used. A reply precedes any future ULAP frame on that pipe.
An unrecognized or malformed command terminates the attempt.

- Native exact I/O: authenticated roundtrip, fragmentation, wrong/preauth peer,
timeout, cancellation, EOF and failed-read wiping using actual child processes.
- Command framing: exact lengths, magic/version/kind/session/mask/reserved/status
checks; malformed, duplicate and replayed commands terminate.
- Consent ordering: refuse already-active/starting streams; STARTING before Arm
commit refuses, Arm before STARTING then STARTED authorizes only once; no Start
or PCM before that transition. Failed OBS start must not manufacture STARTED.
- Lifetime: client close, pending UI task, stop, replacement, forget and EXIT
races cancel/drain/join; late callbacks cannot access OBS or freed runtime.
- Desktop Arm sends once, checks its bound reply, preserves coalesced future PCM,
and closes on error/cancel; concurrent close must unblock it.
- Run focused native fixture drivers and affected Python OBS/pipe tests, then
native integration build/test checks and independent source/evidence review.
Record exact commands and results here as implemented. Fixture results do not
establish real OBS frontend/audio load or physical-device usability.

## Non-goals and stop

Do not add an alternate recorder, change OBS routing, silently attach to existing
streams, publish a plugin, or treat this as completed live transcription. Stop
editing the Arm increment when its observable checks and independent review pass;
continue the actual mix callback, controller, live recognition and acceptance
gates required by the desktop roadmap.

## OBS lifecycle boundary

The pinned OBS 32.2.2
[streaming frontend](https://raw.githubusercontent.com/obsproject/obs-studio/ba2f32bdf791005443988a4955e963663e16b1ed/frontend/widgets/OBSBasic_Streaming.cpp)
emits STARTING before calling the output start operation and STARTED later. Its
synchronous start-error path emits a private Qt stopped signal without necessarily
a public frontend STOPPED event. Once STARTING is observed, the bridge keeps its
frontend busy guard set through STARTED and STOPPING, and clears it only on
STOPPED. The Arm check also requires both current activity queries to be false.

`obs_output_active()` covers active/reconnecting outputs, so false alone does
not rule out an asynchronous connection attempt. The pinned
[libobs output implementation](https://raw.githubusercontent.com/obsproject/obs-studio/ba2f32bdf791005443988a4955e963663e16b1ed/libobs/obs-output.c)
does not provide a public completion event for a synchronously rejected start.
A later queued UI callback is not proof that the start call has returned: nested
frontend event processing can run queued work early. An absent signal or elapsed
grace period therefore cannot safely establish idle. A 30-second deadline
for STARTING to reach STARTED terminally disarms an orphaned/slow startup; it
does not establish that OBS became idle. Waiting while armed and recording after
STARTED have no duration cutoff.

Known limit: after a failed start that omits STOPPED, a fresh Arm remains refused
until OBS emits STOPPED in a later actual lifecycle or OBS restarts. Utterleaf
does not initiate that lifecycle or change stream settings. Clear user-facing
explanation and real OBS failure/recovery acceptance remain requirements for the
pending desktop controller. These component fixtures do not establish that
experience. If EXIT is missed, unload closes native gates without calling OBS.

## Component verification

The final native driver passes **34 build/test commands**, including actual
child-process exact I/O, injected API failures, fixed commands, Windows runtime
thread/event ordering, queued frontend lifecycle and real-libobs vendor dispatch.
Exact-I/O cases cover fragmentation, bounds, partial-read wiping, EOF, typed
cancellation and blocked-write draining. Injected event-creation, zero/over-count
transfer and failed probe cases cannot report success; cancellation or deadline
expiry during the final peer check wipes the read and wins over success.

The runtime fixture covers late generations, accepted/refused Arm, STARTING before
the queued callback, startup expiry, late STARTED, and cancellation through
revocation/close. Bridge fixtures refuse startup/started/stopping states even
when activity flags are false, and prove stale/duplicate/closed UI tasks cannot
touch retired state. The explicit desktop Arm and affected
protocol/session/pipe/privacy/boundary bundle passes **199 tests in 5.50 seconds**,
with no skips. Separate source review cleared the final codec/runtime and
conservative bridge gate; root reviewed the delegated I/O and desktop changes.

The linked x64 DLL builds against the pinned headers and installed OBS runtime.
Headless smoke opens it, refuses initialization before any pairing-store startup,
returns from shutdown, and leaves fixed-store metadata unchanged. The DLL SHA-256
is `9ce5f225fef16af2ccdcdbf5e30ea637df5a306863e0285d8329e33f1e7925a3`.
Three existing linker warnings remain confined to the test heap shim. The native
dialog fixture was not repeated for this increment; its earlier setup evidence
remains separate. No OBS application, source or audio device ran in these checks.
Exact-source CI for this Arm checkpoint remains pending.

From the owning worktree, the desktop command was:

```powershell
$env:PYTHONPATH = (Get-Location).Path
& "C:\Users\unknown\Projects\Mindict\.venv\Scripts\python.exe" -m pytest tests/test_obs_audio_pipe.py tests/test_obs_audio_arm.py tests/test_obs_session.py tests/test_obs_protocol.py tests/test_windows_pipe.py tests/test_privacy.py tests/test_repo_boundaries.py -o addopts='' -q
```

Local native evidence is under `.grok/obs-session-arm/`; the separate exact-I/O
development receipts under `.grok/obs-enrollment-flow/` are historical. The
authoritative final receipts are `build/build-receipt.json`,
`build/smoke-receipt.json`, and `verification/test-receipt.json` under that Arm
directory. Commands from the owning worktree:

```powershell
$py = "C:\Users\unknown\Projects\Mindict\.venv\Scripts\python.exe"
$toolchain = "C:\Users\unknown\.local\llvm-mingw-20260616-ucrt-x86_64"
$obsBin = "C:\Program Files\obs-studio\bin\64bit"
$headers = "C:\Users\unknown\Projects\Mindict\.grok\obs-native-build\headers"
$build = "C:\Users\unknown\Projects\Mindict\.grok\obs-session-arm\build"
$verification = "C:\Users\unknown\Projects\Mindict\.grok\obs-session-arm\verification"
& $py native/obs-plugin/tools/build.py --toolchain $toolchain --obs-bin $obsBin --headers $headers --output $build
& $py native/obs-plugin/tools/smoke.py --build $build
& $py native/obs-plugin/tools/test_native.py --toolchain $toolchain --output $verification --build $build --headers $headers
```

Continue the visible controller and actual PCM path. Future Start/PCM writes must
remain serialized after the complete successful Arm reply, even if native state
already reached STARTING/STARTED while that reply was in flight. This increment
emits no frames. Real frontend recovery, audio load, redistribution/toolchain
review and release acceptance remain open.
14 changes: 12 additions & 2 deletions docs/plans/active/windows-obs-audio-pipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,24 @@ Follows the reviewed
[TCP peer identity gate](windows-obs-peer-identity.md) and
[OBS audio design](obs-audio-design.md).

September 13 follow-up: [explicit session arming](obs-session-arm.md) adds a
separate `ObsAudioPipe.arm()` transaction after authentication. The caller supplies
the prepared additional mix mask and deadline; the exact matching native reply
must commit Arm before `read_frames()` accepts ULAP. Calling Arm twice, reading
before Arm, refusal, malformed replies, cancellation or identity loss closes the
connection. Coalesced bytes following the fixed reply remain available to the
frame reader. This is internal source behavior; the application controller and
live OBS audio workflow are still pending. Evidence below describes the earlier
handshake/receiver increment unless explicitly linked to that follow-up.

## Goal and area

Implement the local client side of the original OBS audio bridge: bind one named
pipe to the already verified OBS process before exchanging session credentials,
then receive bounded binary audio for the existing protocol/session receiver.
The audio connection must retain its own process identity so a degraded control
connection cannot silently replace or invalidate an otherwise healthy active
audio session. This component does not itself arm or record.
audio session. Authentication alone does not arm or record.

The source is `windows_pipe.py`, `obs_audio_pipe.py` and the process-lease additions
to `windows_peer_identity.py`, `obs_websocket.py` and `obs_control.py`, all under
Expand Down Expand Up @@ -85,7 +95,7 @@ its first read and consumes a session once. The client never reconnects a failed
session. Mutable secret/Hello construction buffers are cleared best-effort;
Python/HMAC copies cannot be guaranteed erased.

After ACK, each bounded read rechecks server PID and the independent process
After ACK and the explicit Arm reply, each bounded read rechecks server PID and the independent process
before and after I/O. Bytes feed the existing ULAP decoder; every frame must carry
the same session ID. End must be the last frame, with no trailing partial frame.
Consent, post-arm start ordering, bus selection and temporary stores remain the
Expand Down
4 changes: 2 additions & 2 deletions docs/workspaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ tree, one desktop OBS tree, and published/unpublished release receipts.
| Stream | Checkout | Next PR |
| --- | --- | --- |
| Android keyboard | `android-keyboard-hardening` on `main` | Signed alpha18 candidate. Phone/TalkBack/landscape stay Ernest. No new keyboard features in this slice. |
| Desktop OBS | `desktop-obs-bridge` | Land draft stack from the base: 36 enrollment → 37 arm → 38 PCM → 39 disarm → 40 controller → 41 provenance → 42 timelines. Rebase each onto current `main`/parent before undrafting. |
| Desktop OBS | `desktop-obs-bridge` | PR #36 merged. Next: draft [PR #37](https://github.com/RioPlay/utterleaf/pull/37) arm, then 38 PCM → 39 disarm → 40 controller → 41 provenance → 42 timelines. |
| Android CI split | same Android tree, later | Follow-up only: stop downloading speech models and running the full emulator suite on every keyboard PR. |

Do not mix these in one branch. Desktop CI already skips Android-only paths.

| Branch | Purpose | Next work |
| --- | --- | --- |
| `main` | Current Android keyboard checkout (`android-keyboard-hardening` worktree) | [Alpha17 polish](plans/active/android-alpha17-polish.md) is merged; signed candidate and phone acceptance remain |
| `feat/obs-enrollment-flow` | Desktop OBS stack base (`desktop-obs-bridge` worktree); draft [PR #36](https://github.com/RioPlay/utterleaf/pull/36) | Rebase onto current `main`, then pairing UI/vendor requests. Later stack PRs 37–42 stay parked. |
| `feat/obs-session-arm` | Desktop OBS stack (`desktop-obs-bridge` worktree); draft [PR #37](https://github.com/RioPlay/utterleaf/pull/37) | Arm/lifecycle guards on current `main`. PRs 38–42 stay parked until this lands. |
| `checkpoint/mixed-work-20260912` | Preserved mixed development snapshot; not a release or PR | Recovery/reference only; leave the original source environment intact |
| `release/desktop-0.4.6rc2` | Published Windows x64 CPU prerelease | Preserve the immutable RC2 tag and release evidence |
| `release/desktop-0.4.6rc1` | Unpublished RC1 candidate retained for audit | Reference only; tag and downloaded artifact unchanged |
Expand Down
Loading
Loading