Writing: keep secrets out of Personal History and the next-word predictor

[r3dbars]

What

Personal History (the encrypted log and the next-word predictor) now takes the keyboard's events only after Save my writing has cleared them of secrets. Before this, WritingHistoryIngest sent every raw batch straight to PersonalHistoryController, guarded only by secure input and the password-manager list. So a password typed at a sudo prompt in Terminal, an OTP or a card number got stored, and a password with letters in it could be learned and served as a ghost suggestion (Tr0ub4dor&3 teaches the predictor "dor" after "ub").

How

The day-file composer (WritingEntryComposer) already groups events into entries and scrubs each one with per-app context lines, which is what catches a sudo password (it's the segment after the command) or a card typed over four boxes. So Personal History now rides on that:

• Each open entry keeps the typed/accepted events it was built from.
• When the entry closes, its events are cleared for Personal History only if WritingSecretScrubber changes nothing, checked twice: on the entry as saved, and on the text as the keyboard sent it. The second one matters when Backspace took a secret back out; the log would still store the deleted text.
• An entry with any secret gives Personal History nothing, ordinary words included.
• WritingDayFileRecorder hands cleared events (re-checked against the gate, like a write) to a new PersonalHistoryRelay, which delivers them to the controller in order, in valid batches.
• WritingHistoryIngest no longer calls the controller directly.

PersonalHistoryController and the predictor are unchanged.

Options weighed

• Scrub per event: doesn't work. A secret spans events and segments, and what counts depends on text that hasn't arrived yet ("hun" isn't a password; "hunter22" after sudo is). Anything forwarded early is already in the log.
• Predictor refuses to learn/serve what the scrubber would redact: too weak by itself. The predictor learns letter runs (Tr, ub, dor), and the scrubber can't judge a fragment. The log would still keep the raw text.
• Buffer and scrub before forwarding: picked, and buffered per composer entry rather than a second, parallel segment buffer. One grouping model, so anything redacted from a day file is also kept out of Personal History by construction.

What changes from Tilde (in docs/writing-port-ledger.md Deviations)

• Learning waits until the entry closes (new segment, 2 min idle, app switch, quit).
• The socket acks a batch once it's composed. A batch the controller later refuses (storage down) isn't retried; the storage health line still shows "not saving".
• The entry open at quit is handed off but only lands if the app lives long enough.
• A retried batch reaches the log once (recorder's event-ID check), not twice.

Not done

• Encrypted logs and trained models written by dev builds before this still hold whatever was typed. No release shipped Writing before this (1.1.66 has none). Delete all writing clears both; there's no in-place rescrub. Noted as a follow-up in the ledger.
• No live proof of the real Terminal sudo segment shape, same as Writing: scrub passwords, codes, cards and tokens out of Save my writing day files #1905.

Measured

bash scripts/dev/measure-writing-scrubber.sh: on the repo sweep (about 64k entry-sized chunks of docs, Swift, shell scripts and commit messages with no secrets) the scrubber changed 3 entries, so dropping a whole entry from Personal History when anything is redacted costs almost no ordinary learning. Labelled corpus recall is unchanged (92.4%, the known misses).

Tests

21 behavior tests, written by a separate agent from the promises (PersonalHistorySecretTests.swift, plus a section in WritingEntryComposerTests.swift). End to end through WritingHistoryIngest → recorder → relay → a real PersonalHistoryController over an in-memory store:

• A password typed after sudo in com.apple.Terminal is never stored and never predicted. It starts with a control: the same keystrokes fed straight to the controller do serve "dor" after "Tr ub", so the nil check means something. The sudo apt update line still reaches the log.
• Ordinary writing is still stored and learned.
• A secret typed then removed with Backspace stays out.
• Nothing is released while an entry is open; it arrives on the next segment, the idle sweep, or quit.
• Save my writing off, or delete all, with an entry open: it never arrives.
• A retried batch lands once.
• A long entry arrives whole, in order, in batches the controller accepts.

Each test was proven red by a deliberate break (12 mutations, all caught).

Review

Independent review (writing-reviewer agent; codex review couldn't run, since its configured model isn't allowed on this account). Verdict: no blocking code defect. It confirmed the controller is reachable only through relay ← recorder ← composer, deletions never leak, clean == typed is exact, and ordering is serial end to end. Follow-ups taken: ledger wording on where a refused batch shows up, and what a Backspace split costs. Not taken: waiting at quit for the last entry's Personal History write. That's documented as best-effort.

Things worth knowing, all in the safe direction:

• A Terminal command typed within a minute of a one-word sudo password folds into the password's entry (existing scrap rule), so that command isn't learned either.
• A Backspace can leave a word alone on a line in the as-typed check (Python3 in a browser), which withholds that entry from learning.
• Storage failures are now silent to the user: no Writing tab line shows storage health, and the keyboard no longer retries. Listed as a follow-up in the ledger.

Checks

• CI on the merged head 8e239935: app-build, build-and-test, spm-tests, checks and repo-hygiene all pass (hardware smokes skipped; no audio change).
• Merged current main, which retuned the scrubber and fixed the MeetingPromptDetector flake: swift test on the 10 Writing suites passes, 180 tests.
• bash build.sh --no-open builds clean, with no warnings in the changed files.
• check-source-pins --changed-only, check-test-shape and check-known-traps pass.
• bash check.sh on the merged head: Deterministic proof FAIL on two items outside this diff; everything else passed.
  • build-deps and build.sh passed.
  • run-tests.sh: 19,961 of 19,961 passed.
  • Integration smoke, source pins, test shape and doc paths passed.
  • The two failures:
    • The concurrency census has Sources/UI at 54 warnings against a baseline of 51. That comes from other merged commits (SpeakerPeopleSettingsSection, TodayViewModel and others). Sources/TranscriptedWriting stays at 4/4, and no changed file has a warning.
    • TranscriptionTaskManagerRecoveryTests.testFutureDatedAudioCannotPinRecoveryOwnerOrBlockLaterJournalScan (TranscriptedCore) fails locally at load average 50–100. It races file mtimes against a 20 ms liveness window. CI's spm-tests passes it on this head.

🤖 Generated with Claude Code