Stabilize prompt prefix: move volatile context to the user turn - #118
Merged
Merged
Conversation
Split assembled prompts into a byte-stable, settings-only instruction prefix and a volatile context tail (screen, memory, input target, time, personal dictionary/lexicon/edit rules). The volatile tail now travels in the user turn so the prefix is identical across requests, enabling future prompt/KV-cache reuse and provider-side prefix caching. Legacy builders (buildSystemPrompt, buildCommandSystemPrompt, formattingSystemPrompt, commandSystemPrompt, systemPromptWithPersonalContext, selectionEditSystemPromptWithPersonalContext) keep their signatures and concatenated output, so existing behavior and tests are preserved. Local MLX KV reuse and remote cache_control are deferred to follow-up changes; the mlx-swift-lm prefix-cache spike is recorded in the SDLC verification artifact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every dictation currently rebuilds one opaque system prompt that mixes byte-stable instructions with per-request context, and
LLMEnginecreates a freshChatSessionper call. No prefix is ever reusable, so each request re-prefills the full system prompt.This change makes the instruction prefix byte-stable:
PromptAssembly { stablePrefix, volatileContext, userPrompt(containing:) }.PromptBuildergainsbuildFormattingAssembly/buildCommandAssembly; the existingbuildSystemPrompt/buildCommandSystemPromptdelegate to them and return the identical concatenated string.TextProcessorgainspersonalContextSectionsandformattingAssembly/commandAssembly; processing + personal context (screen, memory, input target, timestamp, dictionary, lexicon, edit rules) move to the volatile tail.stablePrefixas the system prompt andvolatileContext + userContentas the user turn. Translation already had no volatile system context.Local MLX KV reuse and remote
cache_controlare deferred to follow-up changes; the mlx-swift-lm 3.31.4 prefix-cache spike is recorded in the SDLC artifact.Testing
swift build— passswift test— 798 passed, 18 skipped, 0 failures (+1 Swift Testing suite)swift test --filter StablePromptPrefixTests— 4 passedbash scripts/sdlc-checks.sh— passbash scripts/ci-basic-checks.sh— passEnvironment note: the ambient toolchain is CommandLineTools (no XCTest); tests were run with
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift test.SDLC
Bundle:
docs/sdlc/changes/2026-10-01-stable-prompt-prefix-cache/(intent/spec/plan approved; verificationpending approval).Residual risk
Moving volatile context from the system role to the user turn preserves model-visible content and order, but models can weight roles differently; real-model manual QA on the main and command paths is still required before merge. The performance outcome (faster prefill) is not realized until the deferred cache work lands.