fix: 词典替换的 CJK 边界校验（不改 ASCII 行为） - #96
Merged
Merged
Conversation
Vocabulary replacement only checked ASCII word boundaries when the entry's own first/last character was ASCII, so a short CJK entry could rewrite the CJK tail of a mixed identifier (e.g. 密码 inside Wi-Fi密码). Apply the boundary check to the adjacent text character on both sides, which keeps pure ASCII rule behavior identical while giving CJK-only and mixed entries the same validation. Longest-match-first ranking and non-overlapping left-to-right application are unchanged, and empty originals are ignored. Co-authored-by: multica-agent <github@multica.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
VocabularyReplacementEngine只在词条自身的首/尾字符是 ASCII 单词字符时校验边界；纯 CJK 与以 CJK 结尾的混合词条不校验，导致短 CJK 词条会改写混合标识符的 CJK 尾部，例如词典里有密码 -> PW时，Wi-Fi密码被改写成Wi-FiPW，且发生在 LLM 之前（TextProcessor.swift:35），下游无法恢复。改动
Sources/Processing/VocabularyReplacementEngine.swift：Wi-Fi密码）：获得同样的校验，不再改写混合标识符的 CJK 尾部。original词条在排序阶段过滤，避免零长度匹配导致的死循环。行为对照：
Wi-Fi密码密码->PWWi-FiPWWi-Fi密码Wi-Fi密码密码->PW,Wi-Fi密码->WFPWFPWFP菜单栏栏->X菜单X菜单X菜单栏栏->X,菜单栏->MENUMENUMENU使用开放类型输入开放类型->OpenType使用OpenType输入使用OpenType输入（不变）rapidapi->APIrapidrapid（不变）测试
新增
Tests/OpenTypeTests/VocabularyReplacementEngineTests.swift（9 条），覆盖：长条目优先、短 CJK 不误替混合标识符尾部、混合条目在中文句中正常匹配、单字 CJK 在中文句中仍替换、同位置不重叠、ASCII 边界行为不变、personal 优先于 industry、空词条忽略。swift build→Build complete! (4.97 secs)swift test --filter 'VocabularyReplacementEngineTests|PromptAndProcessingTests|PersonalDictionaryLearningTests|IndustryLexiconTests|TextProcessingSnapshotTests'→ 39 passed / 0 failuresswift test --filter 'Processing|Prompt|Fidelity|Sanitizer|Dictionary|Lexicon|Text|Translation|QualityProbe'→ 285 passed / 1 skipped / 0 failures（含行业词库 16 条评估用例与非目标文本保真）范围
仅上述 2 个文件；无阈值、无诊断、无 UI 变更。