Repository navigation
fix: infinite hang on input chars sorting past the last full-dict word (#669) - #670
Merged
Merged
Conversation
…ord (#669) nlp.exe hung forever (observed 11+ hours) when input contained an em-dash (U+2014) or en-dash (U+2013) -- and, generally, any character that sorts after every word in a lazily-loaded "*full" dictionary. Root cause: the lazy full-dict/full-kbb binary search (findFullWord, added in #649). nextDictKey/nextKBBKey set `afterKey = f.stream.tellg()`. Reading the file's last line (no trailing newline) leaves the stream at EOF, where tellg() returns -1. For a search key greater than every word, the lower-bound search walks to that last line; `afterKey = -1` then makes searchDictFile/searchKBBFile do `lo = -1`, so `lo < hi` stays true and the search restarts from the middle forever. Normal words sort within the file and converge before hitting this, so only past-the-end keys (em/en-dash, other high code points) trigger it. Fix: - nextDictKey / nextKBBKey: clamp `afterKey` to `f.size` when tellg() returns negative (EOF after the last line). That is the true "position past the last key", so the search terminates cleanly and reports the word absent. - searchDictFile / searchKBBFile: defensive guard -- if `afterKey <= lo` in the key<str branch, break instead of moving lo backwards/sideways (belt and suspenders against any future non-progress). Verified with the parse-pt-br analyzer (which ships pt-full.dict): em-dash, en-dash, and a full Portuguese travessao dialogue all previously hung and now complete; in-dict words still get their POS/meaning attributes; parse-en-us unaffected. Bumps engine to 3.5.7. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a self-contained fixture analyzer that reproduces the infinite hang: a tiny sorted "zz-full.dict" (last line intentionally has no trailing newline, so the final getline hits EOF and tellg() returns -1) plus an em-dash input (which sorts after every dict word, driving the lower-bound search to that last line). Validated against the pre-fix engine: the fixture hangs (timeout) without the fix and completes with it. Wired into the Windows and Linux builds; a hang trips the step timeout and fails the job. A .gitattributes marks the dict binary so git never appends a newline or normalizes EOLs (which would mask the bug). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #669.
Symptom
nlp.exehangs forever (you observed 11+ hours) when input contains an em-dash—(U+2014) or en-dash–(U+2013). Onlyana001.treeis written, then it spins. ASCII hyphen is fine. Reproduces with the parse-pt-br analyzer (Portuguese dialogue uses travessões).Root cause — the lazy full-dict binary search (#649)
The hang is in
findWordConcept→findFullWord→searchDictFile/searchKBBFile, the binary search over a sorted*full.dict/*full.kbb(parse-pt-br ships a 17 MBpt-full.dict).nextDictKey/nextKBBKeysetafterKey = f.stream.tellg(). Reading the file's last line (no trailing newline) leaves the stream at EOF, wheretellg()returns −1. An em-dash (bytese2 80 94) sorts after every word, so the lower-bound search walks to that last line, getsafterKey = -1, and doeslo = -1—lo < histays true and the search restarts from the middle forever:Normal words sort within the file and converge before reaching the last line, so only past-the-end code points (em/en-dash, etc.) trigger it. This is new because it rides on the lazy-full-dict feature (#649), shipped in 3.5.x.
Fix
nextDictKey/nextKBBKey: clampafterKeytof.sizewhentellg()returns negative — the true "position past the last key" — so the search terminates and reports the word absent.searchDictFile/searchKBBFile: defensiveif (afterKey <= lo) break;in thekey<strbranch (never movelobackwards/sideways).Verification (parse-pt-br, which ships
pt-full.dict)— Bom dia.(em-dash)– Bom dia.(en-dash)In-dict words still resolve (
Bom→("s" 1)("adj" 1)("meaning" concept:"bom")), so normal dictionary lookup is unaffected. Bumps engine to 3.5.7.🤖 Generated with Claude Code