perf(english): name the featured dictionary so it lazy-loads - #223
Merged
ddehilster merged 1 commit intoSep 22, 2026
Merged
Conversation
en-full-feat.dict -> en-feat-full.dict
The engine loads a dictionary word by word, on demand, only when the file name
ends in "full" before the extension. "full" has to be the LAST part of the
name, so a file can contain the word and still be read whole: en-full-feat.dict
ended in "feat", and was loaded eagerly for that reason alone.
The engine says which mode it is in, so it is easy to check. One file in
kb/user, a 60-word input:
en-full-feat.dict [READ dict files time= 2.09 sec] and no other line
en-feat-full.dict [Lazy-loading words from ...en-feat-full.dict]
[READ dict files time= 0.25 sec]
Same bytes, same attributes on the node afterwards. A word still arrives with
its vform, tense, number and, since the frequency work, its freq_<pos>:
jumped verb pos num=2 root=jump vform=past tense=past freq_verb=34
Only the name and the timing change.
Nothing loads this file by name, so nothing breaks. It is worth doing now
rather than later because the file is the source of truth for the English
lexicon and is becoming more attractive to vendor into an analyzer, and the
cost of the wrong name is paid by whoever does that first.
en-lemmas.kbb (2 MB), en-nouns.dict (2 MB) and en-roots.kbb (4 MB) are still
named so that they load eagerly. en-lemmas.kbb is the one that costs something
today -- two analyzers load it with loadkbb -- but renaming it breaks them until
they are updated, so it goes with that change and not this one. lazyload.md now
lists which English files are lazy and which are not, with the measurements.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ddehilster
force-pushed
the
perf/english-feat-dict-lazy-name
branch
from
September 22, 2026 13:17
a1acf3d to
0d0b868
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine loads a dictionary word by word, on demand, only when the file name ends in
fullbefore the extension.fullhas to be the last part of the name — so a file can contain the word and still be read whole at startup.en-full-feat.dictended infeat, and was loaded eagerly for that reason alone.The engine tells you which mode it is in
One file in
kb/user, a 60-word input:Same 8.3 MB of bytes. Same attributes on the node afterwards — a word still arrives with its
vform,tenseandnumberfrom the featured readings:Only the name and the timing change.
Why now
Nothing loads this file by name, so nothing breaks. It is worth doing now rather than later because the file is the source of truth for the English lexicon and is becoming more attractive to vendor into an analyzer — and the cost of the wrong name is paid by whoever does that first.
What is deliberately not in here
en-lemmas.kbb(2 MB),en-nouns.dict(2 MB) anden-roots.kbb(4 MB) are still named so that they load eagerly.en-lemmas.kbbis the one that costs something today — EnglishPhrases and StatuteFrames both pull it in withloadkbb, and renaming it takes EnglishPhrases from 5.32 s to 0.70 s with byte-identical output over 52 inputs. But renaming it breaks both analyzers until they are updated, so it belongs with that change, not this one.lazyload.mdnow lists which English files are lazy and which are not, with the measurements, so the next person does not have to find out by timing it.🤖 Generated with Claude Code