perf(english): name the lemma table so it lazy-loads - #224
Merged
ddehilster merged 1 commit intoSep 22, 2026
Merged
Conversation
en-lemmas.kbb -> en-lemmas-full.kbb The engine loads a knowledge base word by word, on demand, only when the file name ends in "full" before the extension. This one did not, so 2 MB was read whole at startup to serve a few dozen lookups. NOT breaking, despite renaming a file two analyzers depend on. loadkbb resolves against the ANALYZER's own kb/user directory, and EnglishPhrases and StatuteFrames each vendor their own copy of this table, so the library's file name is not what they load. Verified by running EnglishPhrases with its vendored copy renamed while the library file still had the old name: it loaded and answered normally. What the rename does is let an analyzer vendor the file under a name that lazy-loads, which is where the gain is: analyzer run 5.32 s -> 0.70 s on a 60-word input phrases.txt byte-identical across all 52 inputs lemmas found 749 either way so the lazily loaded table answers every lookup the eager one did. The analyzers keep working until they re-vendor, and get the speedup when they do. The file is regenerated rather than moved, so the header inside it names the file it now is. Its content is otherwise unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine loads a knowledge base word by word, on demand, only when the file name ends in
fullbefore the extension. This one did not, so 2 MB was read whole at startup to serve a few dozen lookups.Not breaking, despite two analyzers depending on this table
loadkbbresolves against the analyzer's ownkb/userdirectory, and EnglishPhrases and StatuteFrames each vendor their own copy — so the library's file name is not what they load. Verified by running EnglishPhrases with its vendored copy renamed while the library file still had the old name: it loaded and answered normally.What this rename does is let an analyzer vendor the file under a name that lazy-loads. That is where the gain is:
phrases.txtSo the lazily loaded table answers every lookup the eager one did; what goes away is reading 2 MB to serve a few dozen words. The analyzers keep working until they re-vendor, and get the speedup when they do.
The file is regenerated rather than moved, so the header inside it names the file it now is. Its content is otherwise unchanged.
With this,
languages/Englishhasen-nouns.dict(2 MB) anden-roots.kbb(4 MB) left eager — neither is loaded into an analyzer anywhere in this repository, so neither rename can be verified by a run.lazyload.mdrecords that.🤖 Generated with Claude Code