feat(lite): lazily load "*-full" dictionaries one word at a time

[ddehilster]

Summary

A dictionary file whose stem ends in -full (e.g. en-full.kbb, en-full.dict) is no longer read whole into the KB. It is kept open and binary-searched per word the first time that word is needed during analysis, so only the text's words are loaded instead of the entire (~370k-word) dictionary.

On the TestEnFullKbb analyzer with a 1019-word text this took KB load from ~291 s → ~1–2 s.

What it does

• Detection + open — readKBBs/readDicts detect *-full files and open them lazily. The file must be sorted (binary-search precondition); if not, the engine warns and falls back to the original full load.
• Shared parsers — refactored readKBB/readDict into parseKBBLine/parseDictLine so the lazy loader reuses the exact same parsing as the full load (no behavior drift for non-full files).
• Per-word lookup in CG::findWordConcept — on a KB miss, binary-search the open file(s):
  • the flat .dict (which carries pos directly on the word) is consulted first, so the tokenizer's findAttrs(word) attaches pos;
  • the .kbb is also consulted to populate the top dictionary concept (getConcept(findRoot(),"dictionary")) and to link the word to its kbb entry via a meaning attribute (mirrors the full-load matchDictKB pairing).
• Caching — found words cached via sym->con / dict_get_word; a run-wide negative cache skips re-searching absent words.
• missing-words.log — words not found in any *-full file are recorded (canonical lowercase, deduped) into each input file's output *_log folder, independently per file.

Validation (built with MSVC/x86 + vcpkg ICU)

• All present words resolve and get their pos; e.g. long → noun/verb/adj/adv/abbr.
• meaning attribute appears on every token: ("meaning" concept:"<word>").
• Output kbdumptree of dictionary contained 530 words = 539 unique text words − 9 genuinely missing — i.e. only the text's words loaded.
• missing-words.log verified: all 9 logged words confirmed absent from both files; multi-file run confirmed each file's log is independent.

Files

• cs/libconsh/cg.cpp, include/Api/consh/cg.h — lazy loading, binary search, caching, meaning link, missing-words log.
• lite/nlp_engine.cpp — points the missing-words log at each file's *_log output folder.

Note (not in this PR)

While testing I found a pre-existing crash: nlp.exe builds a std::string from a null workdir when -WORK is omitted (main.cpp:53). Unrelated to this change; happy to fix separately.

🤖 Generated with Claude Code