feat(fts): add MeCab tokenizer for Japanese + tokenizer registry refactor - #57
Closed
chiangchenghsin-hash wants to merge 114 commits into
Closed
feat(fts): add MeCab tokenizer for Japanese + tokenizer registry refactor#57chiangchenghsin-hash wants to merge 114 commits into
chiangchenghsin-hash wants to merge 114 commits into
Conversation
…d_subdirectory for CI
…d_subdirectory for CI
Contributor
Preference is to add third_party as a submodule if necessary instead of vendoring. We're already doing it for datasets submodule in various language bindings (e.g. ladybug-python). Also why does this PR have 112 commits? Given that FTS is a widely used module and the impact of such a large diff on the text size, it's probably best maintained as an out of tree extension. Adding third_party/opengql was a mistake. It should be added to main repo's third_party and then added as a submodule. |
…; fix Linux build
Restructure the vendored third_party layout to fix CI and follow the
conventions already used in this repo:
- Move third_party/mecab -> fts/third_party/mecab (same pattern as the
vendored snowball stemmer). The previous repo-root location broke the
CI build: under the engine build system the extensions repo is checked
out at <engine>/extension, so the fts/CMakeLists.txt path resolved
outside the extensions tree and add_subdirectory() failed with 'not an
existing directory'.
- Drop the vendored third_party/cppjieba copy: the engine repo already
vendors cppjieba (with dictionaries) in its third_party, which the
original fts/CMakeLists.txt referenced via PROJECT_SOURCE_DIR. The
duplicate added here only carried headers, not the dict files, and was
~110 files of extra diff.
- fts/CMakeLists.txt now only adds the mecab bits on top of the original
upstream file (include dir, add_subdirectory, link, DLL_EXPORT for
tokenizer.cpp, ipadic dict copy step).
- mecab/CMakeLists.txt Linux fixes:
* HAVE_WINDOWS_H only on Windows (utils.cpp #includes <windows.h>
under it; defining it on Linux breaks the build)
* HAVE_GCC_ATOMIC_OPS on non-Windows (tagger.cpp instantiates
read_write_mutex, which only compiles under HAVE_ATOMIC_OPS)
* HAVE_ICONV on non-Windows (mecab-dict-index needs glibc iconv to
compile the EUC-JP ipadic CSV into UTF-8; without it the conversion
is a silent no-op and the dictionary is corrupt)
* MECAB_USE_UTF8_ONLY: skips the 4MB ucstable.h legacy-charset table
(upstream --enable-utf8-only configuration). The extension only
loads UTF-8 dictionaries.
- Add files that were missing from the vendored tree and referenced by
MECAB_SRCS / tokenizer.cpp: mecab.h, tagger.cpp, feature_index.cpp.
The vendored sources follow upstream's autoconf convention and guard
every system header/feature behind HAVE_* checks (config.h normally
supplies them). Without them GCC fell back to MSVC-only paths and the
build failed with:
utils.h:35: error: expected initializer before 'uint64_t'
(falls into 'typedef unsigned __int64 uint64_t' - MSVC-only)
mmap.h:146: error: 'O_RDONLY' was not declared in this scope
mmap.h:152: error: '::open' has not been declared
(fcntl.h / unistd.h / sys/stat.h never included)
Added the defines upstream's configure detects on Linux:
HAVE_STDINT_H, HAVE_SYS_TYPES_H, HAVE_SYS_STAT_H, HAVE_FCNTL_H,
HAVE_STRING_H, HAVE_SYS_MMAN_H, HAVE_UNISTD_H, HAVE_DIRENT_H,
HAVE_MMAP, HAVE_TLS_KEYWORD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a MeCab tokenizer for Japanese to the FTS extension, together with a small tokenizer registry refactor that makes adding new tokenizers a one-line registration, and a fix for a silent correctness bug in incremental index updates.
While working on this, I noticed the FTS tokenizer code referenced
third_party/cppjiebabut the directory was never vendored in this repo, so the Jieba (Chinese) tokenizer could not be built from source. This PR includes the vendoredcppjiebatree (from the upstream cppjieba project, MIT license) so the extension builds self-contained.Changes
New tokenizer architecture (
fts/src/utils/tokenizer.{h,cpp}):ITokenizerinterface +TokenizerRegistry(factories registered by name) +TokenizerPool(instances cached per name+params and shared across index builds, incremental inserts and queries).FTSConfig.tokenizerParams(unordered map) replaces the hardcodedjiebaDictDirfield; serialization is backward compatible (a magic marker distinguishes new catalogs from legacy ones, whose dict dir is folded intotokenizerParams["jieba_dict_dir"]).MeCab (Japanese) tokenizer:
third_party/mecab/(mecab 0.996, BSD-3) with the MSVC compatibility fixes needed for modern MSVC (missingWPATH_FORCEdefine,registerkeyword removal,std::binary_function, gatedunsigned long longstream operator, static-linkDLL_EXPORThandling).mecab-dict-index -f euc-jp -t utf-8), so no large dictionary files are committed.tokenizer := 'mecab'option withmecab_dict_dirparameter; default dictionary is copied next to the built extension like the jieba dict.fts_japanese.testcovering basic queries, incremental insert, and a custom dict dir.Bug fix — incremental insert used the wrong tokenizer:
createFTSIndexQueryrewrote the internal_CREATE_FTS_INDEXcall without forwardingtokenizer/jieba_dict_dir, so the index's internal config silently fell back to'simple'. Rows inserted after index creation were tokenized with whitespace splitting, so Chinese/Japanese terms were never indexed (queries could not match new rows). The parameters are now forwarded, with a regression test (ChineseIncrementalInsertinfts_chinese.test).Why it matters
simpletokenizer — affecting existing Chinese users today, not just the new Japanese path.Verification
Built with MSVC (Visual Studio 18 2026, Release) on Windows; all tests pass:
fts_chinese.test(Jieba) incl. new incremental-insert regression casefts_japanese.test(MeCab) — basic, incremental insert, custom dict dirfts_basic/error.testsuites unaffected (tokenizer error message updated in sync)Note: I could not run the full
.testsuite in CI here (test targets are not enabled in my local build config), but all touched paths are covered by the smoke tests above.Thank you for reviewing!