[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer by iamgroot9444 · Pull Request #64667 · apache/doris

iamgroot9444 · 2026-06-22T05:53:45Z

What problem does this PR solve?

Issue Number: #64646

Related PR: None

Problem Summary:
Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and MATCH / MATCH_PHRASE on Japanese columns end up with poor recall and precision.

This PR adds a built-in kuromoji parser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column:

 INDEX content_idx (`content`) USING INVERTED
 PROPERTIES("parser" = "kuromoji", "parser_mode" = "search");

After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms.

How it works:

Native C++ under be/src/storage/index/inverted/analyzer/kuromoji/, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix.
- The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed.
- search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties.

Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses).

Release note

Support Japanese text tokenization in the inverted index via a new kuromoji parser (PROPERTIES("parser"="kuromoji")), with search/normal/extended modes.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)

  CREATE TABLE test_jp (
    id BIGINT,
    content TEXT,
    INDEX idx_content (content) USING INVERTED
      PROPERTIES("parser" = "kuromoji", "parser_mode" = "search")
  ) ENGINE=OLAP
  DUPLICATE KEY(id)
  DISTRIBUTED BY HASH(id) BUCKETS 1
  PROPERTIES("replication_num" = "1");

  INSERT INTO test_jp VALUES
    (1, '東京都に住んでいます'),
    (2, '日本語の形態素解析エンジン');

  -- search-mode decompounding: 東京都 also matches 東京
  SELECT id FROM test_jp WHERE content MATCH '東京';          -- expect: 1
  SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2

  -- inspect segmentation directly
  SELECT TOKENIZE('東京都に住んでいます', '"parser"="kuromoji","parser_mode"="search"');

Behavior changed:
- No.
- Yes. It adds a new opt-in kuromoji parser. Existing parsers and their output are unchanged; the new behavior only applies to indexes that explicitly set parser="kuromoji".
Does this need documentation?
- No.
- Yes. PR Link to Doris-Website.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-06-22T05:53:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

iamgroot9444 · 2026-06-22T06:23:33Z

run buildall

yiguolei · 2026-06-22T06:27:35Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

iamgroot9444 · 2026-06-22T06:31:13Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

@yiguolei The ICU Analyzer is not good as the Kuromoji. There is huge difference between icu and kuromoji when it comes to morphology of the Japanese words. So I think it worth it adding this new parser.

BiteTheDDDDt · 2026-06-22T07:02:18Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

iamgroot9444 · 2026-06-22T07:29:43Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

This is original code but it is modeled on Apache Lucene's kuromoji.

hello-stephen · 2026-06-22T07:56:03Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

iamgroot9444 · 2026-06-22T09:58:03Z

run buildall

hello-stephen · 2026-06-22T12:00:26Z

BE UT Coverage Report

Increment line coverage 82.40% (791/960) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.51% (21439/39327)
Line Coverage	38.17% (205347/537919)
Region Coverage	34.16% (161044/471416)
Branch Coverage	35.14% (70517/200651)

hello-stephen · 2026-06-22T15:49:21Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.50% (21433/39329)
Line Coverage	38.13% (205092/537920)
Region Coverage	34.11% (160793/471446)
Branch Coverage	35.11% (70468/200678)

hello-stephen · 2026-06-22T16:58:18Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.11% (28441/38375)
Line Coverage	58.02% (309954/534209)
Region Coverage	54.69% (258833/473301)
Branch Coverage	56.10% (112608/200725)

hello-stephen · 2026-06-22T17:05:28Z

FE Regression Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

iamgroot9444 · 2026-06-23T06:16:44Z

run buildall

hello-stephen · 2026-06-23T11:38:51Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T11:55:51Z

FE Regression Coverage Report

Increment line coverage 35.29% (6/17) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T14:39:39Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.72% (21523/39332)
Line Coverage	38.18% (205493/538169)
Region Coverage	34.17% (161179/471738)
Branch Coverage	35.13% (70561/200832)

hello-stephen · 2026-06-23T14:44:04Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.19% (28466/38371)
Line Coverage	58.03% (310151/534436)
Region Coverage	54.77% (259367/473580)
Branch Coverage	56.13% (112755/200875)

iamgroot9444 · 2026-06-24T05:18:18Z

run buildall

hello-stephen · 2026-06-24T06:46:43Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-24T10:23:58Z

BE UT Coverage Report

Increment line coverage 84.20% (842/1000) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.58% (21476/39348)
Line Coverage	38.09% (205045/538313)
Region Coverage	34.07% (160754/471838)
Branch Coverage	35.04% (70387/200890)

hello-stephen · 2026-06-24T14:28:27Z

BE Regression && UT Coverage Report

Increment line coverage 84.02% (468/557) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.22% (28491/38387)
Line Coverage	58.10% (310621/534591)
Region Coverage	55.02% (260603/473686)
Branch Coverage	56.29% (113100/200937)

- Modified CMake configuration to conditionally include the Kuromoji dictionary files only for non-test builds (MAKE_TEST=ON). - Adjusted the custom target for generating the Kuromoji dictionary to reflect the new conditional behavior, ensuring it remains a manual target during unit-test builds.

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

- Introduced a penalty calculation mechanism for segmenting words based on the analyzer mode, improving the accuracy of word segmentation. - Added caching for the current same-category run's byte end to optimize processing of grouped unknown words, reducing unnecessary rescans.

- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.

- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.

- Introduced sorting of CSV file paths before processing to ensure consistent order. - Enhanced error handling during lexicon parsing, including detailed error messages for failed parses. - Added sorting of BuilderWord entries by cost and IDs.

- Implemented a new utility function to decode the first UTF-8 code point from a given string view, enhancing character processing capabilities. - Updated the KuromojiTokenizer to skip unknown tokens based on their character category, improving tokenization accuracy.

- Introduced a new executable `kuromoji_build_dict` to compile the UTF-8 mecab-ipadic source into binary files. - Added custom commands and targets to generate the Kuromoji dictionary, ensuring it is built as part of the overall project unless in test mode.

- Updated the CMake configuration to prepend the Kuromoji JVM library directory to the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables during the dictionary generation process.

- Updated CMake configuration to set DYLD_LIBRARY_PATH and LD_LIBRARY_PATH for better library management during dictionary generation. - Improved README documentation for manual dictionary regeneration steps. - Refactored the Kuromoji dictionary loading logic to cache only successful loads, enhancing error handling. - Added assertions in the Japanese analyzer tests to ensure proper configuration and cleanup of the Kuromoji analyzer state.

- Replaced `qt_` queries with direct SQL calls and assertions to validate results in the Japanese analyzer test suite.

- Updated the build script to allow compilation of the Kuromoji dictionary when the COMPILE_BENCH flag is set. - Modified the dictionary building logic to write temporary files before renaming them atomically, improving reliability during the build process. - Ensured that the mecab-ipadic staging in the third-party build script uses a temporary directory for safer file operations.

…d validation - Enhanced the `decode_utf8` function to include additional validation for continuation bytes, ensuring proper handling of malformed UTF-8 input. - Updated the `first_codepoint` function in `KuromojiTokenizer` to align with the new decoding logic, improving consistency in character processing. - Introduced a Unicode-aware lowercasing function that utilizes ICU for better text normalization, ensuring compatibility with other analyzers.

- Added 2-Clause BSD license information for the Darts-clone library in LICENSE.txt and NOTICE.txt. - Updated the Kuromoji analyzer to support the new 'kuromoji' parser type in function_tokenize.cpp. - Improved character category handling in kuromoji_viterbi.cpp to include KANJINUMERIC. - Adjusted CMake configuration to define the source dictionary directory for Kuromoji tests. - Updated test logic to accommodate the new dictionary directory definition. - Enhanced LICENSE-dist.txt to include Darts-clone and mecab-ipadic details.

iamgroot9444 · 2026-08-31T11:33:34Z

run buildall

hello-stephen · 2026-08-31T16:59:16Z

BE UT Coverage Report

Increment line coverage 76.36% (84/110) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.82% (29461/46894)
Line Coverage	47.85% (308732/645146)
Region Coverage	43.43% (249076/573475)
Branch Coverage	44.97% (115965/257858)

hello-stephen · 2026-08-31T19:23:40Z

BE Regression && UT Coverage Report

Increment line coverage 88.18% (97/110) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.19% (34607/45420)
Line Coverage	61.15% (390016/637782)
Region Coverage	57.32% (327465/571289)
Branch Coverage	58.11% (149401/257121)

np

airborne12

LGTM

github-actions · 2026-09-01T08:29:33Z

PR approved by at least one committer and no changes requested.

…k and SNII_CRC32C_X86 -Wundef (#67451) ### What problem does this PR solve? Issue Number: close #67448, close #67445 Related PR: #64667 (introduced `kuromoji_build_dict`), #66052 (introduced the `SNII_CRC32C_X86` test seam), #66615 (the macOS arm64 allocator workaround this PR reuses) Problem Summary: Two BE build failures on arm64, both found while building on Apple Silicon. **1. `kuromoji_build_dict` fails to link on macOS arm64 (#67448, introduced by #64667)** `sh build.sh --be` fails while linking `bin/kuromoji_build_dict`: ``` ld: fixup error (kind=arm64_b26) at '__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o), B/BL out of range (displacement=-135695688, max is +/-128MB), from 0x10816FF98 to 0x100007250 ('___clang_call_terminate') __TEXT addr=0x100000000, size=0x008310000 google_malloc addr=0x10816eac0, size=0x0000014fc malloc_hook addr=0x10816ffbc, size=0x0000001dc ``` The offline dictionary converter only calls the kuromoji builder/parser, but those return `Status`, and `Status` reaches `config.cpp` (-> `ExecEnv`), `status.cpp` (-> thrift/protobuf/`BackendOptions`) and `stack_util.cpp`, so the tool's link closure is effectively the whole BE and its `__TEXT` exceeds arm64's +/-128 MB direct-branch reach. Apple's linker lays tcmalloc's custom `google_malloc` / `malloc_hook` sections out after `__text` and cannot insert branch islands there, so the branch from tcmalloc back to `___clang_call_terminate` at the start of `__TEXT` cannot be relaxed. The failed link leaves the four dictionary files ungenerated, the install-time guard fires, and no `output/` is produced. The triage comment on #67448 reports the same failure on the `macos-15` runner of the `BE UT (macOS)` workflow. Trimming the tool's link line (the issue's first suggestion) is not possible without refactoring `Status`, so this applies the #66615 treatment per target: on macOS arm64 the tool links against the system allocator, and `gperftools_stubs.cpp` supplies the few gperftools symbols that are still referenced unconditionally. `doris_be` keeps tcmalloc, Linux link lines are unchanged, `kuromoji_dict` stays in `ALL` and the install-time file check stays. **2. `SNII_CRC32C_X86` is undefined on non-x86 targets (#67445, introduced by #66052)** `be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86` with `#if` but only defines it on x86_64. The BE compiles with `-Wundef -Werror`, so on aarch64 the four `#if` sites fail: ``` crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0 [-Werror,-Wundef] ``` The file is `BE_TEST`-only, so `run-be-ut.sh` cannot build on Apple Silicon or Linux aarch64, while CI never sees it (the only aarch64 workflow builds with `MAKE_TEST=OFF`). Define the flag as `0` on the other branch: the x86 branch is unchanged, and the non-x86 build keeps only the portable slice-by-8 reference path, which the `hw_*` seams already fall back to.

iamgroot9444 requested review from BiteTheDDDDt, airborne12 and zclllyybb as code owners June 22, 2026 05:53

iamgroot9444 force-pushed the feat/kuromoji-japanese-analyzer branch from 389fcfb to b79db3c Compare June 22, 2026 09:57

morningman self-assigned this Jun 23, 2026

yiguolei reviewed Jun 24, 2026

View reviewed changes

Comment thread be/src/storage/index/inverted/analyzer/analyzer.cpp

iamgroot9444 force-pushed the feat/kuromoji-japanese-analyzer branch from db0ee69 to 06b4ef6 Compare June 24, 2026 03:59

iamgroot9444 requested a review from yiguolei June 24, 2026 05:18

yiguolei reviewed Jun 26, 2026

View reviewed changes

Comment thread be/dict/kuromoji/README.md

iamgroot9444 added 17 commits August 31, 2026 17:01

Enhance Kuromoji dictionary validation and error handling

845d500

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

Enhance inverted index parser mode handling

784a1ee

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

Add mecab-ipadic staging in build script

163634f

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

Refactor Japanese analyzer tests for clarity and consistency

a7789ee

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

Refactor mecab-ipadic staging logic in build script

805bbfd

- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.

Update Kuromoji analyzer mode handling in inverted index

693f81f

- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.

Fix BE UT (MacOS) workflow

c82c634

- Updated the CMake configuration to prepend the Kuromoji JVM library directory to the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables during the dictionary generation process.

Refactor Japanese analyzer tests to use inline assertions

3f79d4c

- Replaced `qt_` queries with direct SQL calls and assertions to validate results in the Japanese analyzer test suite.

iamgroot9444 force-pushed the feat/kuromoji-japanese-analyzer branch from 091be27 to b91e455 Compare August 31, 2026 11:31

yiguolei approved these changes Sep 1, 2026

View reviewed changes

airborne12 approved these changes Sep 1, 2026

View reviewed changes

airborne12 merged commit 1117132 into apache:master Sep 1, 2026
33 of 35 checks passed

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2026

yiguolei added the dev/4.2.x label Sep 1, 2026

morningman mentioned this pull request Sep 2, 2026

[fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef #67451

Merged

16 tasks

Uh oh!

Conversation

iamgroot9444 commented Jun 22, 2026

What problem does this PR solve?

Release note

Uh oh!

hello-stephen commented Jun 22, 2026

Uh oh!

iamgroot9444 commented Jun 22, 2026

Uh oh!

yiguolei commented Jun 22, 2026

Uh oh!

iamgroot9444 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BiteTheDDDDt commented Jun 22, 2026

Uh oh!

iamgroot9444 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

FE UT Coverage Report

Uh oh!

iamgroot9444 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

FE Regression Coverage Report

Uh oh!

iamgroot9444 commented Jun 23, 2026

Uh oh!

hello-stephen commented Jun 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

iamgroot9444 commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

iamgroot9444 commented Aug 31, 2026

Uh oh!

hello-stephen commented Aug 31, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 31, 2026

BE Regression && UT Coverage Report

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

iamgroot9444 commented Jun 22, 2026 •

edited

Loading