[improvement](parser) Skip hidden tokens in internal SQL parsing - #67434
Open
morrySnow wants to merge 1 commit into
Open
[improvement](parser) Skip hidden tokens in internal SQL parsing#67434morrySnow wants to merge 1 commit into
morrySnow wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Internal Nereids AST parsing allocates whitespace and ordinary line-comment tokens even though the parser does not consume them. Add an opt-in lean lexer mode for direct AST paths, preserve full-token public consumers, hints, source offsets, and diagnostics, and fall back to full tokens on syntax errors. JMH GC profiling shows 22.9%-49.2% lower tokenization allocation, 3.9%-12.0% lower parser allocation, and 2.3%-7.2% lower FE LogicalPlan allocation across representative SQL workloads.
### Release note
None
### Check List (For Author)
- Test: Unit Test and benchmark
- Maven fe-sql-parser suite: 88 tests passed
- LeanTokenModeTest and EncryptSQLTest: 7 tests passed
- Tracked SQL corpus: 18,440 full/lean mode cases matched
- JMH benchmark with GC profiler
- Behavior changed: No
- Does this need documentation: No
morrySnow
force-pushed
the
codex/antlr4-lean-token-mode
branch
from
September 2, 2026 08:06
1317d48 to
5ef2869
Compare
morrySnow
changed the title
morrySnow
changed the title
Contributor
Author
|
run buildall |
morrySnow
marked this pull request as ready for review
September 2, 2026 08:08
morrySnow
requested review from
924060929,
englefly and
starocean999
as code owners
September 2, 2026 08:08
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 16770 ms |
Contributor
TPC-DS: Total hot run time: 81388 ms |
Contributor
ClickBench: Total hot run time: 14.62 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Internal Nereids AST parsing allocates whitespace and ordinary line-comment tokens even though ANTLR does not consume hidden-channel tokens. This independent PR adds an opt-in lean lexer mode for direct AST construction and skips only
WSandSIMPLE_COMMENT. Public lexer, scan, and comment-normalization paths remain in full-token mode;BRACKETED_COMMENTremains on channel 2 for hints; character offsets are unchanged. Syntax failures are retried with full tokens so the existing exception type, position, and diagnostic text are preserved.Benchmark
Lower allocation is better. Normalized allocation (
gc.alloc.rate.norm) is the primary metric because wall-clock results were affected by host scheduling noise.-prof gc049410596f4(upstream/master); parser JAR SHA-256923ed2a22142ee9b5dcfefbba5766b9696a653a4218e8208e42a61270e9d986f; benchmark JAR SHA-256f88c09c16f5872ff1e8cef2368e83622cf9cc6e4217350cc845a862dbaa92f565ef28695247; parser JAR SHA-25620104fd49c902833898e1132f6fb6b2bf200a7608356546d92829543e581593e; benchmark JAR SHA-2566ecd1a5566f386729b0a7c8b1c4f3db7e7719975cb06512ed04a5f1fc8e33b64Full and lean values below are measured in the same candidate artifact, so token mode is the only changed variable. Values are bytes/op.
The reduction comes from avoiding
CommonTokenallocation for whitespace and ordinary line comments. Hint and default-channel tokens are still allocated. A frozen-artifact B/C/C/B control with lean mode disabled showed no normalized-allocation regression forSELECT 1(-0.8%) or a typical SELECT (-0.6%). Control latency was scheduling-sensitive, including one severely disturbed baseline fork, so it is not used as acceptance evidence.Commands:
cd fe mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark \ -Dmaven.build.cache.enabled=false package java -jar fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \ LeanTokenModeBenchmark -prof gcSemantic differential
*.sqlfiles; SHA-256 ofgit ls-files -s -- '*.sql'is567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10masterand candidate both accepted 4,063 files and rejected 547 files in Legacy and ANSI modes8036532e957b06a1bbc106338a1caa08cbc9acbd2d08f6eb4fcd7665f6881b2a; ANSIe27f127f60bcc1ce47bbc4ec4cc83351acc0adceb735488fdcb9853727e2c394noBackslashEscapes, and ANSI modes had identical filtered token tuples and CST signaturesRelease note
None
Check List (For Author)
Test
fe-sql-parser: 88 passed, 0 failuresLeanTokenModeTestandEncryptSQLTest: 7 passed, 0 failures viarun-fe-ut.sh; all 60 reactor modules built successfullyBehavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)