[improvement](parser) Factor common EXPLAIN and CTE prefixes by morrySnow · Pull Request #67427 · apache/doris

morrySnow · 2026-09-02T04:08:11Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: The SQL grammar repeatedly predicts the optional EXPLAIN and CTE prefixes in queryOrDmlStatement, dmlStatement, and query, causing long adaptive lookahead for CTE DML. This change consumes the common prefix once, dispatches the remaining query or DML body, and passes the prefix contexts to the FE plan builder. It preserves non-explainable LOAD/EXPORT/REPLAY/COPY/TRUNCATE branches, nested source CTEs, error positions, and FE command semantics.

This is now an independent PR based directly on master. The seven functional, test, and benchmark files have the same stable patch ID (db03fbcb3a0ca4cfee8a1422bc78e7f753e6ae51) as the original P1 commit. Only the TODO status update that depended on the earlier roadmap PR was omitted.

Benchmark

The benchmark results below are reused from the original P1 validation because the replayed functional patch is identical. The benchmark measures the public parser facade and a pre-tokenized parser-only path; lower latency is better.

Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB heap, -prof gc
Standard run: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement
Longer target run: 3 forks, 6 x 500 ms warmup, 10 x 700 ms measurement
Measurement baseline: d7f44fcfedd; benchmark jar SHA-256 2e33becac22a27d8c40eec7c821c506cbc9bcedfda277725fa8fee40c6d35116
Measurement candidate: 5e0eadb13e9; benchmark jar SHA-256 743c133321dbb1a2283ea1836e5a9568a14ecc2ae350b2fc239d354d7988b423
Harness: fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java

Longer parser-only target run

Workload	Baseline us/op	Candidate us/op	Latency change	Baseline B/op	Candidate B/op	Allocation change
CTE SELECT	43.273 +/- 1.101	43.498 +/- 1.528	+0.52%	73,453.8	73,195.1	-0.35%
CTE INSERT	48.439 +/- 1.233	42.907 +/- 1.236	-11.42%	74,096.5	74,560.4	+0.63%
EXPLAIN CTE INSERT	48.032 +/- 1.446	42.356 +/- 0.580	-11.82%	74,360.5	74,963.1	+0.81%

Public-facade control cases from the standard run

Workload	Baseline us/op	Candidate us/op	Latency change	Baseline B/op	Candidate B/op
SELECT control	2.636 +/- 0.187	2.388 +/- 0.167	-9.42%	5,200.0	4,928.0
EXPLAIN SELECT	7.094 +/- 1.151	6.316 +/- 0.274	-10.97%	10,730.8	10,728.1

The previous grammar performs about 56 tokens of top-level lookahead for a long CTE and then about 62 more tokens in the DML decision. Factoring the prefix removes the repeated prediction, which accounts for the CTE INSERT gains. CTE SELECT remains flat within overlapping confidence intervals, and target allocation changes remain below 1%. The standard end-to-end long-CTE forks were noisy, so this PR makes no precise end-to-end long-CTE or whole-FE latency claim.

Semantic differential

Baseline: current master at 049410596f4d; parser jar SHA-256 923ed2a22142ee9b5dcfefbba5766b9696a653a4218e8208e42a61270e9d986f
Candidate parser jar SHA-256: 3e43f2e025f4152d109a692ba68985c512a155fc2f21345a550911b1ef29af8f
Corpus: all 4,610 tracked *.sql files; manifest SHA-256 567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10
Legacy and ANSI results: 4,275 parsed and 335 rejected in both artifacts
Per-file statement count, exception type, first error position, and full error message are byte-identical; result SHA-256 897b6167aa718d5d14887c1752d4bd849cb1c55f7926b474e5dd2f9562a26fbd
Lexer and token behavior are unchanged by construction.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test
- No need to test or manual test
- Parser and benchmark package: 107 tests passed
- FE targeted parser tests: 111 passed, 0 failures
- FE/parser/benchmark Checkstyle: 0 violations
- Tracked SQL corpus differential: identical in Legacy and ANSI modes
- Original P1 JMH standard and longer target runs completed with 3 forks
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-09-02T04:08:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Query and DML dispatch repeatedly predicted optional EXPLAIN and CTE prefixes across queryOrDmlStatement, dmlStatement, and query. Consume the common prefix once, dispatch the remaining query or DML body, and pass the prefix contexts to the FE plan builder while preserving non-explainable statements and nested source CTEs. In the original parser-only JMH validation, CTE INSERT improves from 48.439 to 42.907 us/op (-11.4%) and EXPLAIN CTE INSERT improves from 48.032 to 42.356 us/op (-11.8%); CTE SELECT remains flat at +0.5%. ### Release note None ### Check List (For Author) - Test: Unit Test and manual performance/semantic validation - Maven parser and benchmark package: 107 tests passed - FE targeted parser tests: 111 tests passed - Tracked SQL corpus: 4,610 files, Legacy and ANSI modes, with byte-identical parse/error signatures - Reused the original P1 JMH parser-only baseline and candidate comparison; the replayed functional patch has the same stable patch ID - Behavior changed: No - Does this need documentation: No

morrySnow · 2026-09-02T08:41:32Z

run buildall

### What problem does this PR solve? Issue Number: None Related PR: apache#67427 Problem Summary: The grammar allowed both querySpecification and its outer query to consume ORDER BY and LIMIT clauses, which introduced full-context ambiguity. Assign each clause group to exactly one owner according to ANSI mode and query level, preserve baseline first-error positions, and adapt the FE visitor to the new labeled contexts. LL_EXACT fallback and ambiguity for the target decisions drop to zero. Ordered SELECT end-to-end latency improves by 7.2% in legacy mode and 6.2% in ANSI mode; parser-only latency improves by 6.5% and 2.8% respectively. The plain SELECT control has no regression above the 3% threshold. ### Release note None ### Check List (For Author) - Test: Unit Test and manual performance/semantic validation - Maven parser and benchmark package: 238 tests passed - FE QueryOrganizationPlanTest and NereidsParserTest: 90 tests passed - Tracked SQL corpus: 4,610 files with identical parse/error signatures in both modes - Reused the original LL_EXACT profile, invalid-SQL matrix, and JMH comparison; the P2 functional patch has the same stable patch ID - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-09-02T10:37:28Z

FE UT Coverage Report

Increment line coverage 82.50% (33/40) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-02T12:00:49Z

TPC-H: Total hot run time: 16717 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 374578172baf27d5de2293f47b952842e2c5c933, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17578	3028	3027	3027
q2	2131	256	226	226
q3	10206	949	520	520
q4	4676	258	202	202
q5	7667	549	379	379
q6	133	119	93	93
q7	530	505	389	389
q8	9248	879	912	879
q9	3412	2380	2352	2352
q10	6503	851	696	696
q11	389	198	192	192
q12	613	267	205	205
q13	18118	1518	1144	1144
q14	158	150	141	141
q15	q16	444	394	369	369
q17	1393	872	827	827
q18	3130	2269	2228	2228
q19	1262	879	745	745
q20	395	291	213	213
q21	5608	1657	1883	1657
q22	330	279	233	233
Total cold run time: 93924 ms
Total hot run time: 16717 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3396	3330	3314	3314
q2	508	402	400	400
q3	2213	2275	2176	2176
q4	1208	1184	893	893
q5	2172	2113	2098	2098
q6	165	119	89	89
q7	1018	984	859	859
q8	1598	1400	1398	1398
q9	3127	3119	3087	3087
q10	1841	1806	1648	1648
q11	356	271	251	251
q12	451	436	338	338
q13	1503	1528	1163	1163
q14	186	177	166	166
q15	q16	396	396	357	357
q17	3580	3344	3219	3219
q18	4782	4414	4745	4414
q19	866	887	877	877
q20	1011	974	832	832
q21	3810	3130	3338	3130
q22	389	348	314	314
Total cold run time: 34576 ms
Total hot run time: 31023 ms

hello-stephen · 2026-09-02T12:07:19Z

TPC-DS: Total hot run time: 81903 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 374578172baf27d5de2293f47b952842e2c5c933, data reload: false

query5	4250	405	325	325
query6	391	136	130	130
query7	4954	434	238	238
query8	289	137	121	121
query9	8677	2890	2871	2871
query10	391	201	185	185
query11	5375	1040	906	906
query12	113	69	69	69
query13	1178	436	317	317
query14	6056	2156	2090	2090
query14_1	1976	1960	1920	1920
query15	172	120	114	114
query16	925	366	373	366
query17	796	480	373	373
query18	2329	340	246	246
query19	169	148	112	112
query20	93	73	70	70
query21	203	101	91	91
query22	5327	5327	5258	5258
query23	6776	6307	6114	6114
query23_1	6169	5913	6049	5913
query24	7264	1096	778	778
query24_1	786	784	783	783
query25	471	283	232	232
query26	1222	225	126	126
query27	2801	412	260	260
query28	4699	1489	1496	1489
query29	915	407	338	338
query30	253	158	130	130
query31	817	397	329	329
query32	125	69	72	69
query33	451	216	163	163
query34	997	848	493	493
query35	396	401	332	332
query36	564	570	519	519
query37	130	76	66	66
query38	1002	837	808	808
query39	475	469	479	469
query39_1	437	500	454	454
query40	200	87	77	77
query41	54	51	54	51
query42	79	73	72	72
query43	239	239	211	211
query44	1028	544	545	544
query45	108	106	104	104
query46	801	816	555	555
query47	762	776	717	717
query48	307	318	216	216
query49	547	237	190	190
query50	713	255	190	190
query51	8076	8107	8174	8107
query52	67	73	65	65
query53	201	203	156	156
query54	221	193	179	179
query55	76	58	55	55
query56	203	151	219	151
query57	781	675	647	647
query58	206	185	179	179
query59	1209	1229	1109	1109
query60	270	195	178	178
query61	155	160	141	141
query62	374	209	188	188
query63	189	146	147	146
query64	2817	659	584	584
query65	1614	1622	1634	1622
query66	1817	253	211	211
query67	9861	9791	9586	9586
query68	2920	1275	793	793
query69	331	219	187	187
query70	666	614	632	614
query71	248	179	203	179
query72	2326	1508	1564	1508
query73	658	593	333	333
query74	1975	1211	1118	1118
query75	1167	1081	959	959
query76	2376	743	556	556
query77	257	275	216	216
query78	3899	3675	3183	3183
query79	2803	810	586	586
query80	1619	320	279	279
query81	522	154	132	132
query82	660	119	95	95
query83	282	208	197	197
query84	301	112	89	89
query85	842	384	298	298
query86	474	181	172	172
query87	1032	975	896	896
query88	2899	2119	2118	2118
query89	279	199	170	170
query90	2058	128	134	128
query91	142	122	100	100
query92	90	70	70	70
query93	2172	1128	715	715
query94	650	262	216	216
query95	527	326	222	222
query96	867	592	266	266
query97	1024	1042	994	994
query98	178	138	131	131
query99	436	348	306	306
Total cold run time: 179499 ms
Total hot run time: 81903 ms

hello-stephen · 2026-09-02T12:11:42Z

ClickBench: Total hot run time: 14.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 374578172baf27d5de2293f47b952842e2c5c933, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.04
query3	0.25	0.12	0.13
query4	1.60	0.10	0.09
query5	0.18	0.16	0.15
query6	1.25	0.70	0.70
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.27	0.21	0.21
query10	0.34	0.34	0.34
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.30	0.30	0.30
query14	0.45	0.46	0.45
query15	0.36	0.36	0.35
query16	0.24	0.22	0.23
query17	0.70	0.69	0.67
query18	0.18	0.16	0.17
query19	1.22	1.18	1.20
query20	0.02	0.01	0.01
query21	15.44	0.16	0.11
query22	5.05	0.05	0.04
query23	16.18	0.25	0.10
query24	3.00	0.31	0.26
query25	0.11	0.04	0.04
query26	0.78	0.15	0.11
query27	0.05	0.03	0.04
query28	3.65	0.54	0.27
query29	12.50	3.18	2.57
query30	0.26	0.11	0.13
query31	2.75	0.37	0.17
query32	3.47	0.30	0.24
query33	1.42	1.52	1.42
query34	15.40	2.10	1.79
query35	1.73	1.74	1.70
query36	0.47	0.28	0.29
query37	0.06	0.03	0.04
query38	0.05	0.03	0.02
query39	0.03	0.02	0.03
query40	0.11	0.07	0.08
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.51 s
Total hot run time: 14.7 s

hello-stephen · 2026-09-02T12:36:24Z

FE Regression Coverage Report

Increment line coverage 67.31% (35/52) 🎉
Increment coverage report
Complete coverage report

englefly

LGTM

github-actions · 2026-09-02T16:24:43Z

PR approved by anyone and no changes requested.

github-actions · 2026-09-02T16:24:45Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? Issue Number: None Related PR: apache#67427 Problem Summary: The grammar allowed both querySpecification and its outer query to consume ORDER BY and LIMIT clauses, which introduced full-context ambiguity. Assign each clause group to exactly one owner according to ANSI mode and query level, preserve baseline first-error positions, and adapt the FE visitor to the new labeled contexts. LL_EXACT fallback and ambiguity for the target decisions drop to zero. Ordered SELECT end-to-end latency improves by 7.2% in legacy mode and 6.2% in ANSI mode; parser-only latency improves by 6.5% and 2.8% respectively. The plain SELECT control has no regression above the 3% threshold. ### Release note None ### Check List (For Author) - Test: Unit Test and manual performance/semantic validation - Maven parser and benchmark package: 238 tests passed - FE QueryOrganizationPlanTest and NereidsParserTest: 90 tests passed - Tracked SQL corpus: 4,610 files with identical parse/error signatures in both modes - Reused the original LL_EXACT profile, invalid-SQL matrix, and JMH comparison; the P2 functional patch has the same stable patch ID - Behavior changed: No - Does this need documentation: No

morrySnow mentioned this pull request Sep 2, 2026

[improvement](parser) Assign query organization to a single owner #67429

Merged

11 tasks

morrySnow force-pushed the codex/antlr4-parser-optimization-todo branch from 5e0eadb to 3745781 Compare September 2, 2026 08:39

morrySnow changed the title ~~[improvement](parser) Factor common EXPLAIN and CTE prefixes~~ [improvement](fe) Factor common EXPLAIN and CTE prefixes Sep 2, 2026

morrySnow changed the title ~~[improvement](fe) Factor common EXPLAIN and CTE prefixes~~ [improvement](parser) Factor common EXPLAIN and CTE prefixes Sep 2, 2026

morrySnow marked this pull request as ready for review September 2, 2026 08:41

morrySnow requested review from 924060929, englefly and starocean999 as code owners September 2, 2026 08:41

englefly approved these changes Sep 2, 2026

View reviewed changes

englefly merged commit f054492 into apache:master Sep 2, 2026
35 of 36 checks passed

morrySnow deleted the codex/antlr4-parser-optimization-todo branch September 2, 2026 16:16

github-actions Bot added the reviewed label Sep 2, 2026

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Sep 2, 2026

Navigation Menu

Uh oh!

[improvement](parser) Factor common EXPLAIN and CTE prefixes - #67427

[improvement](parser) Factor common EXPLAIN and CTE prefixes#67427
englefly merged 1 commit into
apache:masterfrom
morrySnow:codex/antlr4-parser-optimization-todo

morrySnow commented Sep 2, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

morrySnow commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

englefly left a comment

Uh oh!

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

morrySnow commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Benchmark

Longer parser-only target run

Public-facade control cases from the standard run

Semantic differential

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

morrySnow commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

Uh oh!

hello-stephen commented Sep 2, 2026

FE Regression Coverage Report

Uh oh!

englefly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

morrySnow commented Sep 2, 2026 •

edited

Loading