Skip to content

[improvement](parser) Assign query organization to a single owner - #67429

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:codex/antlr4-query-organization-owner
Open

[improvement](parser) Assign query organization to a single owner#67429
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:codex/antlr4-query-organization-owner

Conversation

@morrySnow

@morrySnow morrySnow commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #67427

Problem Summary: The grammar allowed both querySpecification and its outer query to consume ORDER BY and LIMIT, so ANTLR repeatedly entered queryOrganization and reported full-context ambiguities. This change gives every clause group one owner according to ANSI mode and query level, makes the rule structurally non-empty, preserves the existing accepted/rejected SQL matrix and first-error positions, and updates the FE plan builder for the labeled contexts.

P1 (#67427) has now been merged into master. This PR was rebased onto its merge commit (f054492cbb9) and now contains only the P2 functional commit (0bf9111810c). The obsolete stacked P1 commit was removed, so the PR is independently reviewable and mergeable against current master. The P2 stable patch ID remains 73da02c572e08918dc6306adbde23d30c7cdd39d, exactly matching the measured patch; the existing benchmark is reused and was not rerun.

Benchmark

The original P2 benchmark is reused without rerunning it. The rebased P2 functional patch has the same stable patch ID (73da02c572e08918dc6306adbde23d30c7cdd39d) as the measured commit. Lower latency is better. The target input is SELECT a, b, c FROM t WHERE a > 1 ORDER BY a, b DESC LIMIT 20 OFFSET 10; the control input has the same SELECT without query-organization clauses.

  • Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
  • Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB heap
  • JMH: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement; C1-B1-C2 interleaving
  • Measurement baseline: 5e0eadb13e9; parser jar SHA-256 530f0ed45c4bac3a096373a1e932aae0025c603a6f905c3d975a2ad53bd0e7ca; benchmark jar SHA-256 7543595b2a87f55b9b53538336cfedf55c13484302ffb487efcfb4fdd66b1c0b
  • Measurement candidate: 773c4ee1027; parser jar SHA-256 65730a22a5262370a832bda7a04ae4b8e0de5c107be65c5ae80f95c711842f59; benchmark jar SHA-256 47284ffd6779e9d534a19a682de5a4d468f83a62a98e048982807497f1f665b6
  • Harness: fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java

Ordered SELECT target

The candidate change is calculated from the mean of C1 and C2 against B1. Values are us/op with JMH's 99.9% error estimate.

Path Mode Baseline B1 Candidate C1 Candidate C2 Candidate mean change
End-to-end Legacy 9.926 +/- 0.347 9.303 +/- 0.333 9.122 +/- 0.267 -7.2%
End-to-end ANSI 9.170 +/- 0.437 8.709 +/- 0.646 8.492 +/- 0.241 -6.2%
Pre-tokenized parser Legacy 7.653 +/- 0.266 7.246 +/- 0.244 7.069 +/- 0.090 -6.5%
Pre-tokenized parser ANSI 7.344 +/- 0.105 7.181 +/- 0.099 7.099 +/- 0.076 -2.8%

Plain SELECT control

Path Mode Baseline us/op Candidate us/op Change
End-to-end Legacy 6.580 +/- 0.710 6.657 +/- 0.809 +1.2%
End-to-end ANSI 6.944 +/- 0.648 6.448 +/- 0.440 -7.1%
Pre-tokenized parser Legacy 5.684 +/- 1.205 5.084 +/- 0.054 -10.6%
Pre-tokenized parser ANSI 5.313 +/- 0.125 5.193 +/- 0.089 -2.3%

No control-path latency regression exceeds the 3% threshold.

Allocation

The same artifacts were measured separately with -prof gc. Timing under the profiler was noisy, so this table uses only normalized allocation.

Path Mode Baseline B/op Candidate B/op Change
End-to-end Legacy 15,473.6 15,719.6 +1.59%
End-to-end ANSI 15,426.8 15,536.1 +0.71%
Pre-tokenized parser Legacy 12,368.1 12,381.4 +0.11%
Pre-tokenized parser ANSI 12,317.4 12,421.4 +0.84%

All allocation changes are below the 3% threshold.

Profile attribution

  • Before this PR, tracked SQL entered queryOrganization 13,196 times and examined 49,902 lookahead tokens; SSB and Trino profiling reported 40 and 20 related ambiguities.
  • After this PR, the 4,275 parseable tracked SQL files enter the target rule 2,794 times and examine 8,645 lookahead tokens, a reduction of about 79% and 83% respectively.
  • LL_EXACT reports zero fallback and zero ambiguity for both the querySpecification and queryOrganization target decisions.
  • The gain comes from eliminating duplicate ownership and adaptive lookahead, not from lexer or token changes.

Semantic differential

  • Corpus: all 4,610 tracked *.sql files; manifest SHA-256 567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10
  • Result: 4,275 parsed and 335 rejected in both artifacts and in both ANSI modes
  • Per-file success/statement-count or error-class/first-position signature SHA-256: 69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a for every baseline/candidate output
  • An additional 15-case invalid SQL matrix has identical exception class and first-error position in both modes
  • Lexer and token behavior are unchanged by construction

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test
    • No need to test or manual test
    • Parser and benchmark package on rebased branch: 238 tests passed
    • FE QueryOrganizationPlanTest and NereidsParserTest: 90 passed, 0 failures
    • FE/parser/benchmark Checkstyle: 0 violations
    • Tracked SQL corpus differential: original validation identical in Legacy and ANSI modes
    • Original LL_EXACT profile, invalid-SQL matrix, and JMH results reused; benchmark was not rerun after rebase
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
morrySnow force-pushed the codex/antlr4-query-organization-owner branch from 773c4ee to 1293889 Compare September 2, 2026 09:01
@morrySnow morrySnow changed the title [improvement](parser) Assign query organization to a single owner [improvement](fe) Assign query organization to a single owner Sep 2, 2026
@morrySnow morrySnow changed the title [improvement](fe) Assign query organization to a single owner [improvement](parser) Assign query organization to a single owner Sep 2, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67427

Problem Summary: The grammar allowed both querySpecification and its outer query to consume ORDER BY and LIMIT clauses, which introduced full-context ambiguity. Assign each clause group to exactly one owner according to ANSI mode and query level, preserve baseline first-error positions, and adapt the FE visitor to the new labeled contexts. LL_EXACT fallback and ambiguity for the target decisions drop to zero. Ordered SELECT end-to-end latency improves by 7.2% in legacy mode and 6.2% in ANSI mode; parser-only latency improves by 6.5% and 2.8% respectively. The plain SELECT control has no regression above the 3% threshold.

### Release note

None

### Check List (For Author)

- Test: Unit Test and manual performance/semantic validation
    - Maven parser and benchmark package: 238 tests passed
    - FE QueryOrganizationPlanTest and NereidsParserTest: 90 tests passed
    - Tracked SQL corpus: 4,610 files with identical parse/error signatures in both modes
    - Reused the original LL_EXACT profile, invalid-SQL matrix, and JMH comparison; the P2 functional patch has the same stable patch ID
- Behavior changed: No
- Does this need documentation: No
@morrySnow
morrySnow force-pushed the codex/antlr4-query-organization-owner branch from 1293889 to 0bf9111 Compare September 2, 2026 16:32
@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@morrySnow
morrySnow marked this pull request as ready for review September 2, 2026 16:36
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16716 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0bf9111810c61ac7584d0c865cb41287f8de9beb, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17570	3060	3014	3014
q2	2134	256	221	221
q3	10226	989	520	520
q4	4673	251	206	206
q5	7675	557	385	385
q6	141	124	95	95
q7	532	501	376	376
q8	9253	854	883	854
q9	3474	2364	2402	2364
q10	6532	840	711	711
q11	393	201	181	181
q12	613	268	203	203
q13	18114	1531	1168	1168
q14	162	152	141	141
q15	q16	444	399	376	376
q17	1303	826	769	769
q18	3106	2243	2262	2243
q19	1291	856	744	744
q20	377	296	204	204
q21	5592	1708	1867	1708
q22	340	271	233	233
Total cold run time: 93945 ms
Total hot run time: 16716 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3380	3336	3324	3324
q2	506	395	368	368
q3	2238	2310	2165	2165
q4	1204	1184	912	912
q5	2183	2092	2114	2092
q6	172	122	90	90
q7	1016	900	853	853
q8	1596	1395	1405	1395
q9	3127	3100	3119	3100
q10	1861	1831	1635	1635
q11	361	275	255	255
q12	448	426	339	339
q13	1497	1549	1165	1165
q14	164	175	159	159
q15	q16	388	390	364	364
q17	3604	3217	3195	3195
q18	4795	4372	4740	4372
q19	834	795	930	795
q20	1027	969	810	810
q21	3823	3101	3295	3101
q22	395	360	339	339
Total cold run time: 34619 ms
Total hot run time: 30828 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82716 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0bf9111810c61ac7584d0c865cb41287f8de9beb, data reload: false

query5	4262	409	335	335
query6	388	136	121	121
query7	4952	433	230	230
query8	300	124	123	123
query9	8700	2905	2904	2904
query10	398	240	185	185
query11	5386	1056	910	910
query12	121	71	72	71
query13	1197	462	337	337
query14	6025	2228	2151	2151
query14_1	1975	2052	2002	2002
query15	172	119	115	115
query16	927	381	372	372
query17	812	473	369	369
query18	2347	332	245	245
query19	183	153	117	117
query20	92	73	69	69
query21	208	101	88	88
query22	5404	5343	5323	5323
query23	6811	6259	6031	6031
query23_1	6028	6202	6073	6073
query24	7292	1104	775	775
query24_1	792	753	808	753
query25	449	310	287	287
query26	1245	239	127	127
query27	2790	416	272	272
query28	4672	1508	1492	1492
query29	933	449	359	359
query30	258	159	131	131
query31	831	397	337	337
query32	162	77	79	77
query33	466	224	189	189
query34	1006	849	489	489
query35	400	411	338	338
query36	562	600	512	512
query37	126	82	73	73
query38	1007	858	840	840
query39	503	483	470	470
query39_1	461	448	480	448
query40	211	93	81	81
query41	59	56	56	56
query42	110	70	74	70
query43	243	246	211	211
query44	1021	555	548	548
query45	115	103	103	103
query46	757	827	520	520
query47	757	741	727	727
query48	300	294	228	228
query49	540	241	187	187
query50	726	264	200	200
query51	8077	8203	8118	8118
query52	67	73	60	60
query53	192	211	152	152
query54	251	189	149	149
query55	76	64	63	63
query56	310	173	163	163
query57	695	670	667	667
query58	203	163	158	158
query59	1246	1245	1149	1149
query60	241	181	173	173
query61	133	127	111	111
query62	344	210	167	167
query63	168	143	151	143
query64	2642	673	647	647
query65	1606	1594	1592	1592
query66	1834	250	201	201
query67	10025	9740	9726	9726
query68	2893	1178	745	745
query69	348	223	203	203
query70	657	638	617	617
query71	255	175	171	171
query72	2307	1727	1574	1574
query73	654	602	364	364
query74	1959	1246	1137	1137
query75	1185	1103	958	958
query76	2293	730	520	520
query77	254	259	206	206
query78	3994	3617	3264	3264
query79	2755	803	583	583
query80	1620	326	292	292
query81	511	159	131	131
query82	995	126	99	99
query83	281	223	190	190
query84	300	115	90	90
query85	848	375	303	303
query86	474	175	178	175
query87	1025	978	888	888
query88	3005	2111	2114	2111
query89	295	194	173	173
query90	2153	134	130	130
query91	131	120	102	102
query92	108	67	69	67
query93	2141	1019	691	691
query94	653	260	213	213
query95	523	243	297	243
query96	784	558	273	273
query97	1082	1032	1036	1032
query98	173	137	137	137
query99	424	344	311	311
Total cold run time: 180086 ms
Total hot run time: 82716 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.62 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0bf9111810c61ac7584d0c865cb41287f8de9beb, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.60	0.09	0.09
query5	0.17	0.17	0.15
query6	1.23	0.67	0.68
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.22
query10	0.36	0.34	0.34
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.29	0.30	0.33
query14	0.44	0.46	0.44
query15	0.36	0.34	0.36
query16	0.22	0.23	0.23
query17	0.73	0.68	0.72
query18	0.16	0.15	0.15
query19	1.20	1.18	1.14
query20	0.02	0.01	0.01
query21	15.44	0.17	0.13
query22	5.01	0.04	0.04
query23	16.18	0.26	0.11
query24	3.03	0.33	0.26
query25	0.09	0.05	0.05
query26	0.74	0.16	0.13
query27	0.04	0.03	0.04
query28	3.66	0.53	0.29
query29	12.46	3.19	2.54
query30	0.27	0.11	0.12
query31	2.76	0.37	0.17
query32	3.52	0.32	0.23
query33	1.34	1.39	1.37
query34	15.36	2.21	1.78
query35	1.75	1.75	1.71
query36	0.46	0.28	0.29
query37	0.07	0.04	0.04
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.08	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.02	0.03
Total cold run time: 90.3 s
Total hot run time: 14.62 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants