Skip to content

Revert "[feature](lance) Add Rust-based Lance format reader for AI-native data (#62182) - #65303

Merged
Gabriel39 merged 1 commit into
apache:masterfrom
zhangstar333:revert_origin_lance_pr
Jul 10, 2026
Merged

Revert "[feature](lance) Add Rust-based Lance format reader for AI-native data (#62182)#65303
Gabriel39 merged 1 commit into
apache:masterfrom
zhangstar333:revert_origin_lance_pr

Conversation

@zhangstar333

@zhangstar333 zhangstar333 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

This reverts commit b613c37. #62182
will be later support lance catalog full.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29944 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60ef7bba55674eeb5f09ed2c2e4ac742a9f770b1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17705	4161	4190	4161
q2	2045	340	222	222
q3	10215	1490	854	854
q4	4677	483	339	339
q5	7530	855	590	590
q6	182	173	134	134
q7	761	834	616	616
q8	9716	1600	1657	1600
q9	6564	4413	4393	4393
q10	6789	1816	1540	1540
q11	517	347	311	311
q12	751	554	422	422
q13	18210	3376	2811	2811
q14	263	270	247	247
q15	q16	785	793	710	710
q17	976	998	951	951
q18	7091	5758	5637	5637
q19	1309	1249	1052	1052
q20	746	686	584	584
q21	5960	2658	2475	2475
q22	433	361	295	295
Total cold run time: 103225 ms
Total hot run time: 29944 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4533	4383	4404	4383
q2	290	325	216	216
q3	4659	4958	4404	4404
q4	2076	2140	1390	1390
q5	4518	4368	4380	4368
q6	226	178	127	127
q7	2324	1950	1710	1710
q8	2554	2256	2262	2256
q9	8284	7948	7877	7877
q10	4776	4773	4351	4351
q11	588	428	401	401
q12	761	766	540	540
q13	3336	3699	2938	2938
q14	301	322	277	277
q15	q16	723	722	632	632
q17	1361	1370	1476	1370
q18	8060	7367	7248	7248
q19	1175	1144	1057	1057
q20	2274	2212	1920	1920
q21	5253	4597	4454	4454
q22	508	442	389	389
Total cold run time: 58580 ms
Total hot run time: 52308 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 180098 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60ef7bba55674eeb5f09ed2c2e4ac742a9f770b1, data reload: false

query5	4317	628	496	496
query6	465	224	210	210
query7	4894	595	323	323
query8	364	187	173	173
query9	8767	4016	4088	4016
query10	497	359	292	292
query11	5958	2358	2191	2191
query12	161	102	103	102
query13	1264	614	455	455
query14	6307	5336	5009	5009
query14_1	4320	4299	4283	4283
query15	219	209	188	188
query16	1045	501	470	470
query17	1169	730	595	595
query18	2454	487	357	357
query19	226	193	164	164
query20	112	108	110	108
query21	235	161	138	138
query22	13588	13607	13438	13438
query23	17448	16586	16387	16387
query23_1	16296	16227	16294	16227
query24	7566	1769	1308	1308
query24_1	1332	1295	1308	1295
query25	594	487	410	410
query26	1337	345	209	209
query27	2678	584	374	374
query28	4502	2028	2019	2019
query29	1104	633	515	515
query30	344	267	226	226
query31	1123	1108	995	995
query32	153	67	60	60
query33	542	345	269	269
query34	1209	1216	648	648
query35	785	791	687	687
query36	1415	1404	1224	1224
query37	161	106	98	98
query38	1939	1713	1637	1637
query39	946	932	942	932
query39_1	900	878	877	877
query40	271	163	139	139
query41	66	64	63	63
query42	97	93	88	88
query43	317	321	275	275
query44	1422	759	778	759
query45	218	183	179	179
query46	1062	1191	749	749
query47	2359	2351	2208	2208
query48	385	440	290	290
query49	586	411	310	310
query50	1042	438	335	335
query51	10815	10716	10671	10671
query52	86	86	74	74
query53	279	277	205	205
query54	282	245	229	229
query55	75	73	68	68
query56	316	316	269	269
query57	1450	1409	1324	1324
query58	294	265	235	235
query59	1566	1628	1466	1466
query60	317	268	255	255
query61	167	147	152	147
query62	713	654	594	594
query63	250	201	206	201
query64	2531	769	595	595
query65	4845	4798	4810	4798
query66	1833	521	378	378
query67	29765	29514	29302	29302
query68	3153	1570	938	938
query69	417	336	281	281
query70	1071	963	916	916
query71	343	329	294	294
query72	3048	2610	2388	2388
query73	858	769	456	456
query74	5139	4937	4743	4743
query75	2630	2589	2206	2206
query76	2351	1188	792	792
query77	360	386	289	289
query78	12323	12457	11722	11722
query79	1503	1216	780	780
query80	1298	539	455	455
query81	520	328	287	287
query82	1089	162	122	122
query83	387	320	300	300
query84	288	161	130	130
query85	1049	594	500	500
query86	500	284	268	268
query87	1822	1825	1790	1790
query88	3733	2801	2776	2776
query89	459	406	365	365
query90	1951	203	198	198
query91	201	191	164	164
query92	64	62	55	55
query93	1781	1477	984	984
query94	801	362	306	306
query95	798	483	567	483
query96	1110	793	363	363
query97	2696	2677	2601	2601
query98	223	206	201	201
query99	1235	1140	1026	1026
Total cold run time: 267814 ms
Total hot run time: 180098 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60ef7bba55674eeb5f09ed2c2e4ac742a9f770b1, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.04	0.05
query3	0.25	0.13	0.13
query4	1.60	0.14	0.13
query5	0.24	0.25	0.22
query6	1.28	1.07	1.07
query7	0.04	0.01	0.01
query8	0.06	0.04	0.05
query9	0.37	0.31	0.31
query10	0.58	0.54	0.55
query11	0.19	0.14	0.14
query12	0.18	0.14	0.15
query13	0.48	0.48	0.47
query14	1.01	1.01	1.01
query15	0.60	0.59	0.60
query16	0.32	0.32	0.32
query17	1.13	1.11	1.11
query18	0.21	0.20	0.22
query19	2.10	2.00	1.95
query20	0.02	0.01	0.01
query21	15.46	0.20	0.15
query22	4.95	0.05	0.04
query23	16.13	0.31	0.13
query24	3.08	0.42	0.36
query25	0.11	0.08	0.04
query26	0.74	0.20	0.15
query27	0.05	0.04	0.04
query28	3.70	0.91	0.52
query29	12.47	4.07	3.23
query30	0.27	0.16	0.16
query31	2.77	0.61	0.32
query32	3.21	0.59	0.49
query33	3.14	3.18	3.20
query34	15.67	4.26	3.53
query35	3.49	3.49	3.56
query36	0.57	0.43	0.43
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.16	0.16
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.08 s
Total hot run time: 25.07 s

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 7, 2026
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.52% (23190/41028)
Line Coverage 40.08% (225045/561519)
Region Coverage 36.05% (177766/493113)
Branch Coverage 37.09% (78955/212889)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.54% (23208/41046)
Line Coverage 40.10% (225198/561570)
Region Coverage 36.08% (177996/493310)
Branch Coverage 37.10% (79046/213049)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 30071 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60ef7bba55674eeb5f09ed2c2e4ac742a9f770b1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17626	4143	4167	4143
q2	2033	336	204	204
q3	10257	1483	859	859
q4	4687	473	338	338
q5	7537	869	575	575
q6	190	177	138	138
q7	795	820	617	617
q8	9387	1770	1707	1707
q9	5612	4407	4403	4403
q10	6784	1765	1526	1526
q11	512	342	314	314
q12	712	578	433	433
q13	18080	3536	2772	2772
q14	276	264	251	251
q15	q16	793	785	715	715
q17	1026	1026	979	979
q18	7226	5822	5530	5530
q19	1288	1279	989	989
q20	743	643	538	538
q21	6423	2973	2724	2724
q22	473	375	316	316
Total cold run time: 102460 ms
Total hot run time: 30071 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5444	5109	5005	5005
q2	314	371	220	220
q3	5025	5357	4697	4697
q4	2225	2255	1402	1402
q5	5180	4909	4848	4848
q6	260	191	131	131
q7	1942	1758	1551	1551
q8	2641	2287	2249	2249
q9	7806	7383	7352	7352
q10	4698	4632	4170	4170
q11	563	411	353	353
q12	753	742	523	523
q13	3067	3429	2705	2705
q14	283	293	257	257
q15	q16	686	702	629	629
q17	1291	1265	1274	1265
q18	7497	6859	6960	6859
q19	1114	1043	1050	1043
q20	2239	2241	1938	1938
q21	5447	4722	4611	4611
q22	537	477	419	419
Total cold run time: 59012 ms
Total hot run time: 52227 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 180394 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60ef7bba55674eeb5f09ed2c2e4ac742a9f770b1, data reload: false

query5	4309	626	496	496
query6	448	217	202	202
query7	4905	553	355	355
query8	329	188	179	179
query9	8758	4083	4082	4082
query10	469	346	301	301
query11	5804	2364	2126	2126
query12	158	105	101	101
query13	1260	624	437	437
query14	6274	5323	5011	5011
query14_1	4305	4262	4339	4262
query15	216	213	177	177
query16	986	454	471	454
query17	918	704	586	586
query18	2458	473	333	333
query19	202	181	147	147
query20	113	109	104	104
query21	236	154	133	133
query22	13667	13581	13644	13581
query23	17499	16636	16142	16142
query23_1	16263	16274	16342	16274
query24	7475	1767	1291	1291
query24_1	1323	1300	1291	1291
query25	526	437	384	384
query26	1334	359	223	223
query27	2596	620	371	371
query28	4447	1990	1998	1990
query29	1063	602	477	477
query30	338	266	222	222
query31	1120	1088	988	988
query32	112	67	72	67
query33	530	319	263	263
query34	1181	1142	669	669
query35	783	792	687	687
query36	1427	1366	1268	1268
query37	156	116	103	103
query38	1873	1697	1677	1677
query39	929	924	902	902
query39_1	883	880	891	880
query40	252	172	146	146
query41	71	73	71	71
query42	95	94	95	94
query43	322	339	286	286
query44	1450	792	779	779
query45	204	198	178	178
query46	1077	1247	774	774
query47	2413	2322	2221	2221
query48	406	386	289	289
query49	581	419	303	303
query50	1112	423	329	329
query51	10838	10818	10725	10725
query52	85	85	74	74
query53	260	280	201	201
query54	278	243	225	225
query55	76	69	67	67
query56	315	273	292	273
query57	1455	1411	1332	1332
query58	276	260	258	258
query59	1601	1665	1490	1490
query60	306	271	250	250
query61	152	154	145	145
query62	690	645	582	582
query63	246	209	199	199
query64	2830	1017	854	854
query65	4851	4745	4800	4745
query66	1819	516	382	382
query67	29421	29414	29245	29245
query68	3131	1611	968	968
query69	406	309	275	275
query70	1058	947	980	947
query71	360	318	310	310
query72	3085	2668	2437	2437
query73	825	816	401	401
query74	5106	4949	4761	4761
query75	2611	2578	2235	2235
query76	2318	1169	771	771
query77	359	381	301	301
query78	12281	12230	11810	11810
query79	1455	1175	720	720
query80	635	543	444	444
query81	463	325	286	286
query82	554	157	124	124
query83	395	322	297	297
query84	323	154	130	130
query85	920	596	534	534
query86	365	297	272	272
query87	1845	1835	1754	1754
query88	3692	2788	2774	2774
query89	457	400	355	355
query90	1927	201	189	189
query91	197	187	161	161
query92	67	62	59	59
query93	1568	1433	1007	1007
query94	533	355	302	302
query95	767	492	470	470
query96	1076	820	358	358
query97	2702	2665	2553	2553
query98	211	208	202	202
query99	1138	1171	1036	1036
Total cold run time: 264406 ms
Total hot run time: 180394 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60ef7bba55674eeb5f09ed2c2e4ac742a9f770b1, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.14	0.14
query4	1.61	0.17	0.14
query5	0.24	0.22	0.22
query6	1.24	1.10	1.10
query7	0.05	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.33	0.32
query10	0.61	0.57	0.59
query11	0.19	0.15	0.14
query12	0.17	0.15	0.15
query13	0.46	0.47	0.48
query14	1.02	1.02	0.99
query15	0.62	0.60	0.60
query16	0.31	0.35	0.33
query17	1.09	1.14	1.13
query18	0.23	0.22	0.21
query19	2.02	1.99	1.97
query20	0.02	0.01	0.01
query21	15.43	0.23	0.14
query22	4.80	0.06	0.06
query23	16.12	0.31	0.13
query24	2.92	0.43	0.33
query25	0.11	0.05	0.04
query26	0.74	0.21	0.16
query27	0.05	0.04	0.04
query28	3.50	0.96	0.53
query29	12.51	4.13	3.22
query30	0.27	0.16	0.16
query31	2.76	0.60	0.31
query32	3.22	0.59	0.48
query33	3.24	3.18	3.28
query34	15.55	4.22	3.55
query35	3.56	3.53	3.57
query36	0.55	0.43	0.42
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.20	0.17	0.16
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.55 s
Total hot run time: 25.26 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.06% (29264/40054)
Line Coverage 56.69% (316367/558047)
Region Coverage 53.14% (263145/495227)
Branch Coverage 54.23% (115569/213127)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/52) 🎉
Increment coverage report
Complete coverage report

@zhangstar333

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review completed for PR #65303.

I did not find a substantiated inline issue in the 39-file GitHub PR patch. The revert removes the active Lance/Rust reader wiring from FE format parsing, BE scanner/schema dispatch, CMake/build hooks, Rust FFI sources, BE tests, and the Lance regression assets. I also checked the current merge ref; the only remaining Lance-related matches are thrift compatibility fields (FORMAT_LANCE, TLanceFileDesc, deprecated enable_rust_lance_reader) and the removed-session-variable compatibility entry in VariableMgr, not an active execution path.

Critical checkpoints:

  • Goal and tests: the patch achieves the stated revert of the Lance reader feature and removes its dedicated tests/data. No new runtime behavior is introduced beyond rejecting format = "lance" again.
  • Scope/focus: the change is narrowly scoped to removing the feature's active implementation and tests; no user focus was provided.
  • Concurrency/lifecycle: no new concurrency, locks, long-lived state, or lifecycle ownership is introduced; deleted Rust reader resources and schema-reader paths are no longer reachable.
  • Config/compatibility: BUILD_RUST_READERS wiring is removed. The old enable_rust_lance_reader name remains only in the removed-session-variable compatibility list; thrift enum/descriptor remnants are compatibility surface and are not used by FE/BE dispatch.
  • Parallel paths: FE TVF format factory, BE scan reader dispatch, BE fetch-table-schema dispatch, build links, and test CMake paths were checked together.
  • Testing/validation: git diff --check passed on the real merge-base range and on the current merge-ref PR paths. Local build/tests were not run because this checkout lacks .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc. PR rollup shows COMPILE, BE UT, FE UT, Cloud UT, P0/non-concurrent/cloud/vault regression, performance, and coverage success; GitHub macOS BE UT fails before Doris build because the job uses JDK 25 while Doris requires JDK 17, and External Regression is currently pending/inaccessible from this runner.

Subagent conclusions:

  • optimizer-rewrite: no candidates; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.
  • tests-session-config: no candidates; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.
  • No candidate became an inline comment; no duplicates were merged.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.75% (29942/40054)
Line Coverage 58.83% (328279/558047)
Region Coverage 55.55% (275113/495227)
Branch Coverage 56.89% (121249/213127)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/643) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.06% (29264/40054)
Line Coverage 56.69% (316350/558047)
Region Coverage 53.12% (263070/495227)
Branch Coverage 54.22% (115562/213127)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/697) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.77% (29947/40054)
Line Coverage 58.83% (328302/558047)
Region Coverage 55.52% (274943/495227)
Branch Coverage 56.84% (121139/213127)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/805) 🎉
Increment coverage report
Complete coverage report

@Gabriel39
Gabriel39 merged commit 639a2d7 into apache:master Jul 10, 2026
34 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants