Skip to content

[Bug](mtmv) fix MV refresh stuck due to precision-lossy hot value collection - #67354

Open
xy720 wants to merge 1 commit into
apache:masterfrom
xy720:hot-values-scale
Open

[Bug](mtmv) fix MV refresh stuck due to precision-lossy hot value collection#67354
xy720 wants to merge 1 commit into
apache:masterfrom
xy720:hot-values-scale

Conversation

@xy720

@xy720 xy720 commented Aug 31, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Root cause chain:
1. Analyze sample round hot_values with ROUND(count / rowCount, 2)), a value with true ratio 0.1858% is rounded as 0.00.
2. FilterEstimation.estimateColumnEqualToConstant unconditionally overwrites 1/ndv with the 0.00 hot value ratio.
3. Filter output rowCount collapses to 0.
4. InitJoinOrder's left > 0 guard rejects the swap.
5. JoinEstimation floors row count back to 1 via Math.max(1, ...).
6. Plan generated wrong join order, the huge side becomes hash build side and the mv refresh failed.

Triggered this bug by enabling force_sample_analyze=true globally in our customers cluster,
which manual ANALYZE through the sample path that collects hot values.

Fix (three-in-one, all minimal):

  • Analyze: ROUND scale 2 -> 4 in both LINEAR and DUJ1 templates.
  • FilterEstimation: guard against hot_value ratio == 0.0 rounding artifact, fall back
    to the uniform 1/ndv estimate.
  • InitJoinOrder: relax left > 0 to left >= 0; the downstream Math.max(1, ..)
    in JoinEstimation already prevents any loop from truly-negative estimates.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xy720

xy720 commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16854 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d40e75503df9d24917bad75c91fc42689a32279, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17585	3040	3049	3040
q2	2111	256	242	242
q3	10213	848	507	507
q4	4671	262	209	209
q5	7653	578	386	386
q6	133	117	94	94
q7	520	499	385	385
q8	9247	921	927	921
q9	3445	2420	2390	2390
q10	6491	842	725	725
q11	409	203	184	184
q12	612	259	195	195
q13	18133	1530	1172	1172
q14	160	154	143	143
q15	q16	435	398	371	371
q17	1346	893	845	845
q18	3049	2316	2270	2270
q19	1109	935	781	781
q20	379	274	197	197
q21	5240	1567	1803	1567
q22	326	265	230	230
Total cold run time: 93267 ms
Total hot run time: 16854 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3408	3347	3288	3288
q2	528	408	369	369
q3	2252	2350	2226	2226
q4	1185	1175	898	898
q5	2186	2146	2125	2125
q6	170	114	88	88
q7	1019	892	862	862
q8	1588	1403	1406	1403
q9	3195	3173	3151	3151
q10	1867	1819	1651	1651
q11	362	275	261	261
q12	458	426	339	339
q13	1470	1569	1182	1182
q14	168	170	159	159
q15	q16	398	402	358	358
q17	3638	3335	3270	3270
q18	4951	4535	4786	4535
q19	890	909	872	872
q20	1027	968	816	816
q21	3819	3127	3216	3127
q22	393	349	320	320
Total cold run time: 34972 ms
Total hot run time: 31300 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82431 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d40e75503df9d24917bad75c91fc42689a32279, data reload: false

query5	4291	421	339	339
query6	398	138	123	123
query7	4927	437	228	228
query8	285	130	117	117
query9	8696	2916	2931	2916
query10	379	219	192	192
query11	5380	1046	914	914
query12	122	70	68	68
query13	1191	453	339	339
query14	6070	2228	2099	2099
query14_1	1988	1973	1964	1964
query15	171	116	110	110
query16	911	383	357	357
query17	803	451	364	364
query18	2337	335	242	242
query19	168	138	109	109
query20	74	73	78	73
query21	207	103	88	88
query22	5528	5289	5460	5289
query23	6676	6205	6224	6205
query23_1	6296	6011	6178	6011
query24	7310	1103	792	792
query24_1	786	775	790	775
query25	440	308	268	268
query26	1222	221	133	133
query27	2792	438	262	262
query28	4685	1535	1500	1500
query29	940	451	366	366
query30	257	159	130	130
query31	816	408	345	345
query32	131	79	75	75
query33	470	224	179	179
query34	986	846	463	463
query35	400	427	359	359
query36	588	596	539	539
query37	124	83	73	73
query38	1006	871	850	850
query39	500	488	472	472
query39_1	470	460	462	460
query40	205	93	78	78
query41	59	60	56	56
query42	73	72	74	72
query43	244	247	215	215
query44	1019	581	557	557
query45	111	102	101	101
query46	764	839	533	533
query47	755	749	706	706
query48	313	309	234	234
query49	548	219	176	176
query50	738	266	197	197
query51	8193	8105	8094	8094
query52	79	76	67	67
query53	245	206	147	147
query54	221	169	154	154
query55	74	69	60	60
query56	213	177	160	160
query57	730	671	635	635
query58	198	176	174	174
query59	1240	1252	1104	1104
query60	232	188	167	167
query61	125	118	122	118
query62	358	205	182	182
query63	171	144	140	140
query64	2785	692	602	602
query65	1629	1622	1611	1611
query66	1810	277	245	245
query67	10173	9749	9520	9520
query68	2749	1108	762	762
query69	346	229	188	188
query70	670	633	614	614
query71	254	168	167	167
query72	2313	1736	1576	1576
query73	631	574	350	350
query74	1562	1231	1150	1150
query75	1193	1108	969	969
query76	2296	725	537	537
query77	254	272	198	198
query78	3865	3650	3124	3124
query79	2917	882	607	607
query80	1573	346	279	279
query81	505	159	135	135
query82	629	122	96	96
query83	284	209	193	193
query84	297	109	88	88
query85	803	354	297	297
query86	399	183	167	167
query87	1030	988	908	908
query88	2860	2127	2114	2114
query89	283	200	173	173
query90	1939	133	125	125
query91	133	128	102	102
query92	81	69	63	63
query93	1557	1112	725	725
query94	641	266	220	220
query95	518	261	227	227
query96	794	603	288	288
query97	1099	1079	1035	1035
query98	177	131	152	131
query99	416	344	313	313
Total cold run time: 178621 ms
Total hot run time: 82431 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d40e75503df9d24917bad75c91fc42689a32279, data reload: false

query1	0.01	0.00	0.00
query2	0.07	0.03	0.03
query3	0.25	0.11	0.12
query4	1.60	0.10	0.10
query5	0.17	0.16	0.16
query6	1.24	0.72	0.68
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.22	0.22
query10	0.34	0.34	0.34
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.31	0.33	0.32
query14	0.46	0.46	0.45
query15	0.37	0.34	0.35
query16	0.22	0.22	0.23
query17	0.68	0.69	0.72
query18	0.19	0.15	0.16
query19	1.16	1.21	1.22
query20	0.02	0.01	0.01
query21	15.43	0.15	0.12
query22	5.06	0.04	0.05
query23	16.14	0.24	0.10
query24	3.06	0.31	0.26
query25	0.12	0.04	0.03
query26	0.75	0.17	0.13
query27	0.04	0.03	0.02
query28	3.61	0.59	0.25
query29	12.47	3.24	2.60
query30	0.26	0.12	0.13
query31	2.75	0.36	0.18
query32	3.52	0.32	0.24
query33	1.35	1.39	1.43
query34	15.37	2.22	1.79
query35	1.77	1.75	1.74
query36	0.46	0.29	0.30
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.07	0.07
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.32 s
Total hot run time: 14.83 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 75.00% (3/4) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title [Bug](nereids) fix MV refresh stuck due to precision-lossy hot value collection [Bug](mtmv) fix MV refresh stuck due to precision-lossy hot value collection Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants