Skip to content

[improvement](gson) Serialize database tables as chunked JSON in image - #67332

Open
zghong wants to merge 2 commits into
apache:masterfrom
zghong:fix/stream-table-metadata-serialization
Open

[improvement](gson) Serialize database tables as chunked JSON in image#67332
zghong wants to merge 2 commits into
apache:masterfrom
zghong:fix/stream-table-metadata-serialization

Conversation

@zghong

@zghong zghong commented Aug 31, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #67331

Problem Summary:

Database.writeTables previously serialized each table with a single GsonUtils.GSON.toJson + Text.writeString call. For databases with very large tables this produces one oversized JSON string, hitting the single-string size limit and causing high peak memory during image write/read.

This adds GsonStreamUtils to stream tables as chunked JSON, and switches Database.writeTables/readTables to the streaming path. FE meta version is bumped to VERSION_141; images written by 141+ FE are not readable by older FE.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Add GsonStreamUtils to serialize/deserialize tables in chunks
instead of one `Text.writeString` call, avoiding the single-string
size limit and high memory use when a database has large tables.

`Database.writeTables`/`readTables` now use the streaming path.
Bumps FE meta version to 141; images written by 141+ are not
readable by older FE.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zghong

zghong commented Aug 31, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16898 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2ed82a9d27895dc735e2f1dbe82c733b11d6230, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18228	3026	2996	2996
q2	2098	252	228	228
q3	10248	893	516	516
q4	4673	254	205	205
q5	7667	559	387	387
q6	135	115	94	94
q7	516	519	384	384
q8	9234	988	983	983
q9	3464	2394	2358	2358
q10	6500	845	725	725
q11	396	201	180	180
q12	605	259	207	207
q13	18112	1524	1145	1145
q14	154	154	140	140
q15	q16	443	391	369	369
q17	1313	868	809	809
q18	3067	2241	2232	2232
q19	1119	814	769	769
q20	372	278	191	191
q21	4778	1752	1893	1752
q22	318	272	228	228
Total cold run time: 93440 ms
Total hot run time: 16898 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3350	3319	3290	3290
q2	497	397	352	352
q3	2222	2298	2133	2133
q4	1184	1165	892	892
q5	2200	2119	2125	2119
q6	163	117	88	88
q7	1014	947	882	882
q8	1584	1398	1419	1398
q9	3135	3140	3076	3076
q10	1850	1790	1612	1612
q11	353	264	248	248
q12	453	426	338	338
q13	1476	1514	1144	1144
q14	185	171	159	159
q15	q16	388	392	350	350
q17	3561	3351	3263	3263
q18	5016	4454	4701	4454
q19	824	775	929	775
q20	999	989	855	855
q21	3881	3107	3275	3107
q22	390	338	324	324
Total cold run time: 34725 ms
Total hot run time: 30859 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81981 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2ed82a9d27895dc735e2f1dbe82c733b11d6230, data reload: false

query5	4241	407	324	324
query6	386	134	122	122
query7	4978	406	226	226
query8	289	126	116	116
query9	8653	2875	2877	2875
query10	401	218	177	177
query11	5373	1034	908	908
query12	112	74	75	74
query13	1188	444	327	327
query14	6106	2185	2094	2094
query14_1	1994	1964	1920	1920
query15	171	116	111	111
query16	913	363	358	358
query17	802	484	400	400
query18	2326	344	245	245
query19	173	140	109	109
query20	72	68	71	68
query21	202	102	86	86
query22	5415	5315	5324	5315
query23	6837	6316	6165	6165
query23_1	5986	6067	6130	6067
query24	7307	1080	755	755
query24_1	775	788	761	761
query25	402	273	245	245
query26	1215	219	124	124
query27	2801	401	246	246
query28	4722	1493	1510	1493
query29	923	431	337	337
query30	246	161	128	128
query31	813	394	337	337
query32	126	74	74	74
query33	458	211	169	169
query34	1002	854	486	486
query35	390	400	339	339
query36	569	548	535	535
query37	119	76	70	70
query38	997	839	813	813
query39	492	472	486	472
query39_1	451	455	457	455
query40	200	97	76	76
query41	55	53	52	52
query42	79	73	77	73
query43	242	238	206	206
query44	1015	550	551	550
query45	110	105	101	101
query46	765	853	519	519
query47	759	771	710	710
query48	324	304	228	228
query49	534	232	192	192
query50	791	260	194	194
query51	7911	8009	7976	7976
query52	68	68	62	62
query53	194	203	164	164
query54	246	208	179	179
query55	79	59	60	59
query56	202	171	181	171
query57	712	651	674	651
query58	204	168	171	168
query59	1208	1219	1090	1090
query60	235	194	182	182
query61	161	141	198	141
query62	360	205	181	181
query63	165	139	144	139
query64	2663	689	612	612
query65	1612	1604	1576	1576
query66	1867	274	208	208
query67	9876	9733	9678	9678
query68	3045	1215	716	716
query69	344	233	181	181
query70	648	619	607	607
query71	258	169	167	167
query72	2366	1738	1530	1530
query73	631	556	323	323
query74	1999	1219	1137	1137
query75	1193	1091	947	947
query76	2399	716	531	531
query77	251	241	210	210
query78	4035	3718	3233	3233
query79	2711	822	595	595
query80	1585	329	276	276
query81	518	155	131	131
query82	612	126	93	93
query83	283	208	198	198
query84	292	115	87	87
query85	828	375	289	289
query86	470	180	172	172
query87	1024	955	905	905
query88	3104	2078	2098	2078
query89	291	200	173	173
query90	2171	132	131	131
query91	131	119	99	99
query92	93	71	66	66
query93	2109	1043	674	674
query94	619	263	216	216
query95	516	326	221	221
query96	757	527	275	275
query97	1093	1099	1017	1017
query98	174	134	133	133
query99	420	339	313	313
Total cold run time: 179454 ms
Total hot run time: 81981 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2ed82a9d27895dc735e2f1dbe82c733b11d6230, data reload: false

query1	0.01	0.00	0.01
query2	0.07	0.04	0.03
query3	0.25	0.11	0.11
query4	1.60	0.11	0.09
query5	0.18	0.16	0.17
query6	1.24	0.74	0.70
query7	0.03	0.00	0.01
query8	0.04	0.02	0.03
query9	0.29	0.21	0.23
query10	0.38	0.35	0.35
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.31	0.31	0.31
query14	0.44	0.45	0.46
query15	0.37	0.36	0.35
query16	0.22	0.24	0.22
query17	0.67	0.73	0.62
query18	0.17	0.17	0.17
query19	1.15	1.13	1.16
query20	0.01	0.02	0.02
query21	15.49	0.17	0.13
query22	5.04	0.04	0.05
query23	16.17	0.25	0.10
query24	3.00	0.32	0.24
query25	0.11	0.04	0.04
query26	0.71	0.16	0.12
query27	0.04	0.03	0.04
query28	3.63	0.56	0.27
query29	12.48	3.14	2.55
query30	0.25	0.12	0.12
query31	2.75	0.36	0.17
query32	3.53	0.31	0.23
query33	1.37	1.41	1.41
query34	15.38	2.16	1.77
query35	1.76	1.76	1.74
query36	0.46	0.29	0.29
query37	0.08	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.11	0.09	0.08
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 90.31 s
Total hot run time: 14.66 s

@zghong
zghong requested a review from morningman as a code owner September 2, 2026 16:42
@zghong

zghong commented Sep 2, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 56.31% (58/103) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] FE checkpoint fails with OutOfMemoryError when serializing a table with very large metadata

2 participants