Skip to content

[fix](compaction) Keep latest 10 versions on new tablet during picking compaction input rowsets when schema change#63112

Open
Hastyshell wants to merge 1 commit intoapache:masterfrom
Hastyshell:fix-keep-latest-versions-alter-compaction
Open

[fix](compaction) Keep latest 10 versions on new tablet during picking compaction input rowsets when schema change#63112
Hastyshell wants to merge 1 commit intoapache:masterfrom
Hastyshell:fix-keep-latest-versions-alter-compaction

Conversation

@Hastyshell
Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: The cumulative compaction policy for NOTREADY tablets is intended to keep the latest 10 versions from being compacted during alter, but the condition skipped older versions instead. This change makes the condition match the documented behavior.

Release note

None

Check List (For Author)

  • Test: No need to test (condition-only fix; verified with git diff --check)
  • Behavior changed: Yes. NOTREADY tablet compaction now preserves the latest versions as documented.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: The NOTREADY tablet cumulative compaction policy intended to keep the latest 10 versions from being compacted during alter, but the condition skipped older versions instead. This change makes the condition match the documented behavior.

### Release note

None

### Check List (For Author)

- Test: No need to test (condition-only fix; verified with git diff --check)
- Behavior changed: Yes (NOTREADY tablet compaction now preserves the latest versions as documented)
- Does this need documentation: No
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Hastyshell
Copy link
Copy Markdown
Collaborator Author

run buildall

@Hastyshell Hastyshell changed the title [fix](be) Keep latest versions during alter compaction [fix](compaction) Keep latest versions during alter compaction May 9, 2026
@Hastyshell Hastyshell changed the title [fix](compaction) Keep latest versions during alter compaction [fix](compaction) Keep latest 10 versions on new tablet during picking compaction input rowsets when schema change May 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 29557 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6236e22b6353fdbd253969c8e38512b059b2c1aa, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17762	3893	3828	3828
q2	q3	10740	893	631	631
q4	4669	461	345	345
q5	7459	1330	1146	1146
q6	190	173	139	139
q7	906	933	752	752
q8	9323	1384	1322	1322
q9	5543	5315	5330	5315
q10	6246	2073	1784	1784
q11	460	266	256	256
q12	640	412	296	296
q13	18093	3263	2728	2728
q14	288	280	270	270
q15	q16	864	862	788	788
q17	984	1095	661	661
q18	6471	5697	5651	5651
q19	1137	1283	1096	1096
q20	497	401	260	260
q21	4367	2462	1927	1927
q22	470	426	362	362
Total cold run time: 97109 ms
Total hot run time: 29557 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4772	4705	4776	4705
q2	q3	4659	4821	4210	4210
q4	2120	2203	1423	1423
q5	4997	5012	5324	5012
q6	196	178	132	132
q7	2064	1772	1632	1632
q8	3371	3128	3117	3117
q9	8397	8390	8432	8390
q10	4483	4478	4234	4234
q11	594	413	394	394
q12	700	755	534	534
q13	3233	3557	2961	2961
q14	437	328	284	284
q15	q16	802	781	721	721
q17	1318	1281	1260	1260
q18	7967	7205	7136	7136
q19	1164	1178	1160	1160
q20	2204	2210	1954	1954
q21	6091	5395	4822	4822
q22	556	484	406	406
Total cold run time: 60125 ms
Total hot run time: 54487 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 172523 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6236e22b6353fdbd253969c8e38512b059b2c1aa, data reload: false

query5	4321	655	517	517
query6	360	222	205	205
query7	4237	582	312	312
query8	317	233	212	212
query9	8855	3986	3973	3973
query10	440	340	304	304
query11	5931	2427	2226	2226
query12	189	128	132	128
query13	1279	586	410	410
query14	6784	5338	5039	5039
query14_1	4366	4364	4318	4318
query15	212	207	193	193
query16	1035	463	429	429
query17	1154	766	641	641
query18	2795	490	361	361
query19	224	213	184	184
query20	144	145	133	133
query21	220	143	120	120
query22	13562	14157	14492	14157
query23	17203	16483	16220	16220
query23_1	16282	16292	16242	16242
query24	7457	1731	1336	1336
query24_1	1357	1348	1337	1337
query25	624	483	422	422
query26	1285	324	169	169
query27	2753	595	342	342
query28	4395	1970	1948	1948
query29	1005	633	507	507
query30	308	232	197	197
query31	1115	1055	956	956
query32	88	73	68	68
query33	532	347	287	287
query34	1214	1140	639	639
query35	770	795	674	674
query36	1331	1299	1170	1170
query37	144	94	88	88
query38	3195	3135	3034	3034
query39	921	907	898	898
query39_1	875	883	874	874
query40	236	167	148	148
query41	63	59	59	59
query42	109	105	107	105
query43	323	323	279	279
query44	
query45	206	200	195	195
query46	1042	1215	749	749
query47	2305	2268	2252	2252
query48	409	416	304	304
query49	631	525	422	422
query50	702	281	218	218
query51	4286	4230	4205	4205
query52	105	105	96	96
query53	257	280	199	199
query54	312	270	259	259
query55	99	97	84	84
query56	295	314	303	303
query57	1402	1396	1327	1327
query58	301	265	273	265
query59	1576	1652	1407	1407
query60	341	341	323	323
query61	157	158	160	158
query62	672	617	563	563
query63	242	201	206	201
query64	2353	825	685	685
query65	
query66	1701	516	389	389
query67	29932	29903	29852	29852
query68	
query69	459	347	305	305
query70	1050	1020	978	978
query71	306	272	264	264
query72	2995	2718	2405	2405
query73	844	802	440	440
query74	5083	4922	4752	4752
query75	2779	2640	2324	2324
query76	2285	1160	760	760
query77	419	415	352	352
query78	13078	13036	12497	12497
query79	1504	1027	761	761
query80	703	583	499	499
query81	464	285	241	241
query82	1353	169	127	127
query83	372	284	256	256
query84	266	146	118	118
query85	867	535	458	458
query86	405	370	326	326
query87	3405	3369	3238	3238
query88	3555	2690	2652	2652
query89	446	384	341	341
query90	1845	183	180	180
query91	189	188	166	166
query92	81	77	72	72
query93	964	966	546	546
query94	546	321	305	305
query95	647	387	353	353
query96	1026	801	378	378
query97	2703	2691	2613	2613
query98	243	229	230	229
query99	1088	1126	975	975
Total cold run time: 254549 ms
Total hot run time: 172523 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 29636 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6236e22b6353fdbd253969c8e38512b059b2c1aa, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17633	3862	3880	3862
q2	q3	10706	876	615	615
q4	4662	463	343	343
q5	7438	1343	1133	1133
q6	192	174	138	138
q7	929	936	750	750
q8	9310	1443	1284	1284
q9	5655	5379	5341	5341
q10	6277	2130	1788	1788
q11	472	263	263	263
q12	634	424	296	296
q13	18084	3443	2702	2702
q14	292	283	272	272
q15	q16	905	865	787	787
q17	933	970	741	741
q18	6566	5748	5698	5698
q19	1209	1241	1048	1048
q20	501	394	266	266
q21	4692	2420	1985	1985
q22	485	404	324	324
Total cold run time: 97575 ms
Total hot run time: 29636 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4816	4614	4920	4614
q2	q3	4673	4788	4237	4237
q4	2130	2184	1378	1378
q5	5045	4969	5237	4969
q6	192	174	141	141
q7	2078	1820	1617	1617
q8	3360	3100	3148	3100
q9	8400	8563	8565	8563
q10	4554	4554	4272	4272
q11	589	429	394	394
q12	710	757	516	516
q13	3227	3653	2925	2925
q14	317	312	282	282
q15	q16	779	954	712	712
q17	1337	1306	1272	1272
q18	8010	7160	7150	7150
q19	1184	1182	1179	1179
q20	2252	2200	2010	2010
q21	6303	5527	4959	4959
q22	547	498	446	446
Total cold run time: 60503 ms
Total hot run time: 54736 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 172039 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6236e22b6353fdbd253969c8e38512b059b2c1aa, data reload: false

query5	4346	643	511	511
query6	327	213	212	212
query7	4249	582	292	292
query8	322	234	215	215
query9	8834	4107	4084	4084
query10	456	362	310	310
query11	5799	2409	2253	2253
query12	189	136	125	125
query13	1306	594	451	451
query14	6707	5420	5063	5063
query14_1	4369	4395	4364	4364
query15	215	208	182	182
query16	1053	499	472	472
query17	1167	784	654	654
query18	2580	487	373	373
query19	221	218	173	173
query20	143	135	138	135
query21	230	139	115	115
query22	13700	14224	14451	14224
query23	17462	16655	16197	16197
query23_1	16335	16241	16188	16188
query24	7483	1749	1357	1357
query24_1	1408	1364	1366	1364
query25	556	491	421	421
query26	1288	304	167	167
query27	2721	587	326	326
query28	4431	1959	1960	1959
query29	1005	612	511	511
query30	302	233	200	200
query31	1128	1060	960	960
query32	84	72	71	71
query33	522	340	291	291
query34	1184	1110	629	629
query35	762	796	666	666
query36	1357	1356	1183	1183
query37	148	98	85	85
query38	3205	3141	3039	3039
query39	920	925	887	887
query39_1	861	888	878	878
query40	229	162	136	136
query41	62	60	58	58
query42	107	107	105	105
query43	327	333	288	288
query44	
query45	209	212	201	201
query46	1094	1194	708	708
query47	2332	2278	2248	2248
query48	401	423	294	294
query49	666	528	428	428
query50	717	285	231	231
query51	4363	4247	4217	4217
query52	106	105	94	94
query53	257	279	208	208
query54	310	278	247	247
query55	94	88	93	88
query56	304	290	301	290
query57	1427	1402	1318	1318
query58	288	270	288	270
query59	1613	1637	1422	1422
query60	353	336	330	330
query61	164	157	160	157
query62	671	611	560	560
query63	265	205	208	205
query64	2393	843	683	683
query65	
query66	1736	498	390	390
query67	29999	29402	29854	29402
query68	
query69	467	344	297	297
query70	1017	985	1014	985
query71	315	287	273	273
query72	2934	2723	2454	2454
query73	876	753	445	445
query74	5060	4915	4713	4713
query75	2792	2658	2337	2337
query76	2293	1130	769	769
query77	409	420	347	347
query78	13045	12959	12252	12252
query79	1483	1062	768	768
query80	674	596	489	489
query81	450	293	240	240
query82	1340	154	124	124
query83	352	282	262	262
query84	282	136	112	112
query85	864	512	433	433
query86	402	336	347	336
query87	3414	3345	3207	3207
query88	3592	2666	2657	2657
query89	446	377	333	333
query90	1901	186	179	179
query91	181	175	140	140
query92	82	77	73	73
query93	972	961	563	563
query94	538	366	307	307
query95	671	465	346	346
query96	1107	795	347	347
query97	2682	2697	2611	2611
query98	238	232	241	232
query99	1072	1099	983	983
Total cold run time: 254796 ms
Total hot run time: 172039 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (1/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.92% (27147/37747)
Line Coverage 55.20% (288516/522707)
Region Coverage 52.27% (239261/457756)
Branch Coverage 53.59% (103153/192502)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants