Skip to content

[refactor](nereids)Remove defer materialize#62917

Merged
englefly merged 7 commits intoapache:masterfrom
englefly:remove-defer-mat
May 9, 2026
Merged

[refactor](nereids)Remove defer materialize#62917
englefly merged 7 commits intoapache:masterfrom
englefly:remove-defer-mat

Conversation

@englefly
Copy link
Copy Markdown
Contributor

@englefly englefly commented Apr 29, 2026

What problem does this PR solve?

defer materialization is replaced by topn-lazy-materialization

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 6, 2026

run buildall

@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 6, 2026

/review

@englefly englefly changed the title Remove defer mat [refactor](nereids)Remove defer materialize May 6, 2026
@englefly englefly marked this pull request as ready for review May 6, 2026 02:54
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a blocking compatibility regression in the session-variable migration. The PR removes the old defer-materialize TopN rule and documents enable_two_phase_read_opt as replaced by topn_lazy_materialization_threshold, but the old variable is still accepted and still used by existing code/tests to disable two-phase read. After this change, setting enable_two_phase_read_opt=false no longer prevents PlanPostProcessors from adding LazyMaterializeTopN because that path only checks topn_lazy_materialization_threshold > 0.

Critical checkpoint conclusions: goal is clear, but the migration does not fully preserve the old disable contract; change is mostly focused but misses compatibility wiring; no new concurrency/lifecycle/static-init concerns were found; this is a session configuration change and the old config no longer affects the replacement path; no storage-format or FE/BE protocol compatibility issue was found; the parallel old/new TopN lazy-materialization paths were considered and the removed path is replaced by the post-processor path; test coverage regresses because the old two-phase-read regression tests are deleted and no replacement covers enable_two_phase_read_opt=false; no result-file correctness issue found in the modified/deleted test files; no additional observability requirement; no transaction/persistence/data-write impact found; no new FE-BE variables added; performance impact is not the concern here, correctness/compatibility is.

User focus: no additional user-provided review focus was specified.

Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java Outdated
@englefly englefly force-pushed the remove-defer-mat branch from 08fb5ed to f474332 Compare May 7, 2026 02:43
@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 7, 2026

run buildall

@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 7, 2026

run external

@englefly englefly dismissed github-actions[bot]’s stale review May 8, 2026 01:23

the old session var is deprecated.

Comment thread fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java Outdated
@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 8, 2026

run buildall

3 similar comments
@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 8, 2026

run buildall

@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 8, 2026

run buildall

@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 8, 2026

run buildall

englefly and others added 7 commits May 8, 2026 16:13
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Remove the legacy DeferMaterializeTopN rewrite and implementation path while keeping enable_two_phase_read_opt itself.

### Release note

None

### Check List (For Author)

- Test: No need to test (waiting for manual compile/test confirmation)
- Behavior changed: Yes (the legacy DeferMaterializeTopN path and its dedicated tests are removed)
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 8, 2026

run buildall

@englefly englefly force-pushed the remove-defer-mat branch from cfb9634 to 484eceb Compare May 8, 2026 23:45
@englefly
Copy link
Copy Markdown
Contributor Author

englefly commented May 8, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.07% (1844/2362)
Line Coverage 64.76% (33005/50963)
Region Coverage 65.26% (16376/25093)
Branch Coverage 55.79% (8736/15658)

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 29401 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 484eceb0b7eb8cf4927dc7ab649bc8b52922ca63, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	16783	3889	3807	3807
q2	q3	10379	876	606	606
q4	4649	451	345	345
q5	7565	1333	1140	1140
q6	190	176	142	142
q7	918	932	758	758
q8	9526	1412	1290	1290
q9	5545	5386	5292	5292
q10	6255	2077	1819	1819
q11	461	269	251	251
q12	629	424	290	290
q13	18105	3839	2741	2741
q14	288	283	259	259
q15	q16	896	872	783	783
q17	880	1022	807	807
q18	6512	5710	5520	5520
q19	1144	1160	1100	1100
q20	509	405	261	261
q21	4509	2299	1880	1880
q22	425	354	310	310
Total cold run time: 96168 ms
Total hot run time: 29401 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4140	4081	4056	4056
q2	q3	4596	4728	4166	4166
q4	2065	2155	1386	1386
q5	4951	4953	5247	4953
q6	190	167	129	129
q7	2010	1764	1585	1585
q8	3700	3216	3184	3184
q9	8616	8384	8376	8376
q10	4486	4508	4218	4218
q11	610	444	417	417
q12	706	755	525	525
q13	3335	3505	2950	2950
q14	306	317	278	278
q15	q16	767	771	693	693
q17	1358	1317	1387	1317
q18	7883	7293	7042	7042
q19	1150	1180	1114	1114
q20	2209	2241	1989	1989
q21	6138	5363	4902	4902
q22	594	509	435	435
Total cold run time: 59810 ms
Total hot run time: 53715 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 169828 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 484eceb0b7eb8cf4927dc7ab649bc8b52922ca63, data reload: false

query5	4308	644	508	508
query6	320	219	207	207
query7	4254	565	291	291
query8	330	229	209	209
query9	8819	3962	3974	3962
query10	463	343	294	294
query11	6043	2393	2152	2152
query12	187	129	126	126
query13	1273	588	405	405
query14	6730	5305	5008	5008
query14_1	4334	4360	4307	4307
query15	207	204	191	191
query16	992	458	437	437
query17	1314	752	639	639
query18	2723	478	346	346
query19	227	198	167	167
query20	141	131	131	131
query21	214	139	116	116
query22	13623	13562	13356	13356
query23	17159	16416	16516	16416
query23_1	16310	16229	16250	16229
query24	7559	1827	1392	1392
query24_1	1433	1397	1407	1397
query25	601	522	450	450
query26	1210	339	175	175
query27	3141	596	348	348
query28	4396	1949	1974	1949
query29	1054	675	524	524
query30	311	233	193	193
query31	1117	1070	934	934
query32	82	74	68	68
query33	528	342	285	285
query34	1179	1103	629	629
query35	760	783	690	690
query36	1361	1319	1122	1122
query37	146	101	88	88
query38	3211	3125	3090	3090
query39	918	918	906	906
query39_1	886	882	875	875
query40	230	152	128	128
query41	64	58	60	58
query42	110	104	107	104
query43	315	320	296	296
query44	
query45	209	197	195	195
query46	1092	1187	708	708
query47	2310	2213	2197	2197
query48	401	399	286	286
query49	620	520	425	425
query50	692	288	215	215
query51	4257	4264	4161	4161
query52	102	104	96	96
query53	243	277	201	201
query54	302	268	255	255
query55	94	89	85	85
query56	302	301	298	298
query57	1402	1384	1254	1254
query58	297	269	280	269
query59	1524	1599	1385	1385
query60	338	328	315	315
query61	159	158	159	158
query62	680	618	559	559
query63	245	203	201	201
query64	2338	901	679	679
query65	
query66	1667	509	376	376
query67	30080	29324	29762	29324
query68	
query69	463	331	305	305
query70	995	970	982	970
query71	319	271	266	266
query72	2885	2700	2540	2540
query73	835	788	427	427
query74	5112	4919	4745	4745
query75	2789	2659	2375	2375
query76	2104	1111	761	761
query77	429	431	354	354
query78	12859	12955	12306	12306
query79	1432	1033	755	755
query80	715	619	499	499
query81	451	277	240	240
query82	1238	162	126	126
query83	357	278	252	252
query84	261	146	119	119
query85	960	590	519	519
query86	397	338	325	325
query87	3448	3357	3250	3250
query88	3538	2666	2644	2644
query89	434	395	338	338
query90	1880	179	186	179
query91	194	188	157	157
query92	86	78	74	74
query93	952	952	559	559
query94	530	329	343	329
query95	673	382	353	353
query96	997	770	347	347
query97	2672	2663	2533	2533
query98	251	230	226	226
query99	1078	1161	1150	1150
Total cold run time: 254177 ms
Total hot run time: 169828 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 81.82% (9/11) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.76% (27769/37646)
Line Coverage 57.66% (300810/521713)
Region Coverage 54.90% (250783/456787)
Branch Coverage 56.44% (108489/192207)

@englefly englefly merged commit 559b000 into apache:master May 9, 2026
28 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants