[improvement](executor) unify current query runtime statistics and expose task progress#60567
Merged
yiguolei merged 50 commits intoapache:masterfrom May 10, 2026
Merged
[improvement](executor) unify current query runtime statistics and expose task progress#60567yiguolei merged 50 commits intoapache:masterfrom
yiguolei merged 50 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 30119 ms |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
ClickBench: Total hot run time: 28.17 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28909 ms |
TPC-DS: Total hot run time: 184883 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
xuchenhao
force-pushed
the
query_progress
branch
from
7c9c9f3 to
24ecf73
Compare
Contributor
|
run buildall |
TPC-H: Total hot run time: 28719 ms |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-DS: Total hot run time: 184913 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Contributor
Author
|
run buildall |
yiguolei
reviewed
May 6, 2026
…ontext and unstable regression test This commit addresses the following review suggestions: - Removes redundant _total_task_num and _finished_task_num from QueryContext since they are already stored in QueryTaskController. - Removes the cached raw pointer _query_task_controller to avoid potential core dumps caused by accessing a destructed object. - Deletes the unstable regression test test_query_progress.groovy which relied on Thread.sleep and was prone to sporadic failures.
Contributor
Author
|
run buildall |
1 similar comment
Contributor
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 30644 ms |
Contributor
TPC-DS: Total hot run time: 170269 ms |
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I reviewed the current PR head after reading the Doris code-review skill, the changed module guides, and all existing inline review threads. I am not adding duplicate inline comments: the blocking issues I still see are already covered by existing threads, including the public proc subtree removal for /current_queries/<query_id> and /fragments, the /current_query_stmts schema/semantics compatibility break, the /rest/v2/manager/query/current_queries compatibility break, and the incomplete aggregation/DTO coverage for the new TQueryStatistics fields in the realtime statistics API.
Checkpoint conclusions:
- Goal/test: The goal appears to be adding current-query runtime/progress statistics. The implementation adds BE counters and FE display/unit tests, but the user-visible compatibility regressions and API inconsistency mean the goal is not safely accomplished yet.
- Scope/focus: The change is broader than necessary because it deletes/repoints existing proc nodes and REST behavior instead of only extending the intended statistics surface.
- Concurrency: BE task counters are atomic and the FE snapshot path avoids the previously discussed synchronous traversal race, but the affected public read paths still need compatibility-safe behavior.
- Lifecycle: Moving task counters into
QueryTaskControlleraddresses the final-report lifetime concern that was previously raised. - Config/compatibility: No new config was added. There are incompatible proc/REST contract changes that remain blocking and are already covered by existing threads.
- Parallel paths: Query-stat surfaces are not all updated consistently; the realtime statistics endpoint still omits the newly added fields, as already noted in the existing thread.
- Tests: Added unit tests cover formatting and merge logic, and the unstable regression test was removed, but there is no replacement end-to-end coverage proving compatibility of existing proc/REST contracts.
- Observability/data correctness/storage: No storage/transaction correctness concerns found in this PR; observability is the feature itself, but API consistency remains incomplete.
User focus: no additional user-provided review focus was specified, and I found no extra focus-specific issue beyond the existing blockers.
Contributor
|
skip check_coverage |
yiguolei
approved these changes
May 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Summary
BE -> FEreporting pipeline, replacing the previous ad-hocRuntimeProfiletraversal path, and enrichescurrent_querieswith task-level progress plus broader resource metrics.What It Solves
QeProcessorImplnow reads aggregatedTQueryStatisticsfromWorkloadRuntimeStatusMgrinstead of relying on the legacyCurrentQueryInfoProviderpath.process_rows,total_tasks_num, andfinished_tasks_num, and exposes computedProgress.current_queriesnow includes richer scan/cpu/memory/shuffle/spill/cache counters./current_queriesand/current_query_stmtsnow share the same statistics view; legacy per-query/per-fragment proc drill-down implementation is removed.Implementation Details
TQueryStatisticswithprocess_rows,finished_tasks_num, andtotal_tasks_num.process_rowsin the execution path.total_tasks_numat pipeline task graph initialization and incrementsfinished_tasks_numin real time when tasks close.QueryTaskControllerso counters remain available even afterQueryContextteardown.ResourceContext::to_thrift_query_statistics.WorkloadRuntimeStatusMgrmerges additional fields (including task progress) and refines timeout cleanup: remove query stats only when they are timed out and the query no longer exists in FE.QueryStatisticsItemnow carriesTQueryStatisticsas the unified data carrier for proc/REST.CurrentQueryStatisticsProcDiradds expanded columns and computesProgress./rest/v2/manager/query/current_queriesinQueryProfileActionnow serves the same unified stats view.CurrentQueryInfoProvider,CurrentQuerySqlProcDir,CurrentQueryFragmentProcNode, andCurrentQueryStatementsProcNode.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)