Enhance task planner/executor to support task groups and ordering #5979
Description
Currently, only one very rudimentary ordering rule is supported by the planner/executor: Exclusive tasks run last
This is rather a low complexity implementation to support exclusive tasks. While there is no technical reason, they need to run last, it's important to run them non concurrently to other tasks. The easiest implementation just evaluates all dependencies first (and in parallel) and then evaluate all exclusive task at the end without concurrency.
The downside of this simple strategy is, that non-exclusive tasks can't depend on exclusive tasks.
We already hit this limitation in out BSP integration, where the BSP process always starts from an anonymous task, which means, we can't run exclusive tasks in BSP in general.
I propose to introduce task groups. Dependencies between task groups result from their contained tasks. When planning the execution and we detect exclusive tasks, we put them in a second task group, which needs to run after the first. We mark this groups as exclusive and ran it with concurrency level of one. This is basically what we currently do.
If we detect non-exclusive tasks that depend on the exclusive task, we put them in a new task group that runs after the exclusive group. This task group can be have a higher concurrency level, since no exclusive task are to be run.
The task grouping should not be an implementation detail of the executor, but an explicit result of the planning.
As a second feature, we could allow explicit grouped requests when planning, so a user can enforce some execution ordering. The main use-case for explicit ordering are destructive tasks that affect the Mill caches and can't run in parallel with other task, e.g. clean. Assuming we introduce a new syntax to request a new task group with ++, a user could clean the cache and run all compilations only after the clean command finished with mill clean ++ __.compile.