How to run multiple stages of a single dvc pipeline?

[Asaad-Pak]

I am using dvc to automate one of my pipeline. In my actually pipeline i have some modules that are running in parallel but when i declared them as stages and tried to run multiple stages in parallel, it ran sequentially one by one which i don't want because it increase latency of my pipeline. So my question is how we can run parallel stages in a single pipeline? I have searched in existing issues I have found about how to run run whole experiment or pipeline in parallel for example using queue or workers with "dvc exp run" command to run two experiments in parallel but these are whole pipelines whereas i am concerned about running two or multiple stages parallel in one single pipeline. Kindly guide regarding this please.

[skshetry]

There is an open issue for #755, and a contributor PR: #10922.

But that won't be parallelizable for a single dvc pipeline, if by pipeline, you meant a pipeline in DAG. For example, this cannot be made parallelizable, as one depends on the other.

flowchart TD
        node1["data/data.xml.dvc"]
        node2["prepare"]
        node3["train"]
        node4["evaluate"]
        node1-->node2
        node2-->node3
        node3-->node4 

Loading

If you meant a single dvc.yaml, that may be parallelizable.

[Asaad-Pak]

Ok let me explain from the dag example you gave above. What i meant is that lets say i have two stages after evaluate stage visualize_1 and visualize_2 both take output of evaluate stage as input but they don't depend on each other (visualize_1 doesn't not depend on visualize_2 and vice versa)

flowchart TD
    node1["data/data.xml.dvc"]
    node2["prepare"]
    node3["train"]
    node4["evaluate"]
    node5["visualize_1"]
    node6["visualize_2"]

    node1 --> node2
    node2 --> node3
    node3 --> node4
    node4 --> node5
    node4 --> node6

Loading

So will i be able to run visualize_1 and visualize_2 together in parallel in a single pipeline?
@skshetry

[skshetry]

So will i be able to run visualize_1 and visualize_2 together in parallel in a single pipeline? @skshetry

Yes, in this case, both of those stages will be parallelized after evaluate completes. You can give #10922 a try and give a feedback.

pip install "dvc[s3] @ git+https://github.com/treeverse/dvc.git@refs/pull/10922/merge"