Clarification of execution order for pre,main,post workflows

[TyIsI]

As I understand from sources, comments and issues, if I wanted to replace a pre,main,post workflow through a simple placeholder script, I would need to set WIREIT_PARALLEL=1 and have each scripting dependency declare the previous step as a dependency?

Concrete example:

{
  ...
  "scripts": {
    ...
    "predb:update": "pnpm run db:pull",
    "db:update": "pnpm run db:generate",
    "postdb:update": "pnpm run db:post-processing"
    ...
  }
  ...
}

By themselves I could migrate them and I think the appropriate way to convert those would look something like this. But it feels a little bit counter-intuitive?

{
    ...
    "wireit": {
        "db:update:step1": {
            "command": "db:pull"
        },
        "db:update:step2": {
            "dependencies": [
                "db:update:step1"
            ],
            "command": "db:generate"
        },
        "db:update:step3": {
            "dependencies": [
                "db:update:step2"
            ],
            "command": "db:dump"
        },
        "db:update": {
            "dependencies": [
                "db:update:step1",
                "db:update:step2",
                "db:update:step3"
            ]
        }
    }
    ...
}

Is there a better/preferred way to build "meta" scripts like this? Or would a simple entry with just the linked tasks as dependencies work and would they be executed in the correct order based on WIREIT_PARALLEL=1?

Thanks!

EDITED optimized example workflow

[aomarks]

I think you want something like this:

{
  "scripts": {
    "db:update": "wireit",
    "db:generate": "wireit",
    "db:pull": "wireit"
  },
  "wireit": {
    "db:update": {
      "command": "..."
      "dependencies": [
        "db:generate"
      ]
    },
    "db:generate": {
      "command": "..."
      "dependencies": [
        "db:pull"
      ]
    },
    "db:pull": {
      "command": "..."
    }
  }
}

You don't need to modify WIREIT_PARALLEL, because wireit will ensure correct execution order based on your dependencies. In this case, when you run npm run db:update, the order that the commands run will always be db:pull, db:generate, db:update no matter what WIREIT_PARALLEL is set to.

[TyIsI]

Sorry, my bad. I forgot to clarify that I don't always want db:generate to run db:pull.
I may want to run db:generate on its own without db:pull. Or as part of a multi step workflow.

Is this something that can be isolated with wireit?

[aomarks]

Ah I see, yeah I missed that detail.

I think for this use-case I would actually just use shell chaining for your macro. Something like this:

{
  "scripts": {
    "db:update": "wireit",
    "db:pull": "wireit",
    "db:generate": "wireit",
    "db:dump": "wireit"
  },
  "wireit": {
    "db:update": {
      "command": "npm run db:pull && npm run db:generate && npm run db:dump"
    },
    "db:pull": {
      "command": "..."
    },
    "db:generate": {
      "command": "..."
    },
    "db:dump": {
      "command": "..."
    }
  }
}

[aomarks]

We could maybe add a feature like that has the same behavior as above, but more concise and without the overhead of those extra processes that each npm run brings. Maybe something like this?:

{
  "scripts": {
    "db:update": "wireit",
    "db:pull": "wireit",
    "db:generate": "wireit",
    "db:dump": "wireit"
  },
  "wireit": {
    "db:update": {
      "sequence": ["db:pull", "db:generate", "db:dump"]
    },
    "db:pull": {
      "command": "..."
    },
    "db:generate": {
      "command": "..."
    },
    "db:dump": {
      "command": "..."
    }
  }
}

Doesn't seem crazy, though I'd like to think a bit more about whether there is some better way to express that.

[TyIsI]

I was kinda thinking about that. Ideally, for something like this, I think it would be amazing to express "downstream" execution dependencies.

Could a sequence flag be a more efficient solution to cut through it? As in, randomization is default unless disabled explicitly? Which would imply sequence mode.

[TyIsI]

Though, now that I've had a chance to look at the code, that would also require having to deal with not just circumventing the shuffle functionality, but also to use a different strategy for iterating over the dependencies, to prevent race conditions from Promise.all.

[TyIsI]

Thanks so far!

[TyIsI]

@aomarks I think I may have a rough idea/solution. I'll look into this and if it's viable, I'm willing to commit implementing it.

In short, the current workflow for dependencies stays the same, but if a dependency entry is a string array, each element is then executed sequentially.

Example:

{
  "scripts": {
    "db:update": "wireit",
    "db:pull": "wireit",
    "db:generate": "wireit",
    "db:dump": "wireit"
  },
  "wireit": {
    "db:update": {
      "dependencies": [["db:pull", "db:generate", "db:dump"]]
    },
    "db:pull": {
      "command": "..."
    },
    "db:generate": {
      "command": "..."
    },
    "db:dump": {
      "command": "..."
    }
  }
}

Reflection: not sure how/if that could would work with the current dependency graphs, but I'm imagining that coercing every "dependencies" config entry into an string[][] object and using uniform sequential execution for the second level would alleviate most of the concerns. And I think a bonus would be that this could allow specific dependency workflows that could be executed in parallel. (Think parallel builds with specific execution order requirements.)

Let me know if this sounds like it might be a viable solution and I'll look into this.

(I'm not sure how wireit does dependency lookup resolution, in terms of whether these hard linked dependencies or if these are vectors that can be looked up. But I'll take a look and see.)

[bitmage]

"dependencies": [["db:pull", "db:generate", "db:dump"]]

A sequence key I think would be more clear / less magical syntax.

[ritschwumm]

imho supporting sequential execution in wireit is not a good idea.

it's easy to understand dependencies have to form a DAG (such that cycles in the graph are forbidden).
as soon as sequential execution comes into play, things become a lot more complicated.

i tried to support this in my own tool (which uses the exact same logic as wireit does), and it's a can of worms.
with sequential execution there are a lot more ways to write dependencies that are impossible to fulfill.
these are not only surprising and hard to understand for the user, but difficult to deal with for the programmer, too.

just one example:

• if A executes B and C in parallel and C depends on B,
  the execution order is clear: first B, then C, and then A
• now if A executes B and C in sequence and C depends on B,
  there is no valid execution order any more.

for even slightly larger dependency graphs, this quickly becomes a mess.

the only halfway good solution i see is to explicitly forbid any dependencies on sequential tasks
so they can only be used as an entry point -- but then using a shell chain is simple enough so it's
probably not worth the effort any more.

[TyIsI]

@ritschwumm not sure if I'm understanding you correctly here, but in my scenario, the perspective is always from "D" and as the graph gets built at "bootstrap" time (i.e. before anything is ran).

I'd have to look at the way the graph is built again. Though I'm sure that there's nothing that a small Dijkstra algorithm couldn't fix.

That said, I'm going to take inspiration from @bitmage and going to look into whether it's possible to modify the role of the script property of the dependency object. Because that would seem like a decent starting point to essentially hoist a dependency chain into the DAG.

[ritschwumm]

@TyIsI dunno where your "D" comes from and what it means - but if by "dijkstra algorithm" you mean topological sorting then once you have cycles in your graph, there is nothing it could possibly solve any more.

[TyIsI]

@ritschwumm for context: #1188 (comment)

[ritschwumm]

@TyIsI that's exactly what i am talking about. "In short, the current workflow for dependencies stays the same, but if a dependency entry is a string array, each element is then executed sequentially." is problematic because sequential execution and execution "by need" don't mix well.