[BUG] Validating installed plugins is still timing out (often)

[dbwiddis]

What is the bug?

Performing validation of installed plugins is failing with a timeout.

This is a continuation of #352 which was apparently not fixed by #353 and needs another approach.

How can one reproduce the bug?

Intermittent failures executing workflows on a test cluster.

What is the expected behavior?

We are able to quickly get installed plugins on a node without blocking input or timing out.

Do you have any additional context?

Other validation steps are fast. The plugin validation requires a transport request which introduces possible latency.

The list of installed plugins shouldn't change often and may be something we can try to collect shortly after startup, and store somewhere. Plugins can't be added during runtime; they can be removed but that may be an acceptable case to ignore in validation.

In any case, we need a much more reliable method of detecting installed plugins.

[dbwiddis]

It looks like the PluginsService instance is bound via injection and we could inject it in the constructor for CreateWorkflowTransportAction, (and Provision... and Deprovision... also). It has a method info() which returns the PluginsAndModules object that the current code is attempting to fetch via a lot of extra overhead.

We could pass that object to the validation method as an argument.

[dbwiddis]

Closing as this was fixed in #375.

@joshpalis if you put "Fixes #369" in the PR description it auto-links to close the respective issue... just putting the issue number without "Fixes" or "Closes" or "Resolves" doesn't do anything.