Add awareness of results from previous runs

[bfonta]

A couple of suggestions for improvements to the bot:

• Run first the workflows which have failed in the previous test. Currently AFAIK the bot reruns all tests without considering the previous run, which is inefficient since, if any job is to fail again, it will most likely be the one which has failed in the past. The sooner a workflow crashes, the faster we can debug it. There is no need to wait for all workflows to finish/crash before stopping the bot (or at least before warning the user and make all logs easily accessible).
• Run tests after the first time only if the files which affect those tests have been modified. If we add a print to module A, there shouldn't be the need to rerun a workflow that only looks at module B (assuming it does not communicate with module A in any way).

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @bfonta.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

@bfonta , thanks for the suggestions.
about Run first the workflows which have failed in the previous test. Currently bot runs Relvals with a single runTheMatrix command with -j N (where N depends on the host's available cores) so yes one has to wait for all Relvals to run. In past we tried running N runTheMatrix in parallel with single workflow each and stop the process as soon as something failed but this approach means you will not get errors for relvals which were not run. I think reco at somepoint explicitly requested to run all regardless of failures (sometime it is just some unrelated DAS error and we do not want to not test other relvals due to such errors).

About Run tests after the first time only if the files which affect those tests have been modified , I would have loved to see this and at somepoint I want to implement such a system but for that we really need a very good mapping of Shared libs/plugins used/loaded by a workflow. Even with such a mapping it is not guaranteed that two runs of same workflow will load same plugin e.g. what if on next run DAS returned a different set of files and due to change in input you might end up running different code. I feel that with plugin system and external conditions system we have this really is very tricky to implement. I will be more than happy if someone comes up with a maintainable solution but I feel there will always be corner cases where you might missed running a test because system did not know the exact dependency.