Fold output of each benchmark

[JaroslavTulach]

Pull Request Description

This PR improves output of benchmarks run on GitHub Actions CI to group and fold output for each benchmark.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• All code follows the
  Java
• Benchmarks output improved
  • engine run
  • stdlib run

[JaroslavTulach]

The ::group:: format seems to be mentioned at this discussion.

[radeusgd]

It is documented here: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions#grouping-log-lines

Although the documentation is not very detailed.

[JaroslavTulach]

Update: solved in 9ba4428

I don't know how to print plain line without any prefix in our Rust build system. If I use info! like in 37d0385 the line gets prefixed and looses its :: command prefix. If I use println! like in b1ef6fb the text isn't displayed in the log at all! Any advice from Rust experts?

[Akirathan]

This is a good idea. I am interested in how the end result will look like. I had no idea that there is some output grouping for GH actions. We could probably integrate this functionality even for other actions.

[JaroslavTulach]

Locally the command lines seem to be determined fine. If I execute, I see:

enso$ ./run backend benchmark runtime --minimal-run
 INFO ide_ci::program::command: sbt ℹ️ [info] # *** WARNING: Use non-forked runs only for debugging purposes, not for actual performance runs. ***
 INFO ide_ci::program::command: sbt ℹ️ [info] Iteration   1: 102.137 ms/op
::endgroup::
::group::org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy
 INFO ide_ci::program::command: sbt ℹ️ [info] # Run progress: 12.68% complete, ETA 00:01:29
 INFO ide_ci::program::command: sbt ℹ️ [info] # Fork: N/A, test runs in the host VM

e.g. both ::endgroup and ::group are properly detected and the "INFO..." prefix isn't printed. Let's see what does the CI print?

Finally it does something reasonable!

[JaroslavTulach]

Certain parts of the stdlibs bench run output look nice as well.

Some messages "escape" the grouping. Not sure why and whether that matters that much to prevent integration. Hypothesis:

• either output prints to stdout and compilation detection prints to System.err and the output gets mixed
• or two two messages are printed by different threads

Let's try to switch to stdout - stdlib benchmark is running.

Update: The latest stdlibs output looks pretty decent. Looks like c698ef7 helped.

No longer convinced this is needed at all

[Akirathan]

After some thinking, I believe that your changes are unnecessary, and seeing the current folded output, it is very messy - The group for benchGenerateList obviously contains output even from some other benchmark for example.

May I ask you again to explain your use-case that this folding will solve, and why do you think it is needed?

Note that at the end of each benchmark run (the very end of the whole job output) there is table-like structure with all the scores. In your case it is at https://github.com/enso-org/enso/actions/runs/10627416248/job/29460495368#step:7:1886. If you want to see the output for a single benchmark, you can just type its name in the search field. You would need to do that even if the output was folded.

EDIT: If you still think the folding will be beneficial, I will revoke my PR reject after I see that the folding works and that no output is assigned to a wrong group.

[JaroslavTulach]

... changes are unnecessary...

I am sad you have changed your mind. I believe that with few more homey touches, we can make this folding work well and increase overall satisfaction.

May I ask you again to explain your use-case that this folding will solve, and why do you think it is needed?

• I want to hide non-interesting crap - folding would achieve that
• I want to organize things per interest - using title of a benchmark for navigation seems human friendly
• I was hoping we tackle the size of the log (didn't work trying more in 2344fac) as we are reaching a limit on stdlibs benchmarks

Note that at the end of each benchmark run (the very end of the whole job output) there is table-like structure with all the scores. In your case it is at https://github.com/enso-org/enso/actions/runs/10627416248/job/29460495368#step:7:1886.

Right now the log stops at 22% of stdlibs benchmark run so the table isn't really visible.

If you want to see the output for a single benchmark, you can just type its name in the search field. You would need to do that even if the output was folded.

I've just tried and folding has no negative effect on searching. These ::group: folds are automatically expanded when the found text is in a collapsed fold.

If you still think the folding will be beneficial

The latest stdlibs output looks pretty decent. Looks like c698ef7 helped.

[JaroslavTulach]

The newest idea c36fdad to solve the too huge log issue is to differentiate between execution of all benchmarks and execution of a selected benchmark(s):

• running bench limits the output to ten lines per benchmark - let's see if that helps
• running benchOnly Xyz dumps full output

As a result the CI run will be less verbose, but one will have a chance to re-run individual benchmark locally and see whole output.

[JaroslavTulach]

at the end of each benchmark run (the very end of the whole job output) there is table-like structure with all the scores.

Yes, with the latest run the table is finally visible.

you can just type its name in the search field.

That is still possible even the output is folded.

If you still think the folding will be beneficial, I will revoke my PR reject

Yes, consider revoking your reject.

[Akirathan]

The latest output seems reasonable. The fact that when all the benchmarks are run, the output is less verbose is good - it helps to be able to interactively browse the output on the CI.