Skip to content

Add more figures to the summary #946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 4, 2021
Merged

Conversation

rylev
Copy link
Member

@rylev rylev commented Jul 29, 2021

This adds more figures to the summary.

Note: I've purposefully haven't worked on finishing the visual presentation.

Screenshot 2021-07-29 at 15 05 15

The figures include:

  • all: all test results including insignificant and filtered out results
  • filtered: whatever has not been filtered out by a filter
  • scenario kinds: the various scenario kinds there are (full, incr-full, incr-unchanged, incr-patched) regardless if they have been filtered.

Also a border (red or green) is applied around any summary element that has more improvements or regressions than unchanged results. This is an attempt to draw attention to summary elements that are "significant".

Questions:

  • which summary elements are most useful?
  • Is it useful to highlight summary elements that show a "significant" number of improvements or regressions? If so, should it be how it's currently done (if there are more improvement or regression results than unchanged) or by some percentage of the total number of results? If by percentage, which?
  • Are we missing any good summary figures?
  • Should we try to summarize the summary (i.e., attempt to interpret the results for the viewer so that they do not have to)?

Fixes #935, fixes #188, fixes #953

@rylev rylev requested a review from Mark-Simulacrum July 29, 2021 13:14
@Mark-Simulacrum
Copy link
Member

Might also fix #953, not sure -- I had forgotten about this when opening that issue.

@rylev
Copy link
Member Author

rylev commented Aug 4, 2021

This fixes #953

I've removed the explicit summaries for the different scenario kinds. We can add those back if we want, but for now that information can be retrieved by explicitly filtering.

image

@rylev rylev merged commit 79aa3ce into rust-lang:master Aug 4, 2021
@rylev rylev deleted the bigger-summary branch August 4, 2021 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants