Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move benchmark download tool and visualization to the CI #9075

Merged
merged 94 commits into from
Feb 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
1d31527
modulearize bench_download tool
Akirathan Feb 15, 2024
261a1cf
Move python version control at the beginning
Akirathan Feb 15, 2024
49223b2
Improve error message for dependencies miss
Akirathan Feb 15, 2024
d6bc3a4
Extract dataclasses into __init__
Akirathan Feb 15, 2024
33575cf
Add gh.py for GH query handling
Akirathan Feb 15, 2024
05c5591
Do not fail if there are more than 1 artifact in a bench report
Akirathan Feb 15, 2024
a1c21c7
Add jinja template functionality to template_render.py
Akirathan Feb 15, 2024
1f6c079
Improve logging
Akirathan Feb 15, 2024
fb02b4a
Add simple test case
Akirathan Feb 15, 2024
25063ce
Implement fetch_file from GH
Akirathan Feb 15, 2024
3c37a48
Add cache skeleton
Akirathan Feb 15, 2024
ad89214
Add git.py
Akirathan Feb 16, 2024
3f49ea1
Fix Jinja template render
Akirathan Feb 16, 2024
c4d1718
gh.fetch_file does not fail
Akirathan Feb 16, 2024
0cf7abd
gh.invoke_gh_api does not fail, and takes method arg
Akirathan Feb 16, 2024
94db479
Add test case to test_gh
Akirathan Feb 16, 2024
07bca0b
Implement remote_cache.py
Akirathan Feb 16, 2024
41f479e
Use readonly remote cache in bench_download script
Akirathan Feb 16, 2024
e8c7f71
Fix path in template rendering
Akirathan Feb 16, 2024
9cb2a26
Remove unused imports
Akirathan Feb 16, 2024
bc1f372
Remove unused Cache classes
Akirathan Feb 19, 2024
923d870
Move some commont functionality into bench_results and utils
Akirathan Feb 19, 2024
6d3ee25
Improve logging
Akirathan Feb 19, 2024
58cdf01
Add skeleton of website_regen.py
Akirathan Feb 19, 2024
54a9ce2
Add some tests on bench_results
Akirathan Feb 19, 2024
fcf1be1
Search bench report artifact by name
Akirathan Feb 19, 2024
c50aac7
Add utils.parse_commit_timestamp
Akirathan Feb 19, 2024
8e3ff5c
Optimize imports
Akirathan Feb 19, 2024
762a256
Redirect output of git.clone
Akirathan Feb 19, 2024
1e8caff
Test SyncRemoteCache
Akirathan Feb 19, 2024
d401604
Move more common functionality into utils and bench_results
Akirathan Feb 19, 2024
555666b
Add git.pull
Akirathan Feb 19, 2024
96c1cc4
RemoteCache does not have initialize method
Akirathan Feb 19, 2024
1e7c761
Move generate_bench_website to utils
Akirathan Feb 19, 2024
51e1b09
SyncRemoteCache can be initialized with local repo
Akirathan Feb 19, 2024
ffe2fab
SyncRemoteCache has html files fields
Akirathan Feb 19, 2024
4ffdadf
Add test_sync_remote_cache_from_local_repo
Akirathan Feb 19, 2024
faa9b41
Finish implementation of website_regen
Akirathan Feb 19, 2024
98f0175
generate_website is in its own module
Akirathan Feb 19, 2024
f8e1848
Fix tests
Akirathan Feb 19, 2024
220ed1c
Add test_website_regen
Akirathan Feb 19, 2024
36dfbd9
Merge branch 'develop' into wip/akirathan/8857-bench-on-ci
Akirathan Feb 20, 2024
4f7f164
Remove deprecated methods
Akirathan Feb 22, 2024
4c9f463
Metnion how to run tests in the README
Akirathan Feb 22, 2024
0de278c
Skip remote repo cloning test
Akirathan Feb 22, 2024
0868f9a
Move some common functionality into utils
Akirathan Feb 22, 2024
7493c2b
Test fetching stdlib job report
Akirathan Feb 22, 2024
9f737d1
bench_report is searched by all the artifact names
Akirathan Feb 22, 2024
52e6c31
website_regen has dry_run option
Akirathan Feb 22, 2024
6793a4f
Remove deprecated method
Akirathan Feb 22, 2024
313c90a
JobRun and JobReport have json encoding
Akirathan Feb 22, 2024
0a7f4d2
Json files are saved in pretty form
Akirathan Feb 22, 2024
2633e5c
Add Benchmarks upload workflow file
Akirathan Feb 22, 2024
de89366
Fix input name in workflow file
Akirathan Feb 22, 2024
7dc9ed7
[WIP] temporarily enable the workflow on all pull requests
Akirathan Feb 22, 2024
7195ffb
Fix repository arg
Akirathan Feb 22, 2024
46ef8ef
Add dry-run step
Akirathan Feb 22, 2024
b650ac6
Fix cd
Akirathan Feb 22, 2024
c53f854
Update deps
Akirathan Feb 22, 2024
4d7c694
Fix path to engine-benchmark-results repo
Akirathan Feb 22, 2024
b4e52bb
Use GH_TOKEn secret
Akirathan Feb 22, 2024
1278ad3
Use GH_TOKEn secret even in dry-run
Akirathan Feb 22, 2024
d4b3259
Better error message in git.commit
Akirathan Feb 22, 2024
dc1f2dd
Add setup git step
Akirathan Feb 22, 2024
9c23c85
Add even more error messages
Akirathan Feb 22, 2024
d640a40
Do not do --dry-run on the CI
Akirathan Feb 23, 2024
6e4c9f0
Run the job every day at 5:30 AM
Akirathan Feb 23, 2024
99286f4
Set the git email to actions-at-github.org
Akirathan Feb 23, 2024
e1ee00d
Use the correct token for bench uploadeds
Akirathan Feb 23, 2024
56c04cb
Set the remote url to enable push to engine-benchmark-results repo
Akirathan Feb 26, 2024
6ecbbff
Add optional fields param to gh.invoke_api
Akirathan Feb 26, 2024
51e9cce
Add git.head_commit function
Akirathan Feb 26, 2024
ce319dc
Push via GH API, and not via `git push`
Akirathan Feb 26, 2024
01faa53
[WIP] Temporarily disable the Upload benchmarks job
Akirathan Feb 27, 2024
d6aa6ca
[WIP] Do plain `git push` in the action
Akirathan Feb 27, 2024
0506edb
Set token in checkout action
Akirathan Feb 27, 2024
3f8b290
Unset extra authentication header
Akirathan Feb 27, 2024
7af7146
Fix field body in gh.invoke_gh_api
Akirathan Feb 27, 2024
c081488
experiment
mwu-tow Feb 28, 2024
37473f9
experiment
mwu-tow Feb 28, 2024
f52c058
experiment
mwu-tow Feb 28, 2024
05d9083
experiment
mwu-tow Feb 28, 2024
aa1d283
Use workflow_run trigger instead of schedule
Akirathan Feb 28, 2024
970e3da
Fix "Set up git" step
Akirathan Feb 28, 2024
010843d
Enable the Upload benchmarks step
Akirathan Feb 28, 2024
7da61c7
Add anchor to the HTML template
Akirathan Feb 28, 2024
9128d33
Revert "Push via GH API, and not via `git push`"
Akirathan Feb 28, 2024
d7283ae
Implement backoff in gh api
Akirathan Feb 28, 2024
8753c7c
Add time when the website was generated
Akirathan Feb 28, 2024
3e765d1
Relax the assertion in git.commit
Akirathan Feb 28, 2024
45d68fc
Add more reasonable commit message
Akirathan Feb 28, 2024
76802d3
Fix git.status
Akirathan Feb 28, 2024
9cefc5f
Remove the `on: pull_request` trigger
Akirathan Feb 28, 2024
846e02e
fmt
Akirathan Feb 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .github/workflows/bench-upload.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# This file is manually managed. It is used to upload benchmarks to to the
# https://github.com/enso-org/engine-benchmark-results repository.

name: Benchmarks upload
on:
workflow_run:
workflows: ["Benchmark Engine", "Benchmark Standard Libraries"]
types:
- completed
jobs:
upload-benchmarks:
name: Upload benchmarks
runs-on: ubuntu-latest
steps:
- name: Checkout enso repository
uses: actions/checkout@v4
with:
repository: enso-org/enso
path: enso
- name: Checkout engine-benchmark-results repository
uses: actions/checkout@v4
with:
repository: enso-org/engine-benchmark-results
path: engine-benchmark-results
token: ${{ secrets.ENSO_BENCHMARK_RESULTS_TOKEN }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
python3 \
python3-jinja2 \
python3-numpy \
python3-pandas
sudo apt-get install -y gh
- name: Set up git
run: |
git config --global user.email "[email protected]"
git config --global user.name "Enso CI Bot"
- name: Upload benchmarks
run: |
cd enso/tools/performance/engine-benchmarks
python3 website_regen.py \
-v \
--local-repo ${{ github.workspace }}/engine-benchmark-results
env:
GITHUB_TOKEN: ${{ secrets.ENSO_BENCHMARK_RESULTS_TOKEN }}
6 changes: 6 additions & 0 deletions tools/performance/engine-benchmarks/README.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to point out that the engine-benchmarks name is no longer fully correct. The tool supports -s engine as well as -s stdlib - e.g. it is not only for engine benchmarks.

Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ This directory contains a python script `bench_download.py` for downloading
Engine and stdlib benchmark results from GitHub, and `Engine_Benchs` Enso
project for analysing the downloaded data.

Note that for convenience, there is `bench_tool` directory that is a Python
package. The `bench_download.py` script uses this package.

To run all the Python tests for that package, run `python -m unittest` in this
directory.

Dependencies for `bench_download.py`:

- python >= 3.7
Expand Down
Loading
Loading