Skip to content

Commit 3b597ba

Browse files
arjunsuresharjunsuresh
authored
Merge branch 'mlcommons:master' into master
2 parents 1c6bc49 + be6ff52 commit 3b597ba

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+2254
-1654
lines changed

‎.github/scripts/format-cpp.sh

Lines changed: 0 additions & 26 deletions
This file was deleted.

‎.github/scripts/format-py.sh

Lines changed: 0 additions & 26 deletions
This file was deleted.

‎.github/workflows/build_wheels.yml

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,15 +74,20 @@ jobs:
7474
strategy:
7575
fail-fast: false
7676
matrix:
77-
os: [ubuntu-latest, windows-latest, macos-latest]
77+
os: [ubuntu-latest, windows-latest, macos-latest, macos-13]
7878

7979
steps:
8080
- uses: actions/checkout@v3
8181

8282
- uses: actions/setup-python@v3
8383

8484
- name: Install requirements
85-
run: python -m pip install cibuildwheel twine
85+
run: python -m pip install cibuildwheel twine build
86+
87+
- name: Build src dist
88+
if: ${{ matrix.os == 'ubuntu-latest' }}
89+
run: |
90+
python -m build --sdist --outdir wheels loadgen
8691
8792
- name: Build wheels
8893
run: git pull && python -m cibuildwheel loadgen/ --output-dir wheels
@@ -110,12 +115,18 @@ jobs:
110115
with:
111116
name: wheels-ubuntu-latest
112117
path: wheels
113-
# Download the built wheels from macOS
114-
- name: Download macOS wheels
118+
# Download the built wheels from macOS-latest
119+
- name: Download macOS-latest wheels
115120
uses: actions/download-artifact@v4
116121
with:
117122
name: wheels-macos-latest
118123
path: wheels
124+
# Download the built wheels from macOS-13 (x86)
125+
- name: Download macOS-13 (x86) wheels
126+
uses: actions/download-artifact@v4
127+
with:
128+
name: wheels-macos-13
129+
path: wheels
119130
# Download the built wheels from Windows
120131
- name: Download Windows wheels
121132
uses: actions/download-artifact@v4

‎.github/workflows/format.yml

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,50 +1,60 @@
11
# Automatic code formatting
2-
name: "format"
2+
name: "Code formatting"
33
on:
4-
pull_request:
5-
branches: [ master ]
6-
types: [opened, closed, synchronize]
7-
4+
push:
5+
branches:
6+
- "**"
87

98
env:
109
python_version: "3.9"
11-
HEAD_REF: ${{ github.head_ref }}
1210

1311
jobs:
1412
format-code:
1513
runs-on: ubuntu-latest
1614
steps:
17-
- uses: actions/checkout@v3
15+
- uses: actions/checkout@v4
1816
with:
1917
fetch-depth: 0
18+
ssh-key: ${{ secrets.DEPLOY_KEY }}
2019
- name: Set up Python ${{ env.python_version }}
2120
uses: actions/setup-python@v3
2221
with:
2322
python-version: ${{ env.python_version }}
2423

25-
- name: Install dependencies
24+
- name: Format modified python files
25+
env:
26+
filter: ${{ github.event.before }}
2627
run: |
2728
python3 -m pip install autopep8
29+
for FILE in $(git diff --name-only $filter | grep -E '.*\.py$')
30+
do
31+
# Check if the file still exists in the working tree
32+
if [ -f "$FILE" ]; then
33+
autopep8 --in-place -a "$FILE"
34+
git add "$FILE"
35+
fi
36+
done
2837
29-
- name: Grant permissions
30-
run: |
31-
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/format-cpp.sh"
32-
chmod +x "${GITHUB_WORKSPACE}/.github/scripts/format-py.sh"
33-
34-
- name: Format Codebase
38+
- name: Format modified C++ files
39+
env:
40+
filter: ${{ github.event.before }}
3541
run: |
36-
git remote add upstream ${{ github.event.pull_request.base.repo.clone_url }}
37-
git fetch upstream ${{ github.event.pull_request.base.ref }}
38-
".github/scripts/format-cpp.sh" "upstream" "${{ github.event.pull_request.base.ref }}"
39-
".github/scripts/format-py.sh" "upstream" "${{ github.event.pull_request.base.ref }}"
42+
for FILE in $(git diff --name-only $filter | grep -E '.*\.(cc|cpp|h|hpp)$')
43+
do
44+
# Check if the file still exists in the working tree
45+
if [ -f "$FILE" ]; then
46+
clang-format -i -style=file $FILE
47+
git add $FILE
48+
fi
49+
done
4050
41-
- name: Commit
51+
- name: Commit and push changes
4252
run: |
4353
HAS_CHANGES=$(git diff --staged --name-only)
4454
if [ ${#HAS_CHANGES} -gt 0 ]; then
45-
git checkout -B "$HEAD_REF"
46-
git config --global user.email "${{ github.actor }}@users.noreply.github.com"
47-
git config --global user.name "${{ github.actor }}"
55+
git config --global user.name mlcommons-bot
56+
git config --global user.email "mlcommons-bot@users.noreply.github.com"
57+
# Commit changes
4858
git commit -m '[Automated Commit] Format Codebase'
49-
git push origin "$HEAD_REF"
50-
fi
59+
git push
60+
fi

‎.github/workflows/test-bert.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,4 @@ jobs:
3636
python3 -m pip install cm4mlops
3737
- name: Test BERT and end to end submission generation
3838
run: |
39-
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=bert-99 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --adr.compiler.tags=gcc --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.$PR_HEAD_REF --adr.loadgen.version=custom
39+
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=bert-99 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --adr.compiler.tags=gcc --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.$PR_HEAD_REF --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom

‎.github/workflows/test-resnet50.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,4 @@ jobs:
3838
python3 -m pip install cm4mlops
3939
- name: Test Resnet50 and end to end submission generation
4040
run: |
41-
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
41+
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}

‎.github/workflows/test-retinanet.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,4 @@ jobs:
3636
python3 -m pip install cm4mlops
3737
- name: Test Retinanet and end to end submission generation
3838
run: |
39-
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.$PR_HEAD_REF --adr.loadgen.version=custom
39+
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.$PR_HEAD_REF --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom

‎.github/workflows/test-rgat.yml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Test for MLPerf inference rgat submission generation using CM script automation
2+
3+
on:
4+
pull_request:
5+
branches: [ "master", "dev" ]
6+
paths:
7+
- graph/R-GAT/**
8+
- loadgen/**
9+
- tools/submission/**
10+
- .github/workflows/test-rgat.yml
11+
- '!**.md'
12+
13+
env:
14+
PR_HEAD_REF: ${{ github.event.pull_request.head.ref }}
15+
16+
jobs:
17+
build:
18+
19+
runs-on: ubuntu-latest
20+
strategy:
21+
fail-fast: false
22+
matrix:
23+
python-version: [ "3.11" ]
24+
backend: [ "pytorch" ]
25+
loadgen-flag: [ "" ]
26+
27+
steps:
28+
- uses: actions/checkout@v3
29+
- name: Set up Python ${{ matrix.python-version }}
30+
uses: actions/setup-python@v3
31+
with:
32+
python-version: ${{ matrix.python-version }}
33+
- name: Install dependencies
34+
run: |
35+
python3 -m pip install cm4mlops
36+
- name: Test R-GAT and end to end submission generation
37+
run: |
38+
cm run script --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@ loadgen/build/
22
libmlperf_loadgen.a
33
__pycache__/
44
generated/
5+
*.swp

‎compliance/nvidia/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,6 @@ The `run_verification.py` found in each test directory will copy the test files
3838
| gpt-j | - |
3939
| stable-diffusion-xl | [TEST01](./TEST01/), [TEST04](./TEST04/) |
4040
| Llama2-70b | [TEST06](./TEST06/) |
41+
| Llama3.1-405b | [TEST06](./TEST06/) |
4142
| mixtral-8x7b | [TEST06](./TEST06/) |
43+
| R-GAT | [TEST01](./TEST01/) |

‎docs/benchmarks/graph/get-rgat-data.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
hide:
3+
- toc
4+
---
5+
6+
# Graph Neural Network using R-GAT
7+
8+
## Dataset
9+
10+
The benchmark implementation run command will automatically download the validation and calibration datasets and do the necessary preprocessing. In case you want to download only the datasets, you can use the below commands.
11+
12+
=== "Full Dataset"
13+
R-GAT validation run uses the IGBH dataset consisting of 547,306,935 nodes and 5,812,005,639 edges.
14+
15+
### Get Full Dataset
16+
```
17+
cm run script --tags=get,dataset,igbh,_full -j
18+
```
19+
20+
=== "Debug Dataset"
21+
R-GAT debug run uses the IGBH debug dataset(tiny).
22+
23+
### Get Full Dataset
24+
```
25+
cm run script --tags=get,dataset,igbh,_debug -j
26+
```
27+
28+
## Model
29+
The benchmark implementation run command will automatically download the required model and do the necessary conversions. In case you want to only download the official model, you can use the below commands.
30+
31+
Get the Official MLPerf R-GAT Model
32+
33+
=== "PyTorch"
34+
35+
### PyTorch
36+
```
37+
cm run script --tags=get,ml-model,rgat -j
38+
```
39+

‎docs/benchmarks/graph/rgat.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
hide:
3+
- toc
4+
---
5+
6+
7+
# Graph Neural Network using R-GAT
8+
9+
10+
=== "MLCommons-Python"
11+
## MLPerf Reference Implementation in Python
12+
13+
{{ mlperf_inference_implementation_readme (4, "rgat", "reference", devices = ["CPU", "CUDA"]) }}

‎docs/index.md

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# MLPerf Inference Benchmarks
22

33
## Overview
4-
The currently valid [MLPerf Inference Benchmarks](index_gh.md) as of MLPerf inference v4.0 round are listed below, categorized by tasks. Under each model you can find its details like the dataset used, reference accuracy, server latency constraints etc.
4+
The currently valid [MLPerf Inference Benchmarks](index_gh.md) as of MLPerf inference v5.0 round are listed below, categorized by tasks. Under each model you can find its details like the dataset used, reference accuracy, server latency constraints etc.
55

66
---
77

@@ -80,7 +80,7 @@ The currently valid [MLPerf Inference Benchmarks](index_gh.md) as of MLPerf infe
8080
- **Server Scenario Latency Constraint**: 130ms
8181
- **Equal Issue mode**: False
8282
- **High accuracy variant**: yes
83-
- **Submission Category**: Datacenter, Edge
83+
- **Submission Category**: Edge
8484

8585
#### [LLAMA2-70B](benchmarks/language/llama2-70b.md)
8686
- **Dataset**: OpenORCA (GPT-4 split, max_seq_len=1024)
@@ -157,11 +157,22 @@ The currently valid [MLPerf Inference Benchmarks](index_gh.md) as of MLPerf infe
157157
- **High accuracy variant**: Yes
158158
- **Submission Category**: Datacenter
159159

160+
## Graph Neural Networks
161+
### [R-GAT](benchmarks/graph/rgat.md)
162+
- **Dataset**: Illinois Graph Benchmark Heterogeneous validation dataset
163+
- **Dataset Size**: 788,379
164+
- **QSL Size**: 788,379
165+
- **Number of Parameters**:
166+
- **Reference Model Accuracy**: ACC = ?
167+
- **Server Scenario Latency Constraint**: N/A
168+
- **Equal Issue mode**: True
169+
- **High accuracy variant**: No
170+
- **Submission Category**: Datacenter
160171
---
161172

162173
## Submission Categories
163-
- **Datacenter Category**: All the current inference benchmarks are applicable to the datacenter category.
164-
- **Edge Category**: All benchmarks except DLRMv2, LLAMA2-70B, and Mixtral-8x7B are applicable to the edge category.
174+
- **Datacenter Category**: All benchmarks except bert are applicable to the datacenter category for inference v5.0.
175+
- **Edge Category**: All benchmarks except DLRMv2, LLAMA2-70B, Mixtral-8x7B and R-GAT are applicable to the edge category for v5.0.
165176

166177
## High Accuracy Variants
167178
- **Benchmarks**: `bert`, `llama2-70b`, `gpt-j`, `dlrm_v2`, and `3d-unet` have a normal accuracy variant as well as a high accuracy variant.

0 commit comments

Comments
 (0)