Skip to content

Commit e8ed453

Browse files
arjunsuresharjunsuresh
authored
Merge branch 'mlperf-inference-results-scc24' into mlperf-inference-results-scc24
2 parents 4c2068f + 07e9df2 commit e8ed453

File tree

35 files changed

+1785
-399
lines changed

35 files changed

+1785
-399
lines changed

‎open/MLCommons/measurements/41485dfb4f36-reference-gpu-pytorch_v2.4.1-scc24-base_cu124/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
2+
|---------------------|------------|-----------------------|--------------|-------------------|
3+
| stable-diffusion-xl | offline | (15.18544, 235.69504) | 0.375 | - |

‎open/MLCommons/measurements/41485dfb4f36-reference-gpu-pytorch_v2.4.1-scc24-base_cu124/stable-diffusion-xl/offline/README.md

Lines changed: 95 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,95 @@
1-
Test
1+
2+
This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).
3+
4+
*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*
5+
6+
## Host platform
7+
8+
* OS version: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
9+
* CPU version: x86_64
10+
* Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]
11+
* MLCommons CM version: 3.0.1
12+
13+
## CM Run Command
14+
15+
See [CM installation guide](https://docs.mlcommons.org/inference/install/).
16+
17+
```bash
18+
pip install -U cmind
19+
20+
cm rm cache -f
21+
22+
cm pull repo gateoverflow@cm4mlops --checkout=52f2ebb1436d6e255d60a3dc0acf99e4f24f492d
23+
24+
cm run script \
25+
--tags=app,mlperf,inference,generic,_reference,_sdxl,_pytorch,_cuda,_test,_r4.1-dev_default,_float16,_offline \
26+
--quiet=true \
27+
--env.CM_MLPERF_MODEL_SDXL_DOWNLOAD_TO_HOST=yes \
28+
--env.CM_QUIET=yes \
29+
--env.CM_MLPERF_IMPLEMENTATION=reference \
30+
--env.CM_MLPERF_MODEL=sdxl \
31+
--env.CM_MLPERF_RUN_STYLE=test \
32+
--env.CM_MLPERF_BACKEND=pytorch \
33+
--env.CM_MLPERF_SUBMISSION_SYSTEM_TYPE=datacenter \
34+
--env.CM_MLPERF_CLEAN_ALL=True \
35+
--env.CM_MLPERF_DEVICE=cuda \
36+
--env.CM_MLPERF_USE_DOCKER=True \
37+
--env.CM_MLPERF_MODEL_PRECISION=float16 \
38+
--env.OUTPUT_BASE_DIR=/home/arjun/scc_gh_action_results \
39+
--env.CM_MLPERF_LOADGEN_SCENARIO=Offline \
40+
--env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/arjun/scc_gh_action_submissions \
41+
--env.CM_MLPERF_INFERENCE_VERSION=4.1-dev \
42+
--env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1-dev_default \
43+
--env.CM_MLPERF_SUBMISSION_GENERATION_STYLE=short \
44+
--env.CM_MLPERF_SUT_NAME_RUN_CONFIG_SUFFIX4=scc24-base \
45+
--env.CM_DOCKER_IMAGE_NAME=scc24-reference \
46+
--env.CM_MLPERF_LOADGEN_ALL_MODES=yes \
47+
--env.CM_MLPERF_LAST_RELEASE=v4.0 \
48+
--env.CM_TMP_CURRENT_PATH=/home/arjun/actions-runner/_work/cm4mlops/cm4mlops \
49+
--env.CM_TMP_PIP_VERSION_STRING= \
50+
--env.CM_MODEL=sdxl \
51+
--env.CM_MLPERF_LOADGEN_COMPLIANCE=no \
52+
--env.CM_MLPERF_CLEAN_SUBMISSION_DIR=yes \
53+
--env.CM_RERUN=yes \
54+
--env.CM_MLPERF_LOADGEN_EXTRA_OPTIONS= \
55+
--env.CM_MLPERF_LOADGEN_MODE=performance \
56+
--env.CM_MLPERF_LOADGEN_SCENARIOS,=Offline \
57+
--env.CM_MLPERF_LOADGEN_MODES,=performance,accuracy \
58+
--env.CM_OUTPUT_FOLDER_NAME=test_results \
59+
--add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r4_1-dev \
60+
--add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r4_1-dev \
61+
--add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r4_1-dev \
62+
--add_deps_recursive.submission-checker.tags=_short-run \
63+
--add_deps_recursive.coco2014-preprocessed.tags=_size.50,_with-sample-ids \
64+
--add_deps_recursive.coco2014-dataset.tags=_size.50,_with-sample-ids \
65+
--add_deps_recursive.nvidia-preprocess-data.extra_cache_tags=scc24-base \
66+
--v=False \
67+
--print_env=False \
68+
--print_deps=False \
69+
--dump_version_info=True \
70+
--env.OUTPUT_BASE_DIR=/home/arjun/scc_gh_action_results \
71+
--env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/home/arjun/scc_gh_action_submissions \
72+
--env.SDXL_CHECKPOINT_PATH=/home/cmuser/CM/repos/local/cache/6be1f30ecbde4c4e/stable_diffusion_fp16
73+
```
74+
*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
75+
you should simply reload gateoverflow@cm4mlops without checkout and clean CM cache as follows:*
76+
77+
```bash
78+
cm rm repo gateoverflow@cm4mlops
79+
cm pull repo gateoverflow@cm4mlops
80+
cm rm cache -f
81+
82+
```
83+
84+
## Results
85+
86+
Platform: 41485dfb4f36-reference-gpu-pytorch_v2.4.1-scc24-base_cu124
87+
88+
Model Precision: fp32
89+
90+
### Accuracy Results
91+
`CLIP_SCORE`: `15.18544`, Required accuracy for closed division `>= 31.68632` and `<= 31.81332`
92+
`FID_SCORE`: `235.69504`, Required accuracy for closed division `>= 23.01086` and `<= 23.95008`
93+
94+
### Performance Results
95+
`Samples per second`: `0.375138`

‎open/MLCommons/measurements/41485dfb4f36-reference-gpu-pytorch_v2.4.1-scc24-base_cu124/stable-diffusion-xl/offline/cm-version-info.json

Lines changed: 24 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@
117117
"script_tags": "detect-os,detect,os,info",
118118
"script_variations": "",
119119
"version": "",
120-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
120+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
121121
}
122122
},
123123
{
@@ -137,7 +137,7 @@
137137
"script_tags": "detect,cpu,detect-cpu,info",
138138
"script_variations": "",
139139
"version": "",
140-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
140+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
141141
}
142142
},
143143
{
@@ -147,7 +147,7 @@
147147
"script_tags": "get,sys-utils-cm",
148148
"script_variations": "",
149149
"version": "",
150-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
150+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
151151
}
152152
},
153153
{
@@ -157,7 +157,7 @@
157157
"script_tags": "get,python,python3,get-python,get-python3",
158158
"script_variations": "",
159159
"version": "3.10.12",
160-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
160+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
161161
}
162162
},
163163
{
@@ -167,7 +167,7 @@
167167
"script_tags": "get,cuda,cuda-compiler,cuda-lib,toolkit,lib,nvcc,get-nvcc,get-cuda,46d133d9ef92422d",
168168
"script_variations": "cudnn",
169169
"version": "12.4",
170-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
170+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
171171
}
172172
},
173173
{
@@ -177,7 +177,7 @@
177177
"script_tags": "get,install,generic,generic-python-lib",
178178
"script_variations": "torch_cuda",
179179
"version": "2.4.1",
180-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
180+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
181181
}
182182
},
183183
{
@@ -187,7 +187,7 @@
187187
"script_tags": "get,install,generic,generic-python-lib",
188188
"script_variations": "torchvision_cuda",
189189
"version": "0.19.1",
190-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
190+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
191191
}
192192
},
193193
{
@@ -197,7 +197,7 @@
197197
"script_tags": "get,raw,ml-model,stable-diffusion,sdxl,text-to-image",
198198
"script_variations": "pytorch",
199199
"version": "",
200-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
200+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
201201
}
202202
},
203203
{
@@ -207,7 +207,7 @@
207207
"script_tags": "get,dataset,coco2014,object-detection,original",
208208
"script_variations": "validation,size.50,with-sample-ids",
209209
"version": "",
210-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
210+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
211211
}
212212
},
213213
{
@@ -287,7 +287,7 @@
287287
"script_tags": "generate,mlperf,inference,user-conf,inference-user-conf",
288288
"script_variations": "",
289289
"version": "",
290-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
290+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
291291
}
292292
},
293293
{
@@ -297,7 +297,7 @@
297297
"script_tags": "get,loadgen,inference,inference-loadgen,mlperf,mlcommons",
298298
"script_variations": "",
299299
"version": "master",
300-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
300+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
301301
}
302302
},
303303
{
@@ -307,7 +307,7 @@
307307
"script_tags": "get,src,source,inference,inference-src,inference-source,mlperf,mlcommons",
308308
"script_variations": "",
309309
"version": "master-git-41fa8aadd1ba0ecc97f6a519d8b42b04278e5f24",
310-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
310+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
311311
}
312312
},
313313
{
@@ -317,7 +317,7 @@
317317
"script_tags": "get,src,source,inference,inference-src,inference-source,mlperf,mlcommons",
318318
"script_variations": "",
319319
"version": "master-git-41fa8aadd1ba0ecc97f6a519d8b42b04278e5f24",
320-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
320+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
321321
}
322322
},
323323
{
@@ -327,7 +327,7 @@
327327
"script_tags": "get,install,generic,generic-python-lib",
328328
"script_variations": "package.psutil",
329329
"version": "6.0.0",
330-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
330+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
331331
}
332332
},
333333
{
@@ -337,7 +337,7 @@
337337
"script_tags": "get,install,generic,generic-python-lib",
338338
"script_variations": "package.diffusers",
339339
"version": "0.30.3",
340-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
340+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
341341
}
342342
},
343343
{
@@ -347,7 +347,7 @@
347347
"script_tags": "get,install,generic,generic-python-lib",
348348
"script_variations": "package.transformers",
349349
"version": "4.45.2",
350-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
350+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
351351
}
352352
},
353353
{
@@ -357,7 +357,7 @@
357357
"script_tags": "get,install,generic,generic-python-lib",
358358
"script_variations": "package.torchvision",
359359
"version": "0.19.1",
360-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
360+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
361361
}
362362
},
363363
{
@@ -367,7 +367,7 @@
367367
"script_tags": "get,install,generic,generic-python-lib",
368368
"script_variations": "package.accelerate",
369369
"version": "1.0.0",
370-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
370+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
371371
}
372372
},
373373
{
@@ -377,7 +377,7 @@
377377
"script_tags": "get,install,generic,generic-python-lib",
378378
"script_variations": "package.torchmetrics",
379379
"version": "1.4.2",
380-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
380+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
381381
}
382382
},
383383
{
@@ -387,7 +387,7 @@
387387
"script_tags": "get,install,generic,generic-python-lib",
388388
"script_variations": "package.torch-fidelity",
389389
"version": "0.3.0",
390-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
390+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
391391
}
392392
},
393393
{
@@ -397,7 +397,7 @@
397397
"script_tags": "get,install,generic,generic-python-lib",
398398
"script_variations": "package.open_clip_torch",
399399
"version": "2.26.1",
400-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
400+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
401401
}
402402
},
403403
{
@@ -407,7 +407,7 @@
407407
"script_tags": "get,install,generic,generic-python-lib",
408408
"script_variations": "package.opencv-python",
409409
"version": "4.10.0.84",
410-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
410+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
411411
}
412412
},
413413
{
@@ -417,7 +417,7 @@
417417
"script_tags": "get,install,generic,generic-python-lib",
418418
"script_variations": "package.scipy",
419419
"version": "1.10.1",
420-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
420+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
421421
}
422422
},
423423
{
@@ -457,7 +457,7 @@
457457
"script_tags": "mlperf,benchmark-mlperf",
458458
"script_variations": "",
459459
"version": "",
460-
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( offline,_sdxl,_cuda,_pytorch,_float16 )"
460+
"parent": "app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( pytorch,_cuda,_sdxl,_offline,_float16 )"
461461
}
462462
}
463463
]

‎open/MLCommons/measurements/41485dfb4f36-reference-gpu-pytorch_v2.4.1-scc24-base_cu124/stable-diffusion-xl/offline/pip_freeze.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"giturlparse": "0.12.0",
1313
"huggingface-hub": "0.25.1",
1414
"idna": "3.10",
15+
"ijson": "3.3.0",
1516
"importlib_metadata": "8.5.0",
1617
"Jinja2": "3.1.4",
1718
"lightning-utilities": "0.11.7",

‎open/MLCommons/measurements/97c3ec750d0e-nvidia-gpu-TensorRT-scc24-base/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
2+
|---------------------|------------|-----------------------|--------------|-------------------|
3+
| stable-diffusion-xl | offline | (15.47816, 232.17873) | 1.141 | - |

0 commit comments

Comments
 (0)