Skip to content

Commit 03af19d

Browse files
gateoverflow-admingateoverflow-admin
committed
Updated results summary
1 parent 2125e9c commit 03af19d

File tree

2 files changed

+93
-19
lines changed

2 files changed

+93
-19
lines changed

‎README.md

+19-17
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
```
2-
[2024-11-18 22:42:37,628 submission_checker1.py:3109 INFO] Results=8, NoResults=0, Power Results=0
3-
[2024-11-18 22:42:37,628 submission_checker1.py:3116 INFO] ---
4-
[2024-11-18 22:42:37,628 submission_checker1.py:3117 INFO] Closed Results=0, Closed Power Results=0
2+
[2024-11-18 22:56:20,857 submission_checker1.py:3109 INFO] Results=10, NoResults=0, Power Results=0
3+
[2024-11-18 22:56:20,857 submission_checker1.py:3116 INFO] ---
4+
[2024-11-18 22:56:20,858 submission_checker1.py:3117 INFO] Closed Results=0, Closed Power Results=0
55
6-
[2024-11-18 22:42:37,628 submission_checker1.py:3122 INFO] Open Results=8, Open Power Results=0
6+
[2024-11-18 22:56:20,858 submission_checker1.py:3122 INFO] Open Results=10, Open Power Results=0
77
8-
[2024-11-18 22:42:37,628 submission_checker1.py:3127 INFO] Network Results=0, Network Power Results=0
8+
[2024-11-18 22:56:20,858 submission_checker1.py:3127 INFO] Network Results=0, Network Power Results=0
99
10-
[2024-11-18 22:42:37,628 submission_checker1.py:3132 INFO] ---
11-
[2024-11-18 22:42:37,628 submission_checker1.py:3134 INFO] Systems=8, Power Systems=0
12-
[2024-11-18 22:42:37,628 submission_checker1.py:3138 INFO] Closed Systems=0, Closed Power Systems=0
13-
[2024-11-18 22:42:37,628 submission_checker1.py:3143 INFO] Open Systems=8, Open Power Systems=0
14-
[2024-11-18 22:42:37,628 submission_checker1.py:3148 INFO] Network Systems=0, Network Power Systems=0
15-
[2024-11-18 22:42:37,628 submission_checker1.py:3153 INFO] ---
16-
[2024-11-18 22:42:37,628 submission_checker1.py:3158 INFO] SUMMARY: submission looks OK
10+
[2024-11-18 22:56:20,858 submission_checker1.py:3132 INFO] ---
11+
[2024-11-18 22:56:20,858 submission_checker1.py:3134 INFO] Systems=10, Power Systems=0
12+
[2024-11-18 22:56:20,858 submission_checker1.py:3138 INFO] Closed Systems=0, Closed Power Systems=0
13+
[2024-11-18 22:56:20,858 submission_checker1.py:3143 INFO] Open Systems=10, Open Power Systems=0
14+
[2024-11-18 22:56:20,858 submission_checker1.py:3148 INFO] Network Systems=0, Network Power Systems=0
15+
[2024-11-18 22:56:20,858 submission_checker1.py:3153 INFO] ---
16+
[2024-11-18 22:56:20,858 submission_checker1.py:3158 INFO] SUMMARY: submission looks OK
1717
INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4mlops/script/run-mlperf-inference-submission-checker/customize.py
1818
1919
```
@@ -23,8 +23,10 @@ INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4ml
2323
| 0 | NTHU | available | open | datacenter | 334907abd4f4 | Kuai-Kuai_702-reference-gpu-pytorch-v2.5.1-scc24-base_cu124 | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.528528 | CLIP_SCORE: 16.4035627245903 FID_SCORE: 235.76925470141796 | 1 | INTEL(R) XEON(R) PLATINUM 8592+ | 2 | 64 | NVIDIA H100 PCIe | 4 | open/NTHU/results/Kuai-Kuai_702-reference-gpu-pytorch-v2.5.1-scc24-base_cu124/stable-diffusion-xl/offline | pytorch v2.5.1 | Ubuntu 22.04 (linux-6.8.0-47-generic-glibc2.35) | Automated by MLCommons CM v3.1.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
2424
| 1 | NTHU | available | open | datacenter | ba7f5c94d10f | ba7f5c94d10f-nvidia-gpu-TensorRT-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 4.2742 | CLIP_SCORE: 31.234263841211796 FID_SCORE: 23.67815832489498 | 1 | INTEL(R) XEON(R) PLATINUM 8592+ | 2 | 64 | NVIDIA H100 PCIe | 4 | open/NTHU/results/ba7f5c94d10f-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 22.04 (linux-6.8.0-47-generic-glibc2.35) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
2525
| 2 | peking1 | available | open | datacenter | mlperf_inference_lry_40 | mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 7.47046 | CLIP_SCORE: 14.028269715607166 FID_SCORE: 84.33061840229146 | 1 | AMD EPYC 9684X 96-Core Processor | 2 | 96 | NVIDIA H100 80GB HBM3 | 5 | open/peking1/results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.33.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
26-
| 3 | UCSD | available | open | datacenter | aqua | aqua-reference-rocm-pytorch-v2.6.0.dev20241109-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.209132 | CLIP_SCORE: 15.227862536907196 FID_SCORE: 236.96183273783743 | 1 | AMD EPYC 9354 32-Core Processor | 2 | 32 | nan | 0 | open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241109-scc24-base/stable-diffusion-xl/offline | pytorch v2.6.0.dev20241109 | Rocky 9.4 (linux-5.14.0-427.42.1.el9_4.x86_64-glibc2.34) | Automated by MLCommons CM v3.1.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
27-
| 4 | UCSD | available | open | datacenter | aqua | aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.20835 | CLIP_SCORE: 16.381833255290985 FID_SCORE: 236.8570675351765 | 1 | AMD EPYC 9354 32-Core Processor | 2 | 32 | nan | 0 | open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-base/stable-diffusion-xl/offline | pytorch v2.6.0.dev20241118 | Ubuntu 22.04 (linux-5.14.0-427.42.1.el9_4.x86_64-glibc2.35) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
28-
| 5 | peking | available | open | datacenter | mlperf_inference_lry_40 | mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 7.64431 | CLIP_SCORE: 14.028269715607166 FID_SCORE: 84.33061840229146 | 1 | AMD EPYC 9684X 96-Core Processor | 2 | 96 | NVIDIA H100 80GB HBM3 | 5 | open/peking/results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.33.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
29-
| 6 | NTUHPC | available | open | datacenter | 1b41d1041a1b | Coffeepot-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 4.18764 | CLIP_SCORE: 16.503750428557396 FID_SCORE: 232.23581854554644 | 1 | AMD EPYC 9654 96-Core Processor | 1 | 96 | NVIDIA H100 PCIe | 4 | open/NTUHPC/results/Coffeepot-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.5.0-27-generic-glibc2.31) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
30-
| 7 | scc2 | available | open | datacenter | f2ea47b0a016 | f2ea47b0a016-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 7.46497 | CLIP_SCORE: 16.63159880042076 FID_SCORE: 234.62921308614995 | 1 | AMD EPYC 9754 128-Core Processor | 2 | 128 | NVIDIA H100 80GB HBM3 | 6 | open/scc2/results/f2ea47b0a016-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.1.0-27-amd64-glibc2.31) | Automated by MLCommons CM v3.3.3. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
26+
| 3 | UCSD | available | open | datacenter | aqua | aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.847525 | CLIP_SCORE: 15.224773076921702 FID_SCORE: 84.24318228239224 | 1 | AMD EPYC 9354 32-Core Processor | 2 | 32 | nan | 0 | open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main/stable-diffusion-xl/offline | pytorch v2.6.0.dev20241118 | Ubuntu 22.04 (linux-5.14.0-427.42.1.el9_4.x86_64-glibc2.35) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
27+
| 4 | UCSD | available | open | datacenter | aqua | aqua-reference-rocm-pytorch-v2.6.0.dev20241109-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.209132 | CLIP_SCORE: 15.227862536907196 FID_SCORE: 236.96183273783743 | 1 | AMD EPYC 9354 32-Core Processor | 2 | 32 | nan | 0 | open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241109-scc24-base/stable-diffusion-xl/offline | pytorch v2.6.0.dev20241109 | Rocky 9.4 (linux-5.14.0-427.42.1.el9_4.x86_64-glibc2.34) | Automated by MLCommons CM v3.1.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
28+
| 5 | UCSD | available | open | datacenter | aqua | aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.20835 | CLIP_SCORE: 16.381833255290985 FID_SCORE: 236.8570675351765 | 1 | AMD EPYC 9354 32-Core Processor | 2 | 32 | nan | 0 | open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-base/stable-diffusion-xl/offline | pytorch v2.6.0.dev20241118 | Ubuntu 22.04 (linux-5.14.0-427.42.1.el9_4.x86_64-glibc2.35) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
29+
| 6 | peking | available | open | datacenter | mlperf_inference_lry_40 | mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 7.64431 | CLIP_SCORE: 14.028269715607166 FID_SCORE: 84.33061840229146 | 1 | AMD EPYC 9684X 96-Core Processor | 2 | 96 | NVIDIA H100 80GB HBM3 | 5 | open/peking/results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.33.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
30+
| 7 | NTUHPC | available | open | datacenter | 1b41d1041a1b | Coffeepot-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 4.18764 | CLIP_SCORE: 16.503750428557396 FID_SCORE: 232.23581854554644 | 1 | AMD EPYC 9654 96-Core Processor | 1 | 96 | NVIDIA H100 PCIe | 4 | open/NTUHPC/results/Coffeepot-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.5.0-27-generic-glibc2.31) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
31+
| 8 | scc2 | available | open | datacenter | f2ea47b0a016 | f2ea47b0a016-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 7.46497 | CLIP_SCORE: 16.63159880042076 FID_SCORE: 234.62921308614995 | 1 | AMD EPYC 9754 128-Core Processor | 2 | 128 | NVIDIA H100 80GB HBM3 | 6 | open/scc2/results/f2ea47b0a016-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-6.1.0-27-amd64-glibc2.31) | Automated by MLCommons CM v3.3.3. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
32+
| 9 | pku | available | open | datacenter | mlperf_inference_lry_40 | mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 8.2807 | CLIP_SCORE: 14.028269715607166 FID_SCORE: 84.33061840229146 | 1 | AMD EPYC 9684X 96-Core Processor | 2 | 96 | NVIDIA H100 80GB HBM3 | 5 | open/pku/results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.33.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v3.4.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |

‎summary_results.json

+74-2
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,42 @@
181181
},
182182
{
183183
"ID": "4.1-0006",
184+
"Submitter": "UCSD",
185+
"Availability": "available",
186+
"Category": "open",
187+
"Suite": "datacenter",
188+
"System": "aqua",
189+
"Platform": "aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main",
190+
"UsedModel": "stable-diffusion-xl",
191+
"Model": "stable-diffusion-xl",
192+
"Scenario": "Offline",
193+
"Accuracy": "CLIP_SCORE: 15.224773076921702 FID_SCORE: 84.24318228239224",
194+
"Nodes": 1,
195+
"Processor": "AMD EPYC 9354 32-Core Processor",
196+
"host_processors_per_node": 2,
197+
"host_processor_core_count": 32,
198+
"Accelerator": "",
199+
"a#": "",
200+
"Location": "open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main/stable-diffusion-xl/offline",
201+
"Software": "pytorch v2.6.0.dev20241118",
202+
"operating_system": "Ubuntu 22.04 (linux-5.14.0-427.42.1.el9_4.x86_64-glibc2.35)",
203+
"Notes": "Automated by MLCommons CM v3.4.1. ",
204+
"compliance": 1,
205+
"errors": 0,
206+
"version": "v4.1",
207+
"inferred": 0,
208+
"has_power": false,
209+
"weight_data_types": "fp32",
210+
"p#": 2,
211+
"Details": "https://github.com/mlcommons/cm4mlperf-inference/tree/mlperf-inference-results-scc24/open/UCSD/results/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main",
212+
"Code": "https://github.com/mlcommons/cm4mlperf-inference/tree/mlperf-inference-results-scc24/open/UCSD/code",
213+
"Unique ID (e.g. for Audit)": "datacenter/open/UCSD/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main/stable-diffusion-xl",
214+
"ColorKey": "availableUCSD",
215+
"Performance_Result": 0.847525,
216+
"Performance_Units": "Samples/s"
217+
},
218+
{
219+
"ID": "4.1-0007",
184220
"Submitter": "peking",
185221
"Availability": "available",
186222
"Category": "open",
@@ -216,7 +252,7 @@
216252
"Performance_Units": "Samples/s"
217253
},
218254
{
219-
"ID": "4.1-0007",
255+
"ID": "4.1-0008",
220256
"Submitter": "peking1",
221257
"Availability": "available",
222258
"Category": "open",
@@ -252,7 +288,43 @@
252288
"Performance_Units": "Samples/s"
253289
},
254290
{
255-
"ID": "4.1-0008",
291+
"ID": "4.1-0009",
292+
"Submitter": "pku",
293+
"Availability": "available",
294+
"Category": "open",
295+
"Suite": "datacenter",
296+
"System": "mlperf_inference_lry_40",
297+
"Platform": "mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main",
298+
"UsedModel": "stable-diffusion-xl",
299+
"Model": "stable-diffusion-xl",
300+
"Scenario": "Offline",
301+
"Accuracy": "CLIP_SCORE: 14.028269715607166 FID_SCORE: 84.33061840229146",
302+
"Nodes": 1,
303+
"Processor": "AMD EPYC 9684X 96-Core Processor",
304+
"host_processors_per_node": 2,
305+
"host_processor_core_count": 96,
306+
"Accelerator": "NVIDIA H100 80GB HBM3",
307+
"a#": 5,
308+
"Location": "open/pku/results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline",
309+
"Software": "TensorRT",
310+
"operating_system": "Ubuntu 20.04 (linux-5.14.0-427.33.1.el9_4.x86_64-glibc2.31)",
311+
"Notes": "Automated by MLCommons CM v3.4.1. ",
312+
"compliance": 1,
313+
"errors": 0,
314+
"version": "v4.1",
315+
"inferred": 0,
316+
"has_power": false,
317+
"weight_data_types": "int8",
318+
"p#": 2,
319+
"Details": "https://github.com/mlcommons/cm4mlperf-inference/tree/mlperf-inference-results-scc24/open/pku/results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main",
320+
"Code": "https://github.com/mlcommons/cm4mlperf-inference/tree/mlperf-inference-results-scc24/open/pku/code",
321+
"Unique ID (e.g. for Audit)": "datacenter/open/pku/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl",
322+
"ColorKey": "availablepku",
323+
"Performance_Result": 8.2807,
324+
"Performance_Units": "Samples/s"
325+
},
326+
{
327+
"ID": "4.1-0010",
256328
"Submitter": "scc2",
257329
"Availability": "available",
258330
"Category": "open",

0 commit comments

Comments
 (0)