Skip to content

Commit c77698c

Browse files
gateoverflow-admingateoverflow-admin
committed
Updated results summary
1 parent 000872a commit c77698c

File tree

2 files changed

+107
-33
lines changed

2 files changed

+107
-33
lines changed

‎README.md

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
```
2-
[2024-10-12 06:45:29,981 submission_checker1.py:2936 INFO] Results=21, NoResults=0, Power Results=0
3-
[2024-10-12 06:45:29,981 submission_checker1.py:2943 INFO] ---
4-
[2024-10-12 06:45:29,981 submission_checker1.py:2944 INFO] Closed Results=0, Closed Power Results=0
2+
[2024-10-16 10:38:18,552 submission_checker1.py:2936 INFO] Results=23, NoResults=0, Power Results=0
3+
[2024-10-16 10:38:18,552 submission_checker1.py:2943 INFO] ---
4+
[2024-10-16 10:38:18,552 submission_checker1.py:2944 INFO] Closed Results=0, Closed Power Results=0
55
6-
[2024-10-12 06:45:29,981 submission_checker1.py:2949 INFO] Open Results=21, Open Power Results=0
6+
[2024-10-16 10:38:18,552 submission_checker1.py:2949 INFO] Open Results=23, Open Power Results=0
77
8-
[2024-10-12 06:45:29,981 submission_checker1.py:2954 INFO] Network Results=0, Network Power Results=0
8+
[2024-10-16 10:38:18,552 submission_checker1.py:2954 INFO] Network Results=0, Network Power Results=0
99
10-
[2024-10-12 06:45:29,981 submission_checker1.py:2959 INFO] ---
11-
[2024-10-12 06:45:29,981 submission_checker1.py:2961 INFO] Systems=20, Power Systems=0
12-
[2024-10-12 06:45:29,981 submission_checker1.py:2962 INFO] Closed Systems=0, Closed Power Systems=0
13-
[2024-10-12 06:45:29,981 submission_checker1.py:2967 INFO] Open Systems=20, Open Power Systems=0
14-
[2024-10-12 06:45:29,981 submission_checker1.py:2972 INFO] Network Systems=0, Network Power Systems=0
15-
[2024-10-12 06:45:29,981 submission_checker1.py:2977 INFO] ---
16-
[2024-10-12 06:45:29,981 submission_checker1.py:2982 INFO] SUMMARY: submission looks OK
10+
[2024-10-16 10:38:18,552 submission_checker1.py:2959 INFO] ---
11+
[2024-10-16 10:38:18,552 submission_checker1.py:2961 INFO] Systems=22, Power Systems=0
12+
[2024-10-16 10:38:18,552 submission_checker1.py:2962 INFO] Closed Systems=0, Closed Power Systems=0
13+
[2024-10-16 10:38:18,552 submission_checker1.py:2967 INFO] Open Systems=22, Open Power Systems=0
14+
[2024-10-16 10:38:18,552 submission_checker1.py:2972 INFO] Network Systems=0, Network Power Systems=0
15+
[2024-10-16 10:38:18,552 submission_checker1.py:2977 INFO] ---
16+
[2024-10-16 10:38:18,552 submission_checker1.py:2982 INFO] SUMMARY: submission looks OK
1717
INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4mlops/script/run-mlperf-inference-submission-checker/customize.py
1818
1919
```
@@ -40,4 +40,6 @@ INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4ml
4040
| 17 | UNM-Roadrunners | available | open | datacenter | d8727d7820f0 | d8727d7820f0-nvidia-gpu-TensorRT-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 1.61345 | CLIP_SCORE: 15.495350807905197 FID_SCORE: 233.70007081366145 | 1 | INTEL(R) XEON(R) PLATINUM 8580 | 2 | 60 | NVIDIA H100 NVL | 4 | open/UNM-Roadrunners/results/d8727d7820f0-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.37.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v2.4.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
4141
| 18 | UNM-Roadrunners | available | open | datacenter | 7b06989ff188 | 7b06989ff188-nvidia-gpu-TensorRT-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 1.61853 | CLIP_SCORE: 15.527581945061684 FID_SCORE: 239.84603575864136 | 1 | INTEL(R) XEON(R) PLATINUM 8580 | 2 | 60 | NVIDIA H100 NVL | 4 | open/UNM-Roadrunners/results/7b06989ff188-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.37.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v2.4.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
4242
| 19 | UNM-Roadrunners | available | open | datacenter | 4751fbf98487 | 4751fbf98487-reference-gpu-pytorch_v2.4.1-scc24-main | stable-diffusion-xl | stable-diffusion-xl | Offline | 0.665545 | CLIP_SCORE: 13.906862080842256 FID_SCORE: 84.21082273715388 | 1 | INTEL(R) XEON(R) PLATINUM 8580 | 2 | 60 | NVIDIA H100 NVL | 1 | open/UNM-Roadrunners/results/4751fbf98487-reference-gpu-pytorch_v2.4.1-scc24-main/stable-diffusion-xl/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-5.14.0-427.37.1.el9_4.x86_64-glibc2.35) | Automated by MLCommons CM v3.0.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
43-
| 20 | UNM-Roadrunners | available | open | datacenter | roadrunner | roadrunner-nvidia-gpu-TensorRT-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 1.60725 | CLIP_SCORE: 15.527581945061684 FID_SCORE: 239.84603550136694 | 1 | INTEL(R) XEON(R) PLATINUM 8580 | 2 | 60 | NVIDIA H100 NVL | 4 | open/UNM-Roadrunners/results/roadrunner-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.37.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v2.4.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
43+
| 20 | UNM-Roadrunners | available | open | datacenter | roadrunner | roadrunner-nvidia-gpu-TensorRT-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline | 1.60725 | CLIP_SCORE: 15.527581945061684 FID_SCORE: 239.84603550136694 | 1 | INTEL(R) XEON(R) PLATINUM 8580 | 2 | 60 | NVIDIA H100 NVL | 4 | open/UNM-Roadrunners/results/roadrunner-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline | TensorRT | Ubuntu 20.04 (linux-5.14.0-427.37.1.el9_4.x86_64-glibc2.31) | Automated by MLCommons CM v2.4.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | int8 |
44+
| 21 | Dreambrook_Team | available | open | edge | a65eca5827dc | a65eca5827dc-reference-cpu-pytorch_v2.4.1-default_config | bert-99 | bert-99 | Offline | 3.19276 | F1: 90.87487229720105 | 1 | AMD EPYC-Milan Processor | 32 | 1 | nan | 0 | open/Dreambrook_Team/results/a65eca5827dc-reference-cpu-pytorch_v2.4.1-default_config/bert-99/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.1.112-1.el9.elrepo.x86_64-glibc2.35) | Automated by MLCommons CM v3.2.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
45+
| 22 | Dreambrook_Team | available | open | edge | 666dfb9811e4 | 666dfb9811e4-reference-gpu-pytorch-cu124 | bert-99 | bert-99 | Offline | 85.4467 | F1: 90.87601905331469 | 1 | AMD EPYC-Milan Processor | 16 | 1 | GRID A100X-20C | 1 | open/Dreambrook_Team/results/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99/offline | pytorch | Ubuntu 22.04 (linux-6.1.112-1.el9.elrepo.x86_64-glibc2.35) | Automated by MLCommons CM v3.2.2. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |

‎summary_results.json

Lines changed: 92 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,78 @@
11
[
22
{
33
"ID": "4.1-0001",
4+
"Submitter": "Dreambrook_Team",
5+
"Availability": "available",
6+
"Category": "open",
7+
"Suite": "edge",
8+
"System": "666dfb9811e4",
9+
"Platform": "666dfb9811e4-reference-gpu-pytorch-cu124",
10+
"UsedModel": "bert-99",
11+
"Model": "bert-99",
12+
"Scenario": "Offline",
13+
"Accuracy": "F1: 90.87601905331469",
14+
"Nodes": 1,
15+
"Processor": "AMD EPYC-Milan Processor",
16+
"host_processors_per_node": 16,
17+
"host_processor_core_count": 1,
18+
"Accelerator": "GRID A100X-20C",
19+
"a#": 1,
20+
"Location": "open/Dreambrook_Team/results/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99/offline",
21+
"Software": "pytorch",
22+
"operating_system": "Ubuntu 22.04 (linux-6.1.112-1.el9.elrepo.x86_64-glibc2.35)",
23+
"Notes": "Automated by MLCommons CM v3.2.2. ",
24+
"compliance": 1,
25+
"errors": 0,
26+
"version": "v4.1",
27+
"inferred": 0,
28+
"has_power": false,
29+
"weight_data_types": "fp32",
30+
"p#": 16,
31+
"Details": "https://github.com/mlcommons/submissions_inference_4.0/tree/main/open/Dreambrook_Team/results/666dfb9811e4-reference-gpu-pytorch-cu124",
32+
"Code": "https://github.com/mlcommons/submissions_inference_4.0/tree/main/open/Dreambrook_Team/code",
33+
"Unique ID (e.g. for Audit)": "edge/open/Dreambrook_Team/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99",
34+
"ColorKey": "availableDreambrook_Team",
35+
"Performance_Result": 85.4467,
36+
"Performance_Units": "Samples/s"
37+
},
38+
{
39+
"ID": "4.1-0002",
40+
"Submitter": "Dreambrook_Team",
41+
"Availability": "available",
42+
"Category": "open",
43+
"Suite": "edge",
44+
"System": "a65eca5827dc",
45+
"Platform": "a65eca5827dc-reference-cpu-pytorch_v2.4.1-default_config",
46+
"UsedModel": "bert-99",
47+
"Model": "bert-99",
48+
"Scenario": "Offline",
49+
"Accuracy": "F1: 90.87487229720105",
50+
"Nodes": 1,
51+
"Processor": "AMD EPYC-Milan Processor",
52+
"host_processors_per_node": 32,
53+
"host_processor_core_count": 1,
54+
"Accelerator": "",
55+
"a#": "",
56+
"Location": "open/Dreambrook_Team/results/a65eca5827dc-reference-cpu-pytorch_v2.4.1-default_config/bert-99/offline",
57+
"Software": "pytorch v2.4.1",
58+
"operating_system": "Ubuntu 22.04 (linux-6.1.112-1.el9.elrepo.x86_64-glibc2.35)",
59+
"Notes": "Automated by MLCommons CM v3.2.1. ",
60+
"compliance": 1,
61+
"errors": 0,
62+
"version": "v4.1",
63+
"inferred": 0,
64+
"has_power": false,
65+
"weight_data_types": "fp32",
66+
"p#": 32,
67+
"Details": "https://github.com/mlcommons/submissions_inference_4.0/tree/main/open/Dreambrook_Team/results/a65eca5827dc-reference-cpu-pytorch_v2.4.1-default_config",
68+
"Code": "https://github.com/mlcommons/submissions_inference_4.0/tree/main/open/Dreambrook_Team/code",
69+
"Unique ID (e.g. for Audit)": "edge/open/Dreambrook_Team/a65eca5827dc-reference-cpu-pytorch_v2.4.1-default_config/bert-99",
70+
"ColorKey": "availableDreambrook_Team",
71+
"Performance_Result": 3.19276,
72+
"Performance_Units": "Samples/s"
73+
},
74+
{
75+
"ID": "4.1-0003",
476
"Submitter": "MLCommons",
577
"Availability": "available",
678
"Category": "open",
@@ -36,7 +108,7 @@
36108
"Performance_Units": "Samples/s"
37109
},
38110
{
39-
"ID": "4.1-0002",
111+
"ID": "4.1-0004",
40112
"Submitter": "MLCommons",
41113
"Availability": "available",
42114
"Category": "open",
@@ -72,7 +144,7 @@
72144
"Performance_Units": "Samples/s"
73145
},
74146
{
75-
"ID": "4.1-0003",
147+
"ID": "4.1-0005",
76148
"Submitter": "MLCommons",
77149
"Availability": "available",
78150
"Category": "open",
@@ -108,7 +180,7 @@
108180
"Performance_Units": "Samples/s"
109181
},
110182
{
111-
"ID": "4.1-0004",
183+
"ID": "4.1-0006",
112184
"Submitter": "MLCommons",
113185
"Availability": "available",
114186
"Category": "open",
@@ -144,7 +216,7 @@
144216
"Performance_Units": "Samples/s"
145217
},
146218
{
147-
"ID": "4.1-0005",
219+
"ID": "4.1-0007",
148220
"Submitter": "MLCommons",
149221
"Availability": "available",
150222
"Category": "open",
@@ -180,7 +252,7 @@
180252
"Performance_Units": "Samples/s"
181253
},
182254
{
183-
"ID": "4.1-0006",
255+
"ID": "4.1-0008",
184256
"Submitter": "MLCommons",
185257
"Availability": "available",
186258
"Category": "open",
@@ -216,7 +288,7 @@
216288
"Performance_Units": "Samples/s"
217289
},
218290
{
219-
"ID": "4.1-0007",
291+
"ID": "4.1-0009",
220292
"Submitter": "MLCommons",
221293
"Availability": "available",
222294
"Category": "open",
@@ -252,7 +324,7 @@
252324
"Performance_Units": "Samples/s"
253325
},
254326
{
255-
"ID": "4.1-0008",
327+
"ID": "4.1-0010",
256328
"Submitter": "MLCommons",
257329
"Availability": "available",
258330
"Category": "open",
@@ -288,7 +360,7 @@
288360
"Performance_Units": "Samples/s"
289361
},
290362
{
291-
"ID": "4.1-0009",
363+
"ID": "4.1-0011",
292364
"Submitter": "MLCommons",
293365
"Availability": "available",
294366
"Category": "open",
@@ -324,7 +396,7 @@
324396
"Performance_Units": "Samples/s"
325397
},
326398
{
327-
"ID": "4.1-0010",
399+
"ID": "4.1-0012",
328400
"Submitter": "MLCommons",
329401
"Availability": "available",
330402
"Category": "open",
@@ -360,7 +432,7 @@
360432
"Performance_Units": "Samples/s"
361433
},
362434
{
363-
"ID": "4.1-0011",
435+
"ID": "4.1-0013",
364436
"Submitter": "MLCommons",
365437
"Availability": "available",
366438
"Category": "open",
@@ -396,7 +468,7 @@
396468
"Performance_Units": "Samples/s"
397469
},
398470
{
399-
"ID": "4.1-0012",
471+
"ID": "4.1-0014",
400472
"Submitter": "MLCommons",
401473
"Availability": "available",
402474
"Category": "open",
@@ -432,7 +504,7 @@
432504
"Performance_Units": "Tokens/s"
433505
},
434506
{
435-
"ID": "4.1-0013",
507+
"ID": "4.1-0015",
436508
"Submitter": "MLCommons",
437509
"Availability": "available",
438510
"Category": "open",
@@ -468,7 +540,7 @@
468540
"Performance_Units": "Samples/s"
469541
},
470542
{
471-
"ID": "4.1-0014",
543+
"ID": "4.1-0016",
472544
"Submitter": "UNM-Roadrunners",
473545
"Availability": "available",
474546
"Category": "open",
@@ -504,7 +576,7 @@
504576
"Performance_Units": "Samples/s"
505577
},
506578
{
507-
"ID": "4.1-0015",
579+
"ID": "4.1-0017",
508580
"Submitter": "UNM-Roadrunners",
509581
"Availability": "available",
510582
"Category": "open",
@@ -540,7 +612,7 @@
540612
"Performance_Units": "Samples/s"
541613
},
542614
{
543-
"ID": "4.1-0016",
615+
"ID": "4.1-0018",
544616
"Submitter": "UNM-Roadrunners",
545617
"Availability": "available",
546618
"Category": "open",
@@ -576,7 +648,7 @@
576648
"Performance_Units": "Samples/s"
577649
},
578650
{
579-
"ID": "4.1-0017",
651+
"ID": "4.1-0019",
580652
"Submitter": "UNM-Roadrunners",
581653
"Availability": "available",
582654
"Category": "open",
@@ -612,7 +684,7 @@
612684
"Performance_Units": "Samples/s"
613685
},
614686
{
615-
"ID": "4.1-0018",
687+
"ID": "4.1-0020",
616688
"Submitter": "UNM-Roadrunners",
617689
"Availability": "available",
618690
"Category": "open",
@@ -648,7 +720,7 @@
648720
"Performance_Units": "Samples/s"
649721
},
650722
{
651-
"ID": "4.1-0019",
723+
"ID": "4.1-0021",
652724
"Submitter": "UNM-Roadrunners",
653725
"Availability": "available",
654726
"Category": "open",
@@ -684,7 +756,7 @@
684756
"Performance_Units": "Samples/s"
685757
},
686758
{
687-
"ID": "4.1-0020",
759+
"ID": "4.1-0022",
688760
"Submitter": "UNM-Roadrunners",
689761
"Availability": "available",
690762
"Category": "open",
@@ -720,7 +792,7 @@
720792
"Performance_Units": "Samples/s"
721793
},
722794
{
723-
"ID": "4.1-0021",
795+
"ID": "4.1-0023",
724796
"Submitter": "UNM-Roadrunners",
725797
"Availability": "available",
726798
"Category": "open",

0 commit comments

Comments
 (0)