mlcommons · arjunsuresh · Nov 18, 2024 · Nov 18, 2024
@@ -0,0 +1,3 @@
+| Model               | Scenario   | Accuracy             |   Throughput | Latency (in ms)   |
+|---------------------|------------|----------------------|--------------|-------------------|
+| stable-diffusion-xl | offline    | (15.22477, 84.24318) |        0.848 | -                 |
@@ -0,0 +1,59 @@
+This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).
+
+*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*
+
+## Host platform
+
+* OS version: Linux-5.14.0-427.42.1.el9_4.x86_64-x86_64-with-glibc2.35
+* CPU version: x86_64
+* Python version: 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0]
+* MLCommons CM version: 3.4.1
+
+## CM Run Command
+
+See [CM installation guide](https://docs.mlcommons.org/inference/install/).
+
+```bash
+pip install -U cmind
+
+cm rm cache -f
+
+cm pull repo mlcommons@cm4mlops --checkout=b32ded2a4c3039ad16dadc734bee03dd1a97f228
+
+cm run script \
+	--tags=run-mlperf,inference,_r4.1-dev,_scc24-main \
+	--model=sdxl \
+	--framework=pytorch \
+	--category=datacenter \
+	--scenario=Offline \
+	--execution_mode=test \
+	--device=rocm \
+	--quiet \
+	--precision=float16 \
+	--adr.mlperf-implementation.tags=_branch.multinode-test,_repo.https://github.com/zixianwang2022/mlperf-scc24 \
+	--adr.mlperf-implementation.version=custom \
+	--env.CM_GET_PLATFORM_DETAILS=no \
+	--target_qps=1.8
+```
+*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
+ you should simply reload mlcommons@cm4mlops without checkout and clean CM cache as follows:*
+
+```bash
+cm rm repo mlcommons@cm4mlops
+cm pull repo mlcommons@cm4mlops
+cm rm cache -f
+
+```
+
+## Results
+
+Platform: aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main
+
+Model Precision: fp32
+
+### Accuracy Results 
+`CLIP_SCORE`: `15.22477`, Required accuracy for closed division `>= 31.68632` and `<= 31.81332`
+`FID_SCORE`: `84.24318`, Required accuracy for closed division `>= 23.01086` and `<= 23.95008`
+
+### Performance Results 
+`Samples per second`: `0.847525`
@@ -0,0 +1,12 @@
+INFO:main:Namespace(sut_server=['http://10.0.0.14:8008', 'http://10.0.0.12:8008'], dataset='coco-1024', dataset_path='/root/CM/repos/local/cache/61dd835801c542a3/install', profile='stable-diffusion-xl-pytorch', scenario='Offline', max_batchsize=1, threads=1, accuracy=True, find_peak_performance=False, backend='pytorch', model_name='stable-diffusion-xl', output='/root/CM/repos/local/cache/d549713c4a534705/test_results/aqua-reference-rocm-pytorch-v2.6.0.dev20241118-scc24-main/stable-diffusion-xl/offline/accuracy', qps=None, model_path='/root/CM/repos/local/cache/c4b6bbbebe504f28/stable_diffusion_fp16', dtype='fp16', device='cuda', latent_framework='torch', mlperf_conf='mlperf.conf', user_conf='/root/CM/repos/mlcommons@cm4mlops/script/generate-mlperf-inference-user-conf/tmp/6626c9658bff4d2291e3121038a4cfca.conf', audit_conf='audit.config', ids_path='/root/CM/repos/local/cache/61dd835801c542a3/install/sample_ids.txt', time=None, count=10, debug=False, performance_sample_count=5000, max_latency=None, samples_per_query=8)
+WARNING:backend-pytorch:Model path not provided, running with default hugging face weights
+This may not be valid for official submissions
+Keyword arguments {'safety_checker': None} are not expected by StableDiffusionXLPipeline and will be ignored.
+Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]Using the `SDPA` attention implementation on multi-gpu setup with ROCM may lead to performance issues due to the FA backend. Disabling it to use alternative backends.
+Loading pipeline components...:  57%|█████▋    | 4/7 [00:00<00:00, 12.44it/s]Loading pipeline components...:  86%|████████▌ | 6/7 [00:00<00:00,  7.56it/s]Loading pipeline components...: 100%|██████████| 7/7 [00:00<00:00,  9.15it/s]
+RETURNED from requests.post on predict at time 	 1731969667.0400865
+BEFORE lg.QuerySamplesComplete(response)
+AFTER lg.QuerySamplesComplete(response)
+RETURNED from requests.post on predict at time 	 1731969689.698808
+BEFORE lg.QuerySamplesComplete(response)
+AFTER lg.QuerySamplesComplete(response)

@@ -0,0 +1,7 @@
+{
+  "starting_weights_filename": "https://github.com/mlcommons/inference/tree/master/text_to_image#download-model",
+  "retraining": "no",
+  "input_data_types": "fp32",
+  "weight_data_types": "fp32",
+  "weight_transformations": "no"
+}