Documentation updates (mlcommons#1821)

* Support batch-size in llama2 run * Add Rclone-Cloudflare download instructions to README.md * Add Rclone-Cloudflare download instructiosn to README.md * Minor wording edit to README.md * Add Rclone-Cloudflare download instructions to README.md * Add Rclone-GDrive download instructions to README.md * Add new and old instructions to README.md * Tweak language in README.md * Language tweak in README.md * Minor language tweak in README.md * Fix typo in README.md * Count error when logging errors: submission_checker.py * Fixes mlcommons#1648, restrict loadgen uncommitted error message to within the loadgen directory * Update test-rnnt.yml (mlcommons#1688) Stopping the github action for rnnt * Added docs init Added github action for website publish Update benchmark documentation Update publish.yaml Update publish.yaml Update benchmark documentation Improved the submission documentation Fix taskname Removed unused images * Fix benchmark URLs * Fix links * Add _full variation to run commands * Added script flow diagram * Added docker setup command for CM, extra run options * Added support for docker options in the docs * Added --quiet to the CM run_cmds in docs * Fix the test query count for cm commands * Support ctuning-cpp implementation * Added commands for mobilenet models * Docs cleanup * Docs cleanup * Added separate files for dataset and models in the docs * Remove redundant tab in the docs * Fixes some WIP models in the docs * Use the official docs page for CM installation * Fix the deadlink in docs * Fix indendation issue in docs * Added dockerinfo for nvidia implementation * Added run options for gptj * Added execution environment tabs * Cleanup of the docs * Cleanup of the docs * Reordered the sections of the docs page * Removed an unnecessary heading in the docs * Fixes the commands for datacenter * Fix the build --sdist for loadgen * Fixes mlcommons#1761, llama2 and mixtral runtime error on CPU systems * Added mixtral to the benchmark list, improved benchmark docs * Update docs for MLPerf inference v4.1 * Update docs for MLPerf inference v4.1 * Fix typo * Gave direct link to implementation readmes * Added tables detailing implementations * Update vision README.md, split the frameworks into separate rows * Update README.md * pointed links to specific frameworks * pointed links to specific frameworks * Update Submission_Guidelines.md * Update Submission_Guidelines.md * Update Submission_Guidelines.md * api support llama2 * Added request module and reduced max token len * Fix for llama2 api server * Update SUT_API offline to work for OpenAI * Update SUT_API.py * Minor fixes * Fix json import in SUT_API.py * Fix llama2 token length * Added model name verification with server * clean temp files * support num_workers in LLAMA2 SUTs * Remove batching from Offline SUT_API.py * Update SUT_API.py * Minor fixes for llama2 API * Fix for llama2 API * removed table of contents * enabled llama2-nvidia + vllm-NM : WIP * enabled dlrm for intel * lower cased implementation * added raw data input * corrected data download commands * renamed filename * changes for bert and vllm * documentation to work on custom repo and branch * benchmark index page update * enabled sdxl for nvidia and intel * updated vllm server run cmd * benchmark page information addition * fix indendation issue * Added submission categories * update submission page - generate submission with or w/o using CM for benchmarking * Updated kits dataset documentation * Updated model parameters * updation of information * updated non cm based benchmark * added info about hf password * added links to model and access tokens * Updated reference results structuree tree * submission docs cleanup * Some cleanups for benchmark info * Some cleanups for benchmark info * Some cleanups for benchmark info * added generic stubs deepsparse * Some cleanups for benchmark info * Some cleanups for benchmark info * Some cleanups for benchmark info * Some cleanups for benchmark info (FID and CLIP data added) * typo fix for bert deepsparse framework * added min system requirements for models * fixed code version * changes for displaying reference and intel implementation tip * added reference to installation page * updated neural magic documentation --------- Co-authored-by: Nathan Wasson <[email protected]> Co-authored-by: anandhu-eng <[email protected]> Co-authored-by: ANANDHU S <[email protected]> Co-authored-by: Michael Goin <[email protected]>
GATEOverflow · Sep 3, 2024 · 81c2de6 · 81c2de6
1 parent 73b0279
commit 81c2de6
Show file tree

Hide file tree

Showing 31 changed files with 1,093 additions and 150 deletions.
diff --git a/docs/benchmarks/image_classification/get-resnet50-data.md b/docs/benchmarks/image_classification/get-resnet50-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Image Classification using ResNet50 
 
 ## Dataset

diff --git a/docs/benchmarks/image_classification/mobilenets.md b/docs/benchmarks/image_classification/mobilenets.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Image Classification using Mobilenet models
 
 Mobilenet models are not official MLPerf models and so cannot be used for a Closed division MLPerf inference submission. But since they can be run with Imagenet dataset, we are allowed to use them for Open division submission. Only CPU runs are supported now. 

diff --git a/docs/benchmarks/image_classification/resnet50.md b/docs/benchmarks/image_classification/resnet50.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Image Classification using ResNet50 
 
 === "MLCommons-Python"

diff --git a/docs/benchmarks/index.md b/docs/benchmarks/index.md
diff --git a/docs/benchmarks/language/bert.md b/docs/benchmarks/language/bert.md
@@ -1,36 +1,34 @@
+---
+hide:
+  - toc
+---
+
 # Question Answering using Bert-Large
 
 === "MLCommons-Python"
     ## MLPerf Reference Implementation in Python
 
-    BERT-99
 {{ mlperf_inference_implementation_readme (4, "bert-99", "reference") }}
 
-    BERT-99.9
 {{ mlperf_inference_implementation_readme (4, "bert-99.9", "reference") }}
 
 === "Nvidia"
     ## Nvidia MLPerf Implementation
 
-    BERT-99
 {{ mlperf_inference_implementation_readme (4, "bert-99", "nvidia") }}
 
-    BERT-99.9
 {{ mlperf_inference_implementation_readme (4, "bert-99.9", "nvidia") }}
 
 === "Intel"
     ## Intel MLPerf Implementation
-    BERT-99
+
 {{ mlperf_inference_implementation_readme (4, "bert-99", "intel") }}
 
-    BERT-99.9
 {{ mlperf_inference_implementation_readme (4, "bert-99.9", "intel") }}
 
 === "Qualcomm"
     ## Qualcomm AI100 MLPerf Implementation
 
-    BERT-99
 {{ mlperf_inference_implementation_readme (4, "bert-99", "qualcomm") }}
 
-    BERT-99.9
 {{ mlperf_inference_implementation_readme (4, "bert-99.9", "qualcomm") }}
diff --git a/docs/benchmarks/language/get-bert-data.md b/docs/benchmarks/language/get-bert-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Question Answering using Bert-Large
 
 ## Dataset

diff --git a/docs/benchmarks/language/get-gptj-data.md b/docs/benchmarks/language/get-gptj-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Text Summarization using GPT-J
 
 ## Dataset

diff --git a/docs/benchmarks/language/get-llama2-70b-data.md b/docs/benchmarks/language/get-llama2-70b-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Text Summarization using LLAMA2-70b
 
 ## Dataset
@@ -23,4 +28,8 @@ Get the Official MLPerf LLAMA2-70b Model
     ```
     cm run script --tags=get,ml-model,llama2-70b,_pytorch -j
     ```
+
+!!! tip
+
+    Downloading llama2-70B model from Hugging Face will prompt you to enter the Hugging Face username and password. Please note that the password required is the [**access token**](https://huggingface.co/settings/tokens) generated for your account. Additionally, ensure that your account has access to the [llama2-70B](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) model.
 
diff --git a/docs/benchmarks/language/get-mixtral-8x7b-data.md b/docs/benchmarks/language/get-mixtral-8x7b-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 ## Dataset
 
 The benchmark implementation run command will automatically download the preprocessed validation and calibration datasets. In case you want to download only the datasets, you can use the below commands.

diff --git a/docs/benchmarks/language/gpt-j.md b/docs/benchmarks/language/gpt-j.md
@@ -1,39 +1,38 @@
+---
+hide:
+  - toc
+---
+
 # Text Summarization using GPT-J
 
 
 === "MLCommons-Python"
     ## MLPerf Reference Implementation in Python
-
-    GPT-J-99
+
 
 {{ mlperf_inference_implementation_readme (4, "gptj-99", "reference") }}
 
-    GPTJ-99.9
 
 {{ mlperf_inference_implementation_readme (4, "gptj-99.9", "reference") }}
 
 === "Nvidia"
     ## Nvidia MLPerf Implementation
 
-    GPTJ-99
 
 {{ mlperf_inference_implementation_readme (4, "gptj-99", "nvidia") }}
 
-    GPTJ-99.9
 
 {{ mlperf_inference_implementation_readme (4, "gptj-99.9", "nvidia") }}
 
 === "Intel"
     ## Intel MLPerf Implementation
-    GPTJ-99
 
 {{ mlperf_inference_implementation_readme (4, "gptj-99", "intel") }}
 
 
 === "Qualcomm"
     ## Qualcomm AI100 MLPerf Implementation
 
-    GPTJ-99
 
 {{ mlperf_inference_implementation_readme (4, "gptj-99", "qualcomm") }}
 
diff --git a/docs/benchmarks/language/llama2-70b.md b/docs/benchmarks/language/llama2-70b.md
@@ -1,28 +1,28 @@
+---
+hide:
+  - toc
+---
+
 # Text Summarization using LLAMA2-70b
 
 
 === "MLCommons-Python"
     ## MLPerf Reference Implementation in Python
 
-    LLAMA2-70b-99
 {{ mlperf_inference_implementation_readme (4, "llama2-70b-99", "reference") }}
 
-    LLAMA2-70b-99.9
 {{ mlperf_inference_implementation_readme (4, "llama2-70b-99.9", "reference") }}
 
 === "Nvidia"
     ## Nvidia MLPerf Implementation
 
-    LLAMA2-70b-99
 {{ mlperf_inference_implementation_readme (4, "llama2-70b-99", "nvidia") }}
 
-    LLAMA2-70b-99.9
 {{ mlperf_inference_implementation_readme (4, "llama2-70b-99.9", "nvidia") }}
 
+=== "Neural Magic"
+    ## Neural Magic MLPerf Implementation
+
+{{ mlperf_inference_implementation_readme (4, "llama2-70b-99", "neuralmagic") }}
 
-=== "Qualcomm"
-    ## Qualcomm AI100 MLPerf Implementation
-
-    LLAMA2-70b-99
-{{ mlperf_inference_implementation_readme (4, "llama2-70b-99", "qualcomm") }}
-
+{{ mlperf_inference_implementation_readme (4, "llama2-70b-99.9", "neuralmagic") }}
diff --git a/docs/benchmarks/language/mixtral-8x7b.md b/docs/benchmarks/language/mixtral-8x7b.md
@@ -1,6 +1,9 @@
+---
+hide:
+  - toc
+---
 
 === "MLCommons-Python"
     ## MLPerf Reference Implementation in Python
 
-    MIXTRAL-8x7b
 {{ mlperf_inference_implementation_readme (4, "mixtral-8x7b", "reference") }}
diff --git a/docs/benchmarks/medical_imaging/3d-unet.md b/docs/benchmarks/medical_imaging/3d-unet.md
@@ -1,33 +1,32 @@
+---
+hide:
+  - toc
+---
+
 # Medical Imaging using 3d-unet (KiTS 2019 kidney tumor segmentation task)
 
 
 === "MLCommons-Python"
     ## MLPerf Reference Implementation in Python
 
-    3d-unet-99
 
 {{ mlperf_inference_implementation_readme (4, "3d-unet-99", "reference") }}
 
-    3d-unet-99.9
 
 {{ mlperf_inference_implementation_readme (4, "3d-unet-99.9", "reference") }}
 
 === "Nvidia"
     ## Nvidia MLPerf Implementation
-    3d-unet-99
 
 {{ mlperf_inference_implementation_readme (4, "3d-unet-99", "nvidia") }}
 
-    3d-unet-99.9
 
 {{ mlperf_inference_implementation_readme (4, "3d-unet-99.9", "nvidia") }}
 
 === "Intel"
     ## Intel MLPerf Implementation
-    3d-unet-99
 
 {{ mlperf_inference_implementation_readme (4, "3d-unet-99", "intel") }}
 
-    3d-unet-99.9
 
 {{ mlperf_inference_implementation_readme (4, "3d-unet-99.9", "intel") }}
diff --git a/docs/benchmarks/medical_imaging/get-3d-unet-data.md b/docs/benchmarks/medical_imaging/get-3d-unet-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Medical Imaging using 3d-unet (KiTS 2019 kidney tumor segmentation task)
 
 ## Dataset
@@ -7,9 +12,14 @@ The benchmark implementation run command will automatically download the validat
 === "Validation"
     3d-unet validation run uses the KiTS19 dataset performing [KiTS 2019](https://kits19.grand-challenge.org/) kidney tumor segmentation task
 
-    ### Get Validation Dataset
+    ### Get Validation Dataset(Original)
+    ```
+    cm run script --tags=get,dataset,kits19,_validation -j
+    ```
+
+    ### Get Validation Dataset(Preprocessed)
     ```
-    cm run script --tags=get,dataset,kits19,validation -j
+    cm run script --tags=get,dataset,kits19,preprocessed -j
     ```
 
 ## Model

diff --git a/docs/benchmarks/object_detection/get-retinanet-data.md b/docs/benchmarks/object_detection/get-retinanet-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Object Detection using Retinanet
 
 ## Dataset

diff --git a/docs/benchmarks/object_detection/retinanet.md b/docs/benchmarks/object_detection/retinanet.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Object Detection using Retinanet
 
 === "MLCommons-Python"

diff --git a/docs/benchmarks/recommendation/dlrm-v2.md b/docs/benchmarks/recommendation/dlrm-v2.md
@@ -1,22 +1,29 @@
+---
+hide:
+  - toc
+---
+
 # Recommendation using DLRM v2
 
 
 ## Benchmark Implementations
 === "MLCommons-Python"
     ## MLPerf Reference Implementation in Python
 
-    DLRM-v2-99
-{{ mlperf_inference_implementation_readme (4, "dlrm_v2-99", "reference") }}
+{{ mlperf_inference_implementation_readme (4, "dlrm-v2-99", "reference") }}
 
-    DLRM-v2-99.9
-{{ mlperf_inference_implementation_readme (4, "dlrm_v2-99.9", "reference") }}
+{{ mlperf_inference_implementation_readme (4, "dlrm-v2-99.9", "reference") }}
 
 === "Nvidia"
     ## Nvidia MLPerf Implementation
-
-    DLRM-v2-99
-{{ mlperf_inference_implementation_readme (4, "dlrm_v2-99", "nvidia") }}
 
-    DLRM-v2-99.9
-{{ mlperf_inference_implementation_readme (4, "dlrm_v2-99.9", "nvidia") }}
+{{ mlperf_inference_implementation_readme (4, "dlrm-v2-99", "nvidia") }}
+
+{{ mlperf_inference_implementation_readme (4, "dlrm-v2-99.9", "nvidia") }}
+
+=== "Intel"
+    ## Intel MLPerf Implementation
+
+{{ mlperf_inference_implementation_readme (4, "dlrm-v2-99", "intel") }}
 
+{{ mlperf_inference_implementation_readme (4, "dlrm-v2-99.9", "intel") }}
diff --git a/...hmarks/recommendation/get-dlrm_v2-data.md → ...hmarks/recommendation/get-dlrm-v2-data.md b/...hmarks/recommendation/get-dlrm_v2-data.md → ...hmarks/recommendation/get-dlrm-v2-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Recommendation using DLRM v2
 
 ## Dataset
@@ -9,7 +14,7 @@ The benchmark implementation run command will automatically download the validat
 
     ### Get Validation Dataset
     ```
-    cm run script --tags=get,dataset,criteo,validation -j
+    cm run script --tags=get,dataset,criteo,_validation -j
     ```
 ## Model
 The benchmark implementation run command will automatically download the required model and do the necessary conversions. In case you want to only download the official model, you can use the below commands.
@@ -20,6 +25,6 @@ Get the Official MLPerf DLRM v2 Model
 
     ### Pytorch
     ```
-    cm run script --tags=get,ml-model,dlrm_v2,_pytorch -j
+    cm run script --tags=get,ml-model,dlrm,_pytorch -j
     ```
 
diff --git a/docs/benchmarks/text_to_image/get-sdxl-data.md b/docs/benchmarks/text_to_image/get-sdxl-data.md
@@ -1,3 +1,8 @@
+---
+hide:
+  - toc
+---
+
 # Text to Image using Stable Diffusion
 
 ## Dataset