Inference v5.1.1

What's Changed

• Fix small llm readme by @pgmpablo157321 in #2220
• [DLRMv2] Updating DLRMv2 dataset size by @keithachorn-intel in #2222
• Remove incorrect automation commands by @anandhu-eng in #2218
• Fix Typo in Interactive Latencies by @mrmhodak in #2147
• Host speech2text artifacts in MLC storage bucket by @pgmpablo157321 in #2223
• Add missing interactive configurations by @pgmpablo157321 in #2224
• Update compliance table by @pgmpablo157321 in #2243
• Add deepseek dataset sources by @pgmpablo157321 in #2242
• Adding mlperf.conf setting for Whisper by @keithachorn-intel in #2238
• Fix Docs by @arjunsuresh in #2229
• Incorrect Regex for RougeLSum by @hvagadia in #2230
• update eval_accuracy.py and deepseek thresholds by @viraatc in #2233
• Add llama3.1-8b-edge as a separated benchmark by @pgmpablo157321 in #2231
• [Whisper] accuracy threshold by @wu6u3tw in #2259
• [Whisper] fix regex part by @wu6u3tw in #2260
• Update download path for llama3.1_8b dataset by @pgmpablo157321 in #2261
• Add FAQ addressing Whisper input padding. by @keithachorn-intel in #2255
• Update version generate_final_report.py by @pgmpablo157321 in #2269
• Add interactive scenario in the TEST06, bump loadgen version to 5.1 by @nvzhihanj in #2272
• Pinning vllm for speech-to-text reference by @keithachorn-intel in #2273
• Fix SingleStream llama3.1-8b typo by @pgmpablo157321 in #2274
• Update download path for DeepSeek-R1 Dataset by @pgmpablo157321 in #2275
• Update documentation by @anandhu-eng in #2279
• Address issue that logger.info not captured by stdout; remove redundant logging by @nvzhihanj in #2278
• Bugfix: Remove TEST01 for interactive scenario; add TEST06 for them by @nvzhihanj in #2281
• Fix list of workloads (LLMs) not requiring compliance TEST01 by @psyhtest in #2283
• Bugfix: Fix ds-r1 acc checker output format not captured by submission checker by @nvzhihanj in #2285
• Fixing 5.1 Submission Date by @mrmhodak in #2288
• fix: update ds-r1 truncate max-output-len to 20k (was 32k) by @viraatc in #2290
• Fix llama3.1-8b edge metrics and datasets by @pgmpablo157321 in #2300
• Add interactive scenario to final report to llama3.1 models by @pgmpablo157321 in #2299
• Allow more flexible datatypes in measurements file by @pgmpablo157321 in #2298
• Update evaluation.py by @taran2210 in #2303
• Only require Server or Interactive for closed by @pgmpablo157321 in #2304
• [Whisper] Adding n_token return for compliance fix by @keithachorn-intel in #2305
• Fix checking power directory by @anandhu-eng in #2306
• Only check for token latency requirements for server scenario by @pgmpablo157321 in #2313
• Use server SUT for SingleStream by @pgmpablo157321 in #2314
• Update the default value for repository arg by @anandhu-eng in #2317
• Update preprocess_submission.py | Skip inferring offline scenario if … by @arjunsuresh in #2316
• Fix: add llama3.1-8b-edge to generate_final_report by @pgmpablo157321 in #2319
• Allow lowercase 'interactive' as scenario name by @psyhtest in #2315
• Use sample latency as the metric for llama3.1_8b_edge SingleStream by @pgmpablo157321 in #2324
• Remove rclone references and update download instructions for DeepSeek-R1, Llama 3.1 8b, and Whisper by @anivar in #2289
• Hide long time untested implementations from docs by @anandhu-eng in #2328
• Initial draft for SCC 25 documentation by @anandhu-eng in #2331
• fix for fstring by @anandhu-eng in #2332
• Updation of automation run commands - v5.1_dev by @anandhu-eng in #2333
• Fixes for docs by @anandhu-eng in #2334
• Update submission_checker.py | Fixes #2325 by @arjunsuresh in #2326
• Fixes for scc doc by @anandhu-eng in #2339
• Update link to AMD readme for SCC by @anandhu-eng in #2340
• Add additional information for dataset and model downloads by @anandhu-eng in #2343
• Automation Docs: Provide correct syntax for NVIDIA batch size by @anandhu-eng in #2335
• Fix lookup of required accuracy delta by @psyhtest in #2337
• Llama3.1-405b: Add commands for model/dataset download using R2-downl… by @anandhu-eng in #2351
• Update commands to include R2-downloader for RGAT model download by @anandhu-eng in #2350
• Update minimum disk space required for PointPainting by @anandhu-eng in #2353
• Automation Docs: Add information about launching docker in privileged… by @anandhu-eng in #2352
• Remove Nvidia folder from compliance tree by @pgmpablo157321 in #2354
• Provide help - insufficient max locked memory error for Nvidia runs by @anandhu-eng in #2355
• fix: correct top-p and min-output-len for llama3.1-405b reference implementation by @viraatc in #2349
• Fix minor typos in reference_mlperf_perf.sh and reference_mlperf_accuracy.sh by @naveenmiriyaluredhat in #2327
• [LoadGen] Time to Output Token -> Time per Output Token by @wangshangsam in #2360
• [Whisper] Add labels' in the whisper output by @wu6u3tw in #2252
• Update gptj model download command to support R2 downloader by @anandhu-eng in #2368
• Update MLCFlow model and dataset download commands to support R2 downloader by @anandhu-eng in #2369
• Discard duplicate information about external model download by @anandhu-eng in #2370
• MIXTRAL - Update MLCFlow dataset and model download commands to suppo… by @anandhu-eng in #2371
• Update DLRM v2 assets download commands to support R2 downloader by @anandhu-eng in #2372
• SDXL: Update MLC commands to support model download through R2 by @anandhu-eng in #2373
• Updation for inference docs page: migration to R2 by @anandhu-eng in #2374
• Update MLC commands to support downloads through R2 by @anandhu-eng in #2367
• Replaced shell commands with Python for Windows compliance script compatibility by @sujik18 in #2344

New Contributors

• @hvagadia made their first contribution in #2230
• @wu6u3tw made their first contribution in #2259
• @taran2210 made their first contribution in #2303
• @anivar made their first contribution in #2289
• @naveenmiriyaluredhat made their first contribution in #2327
• @wangshangsam made their first contribution in #2360
• @sujik18 made their first contribution in #2344

Full Changelog: v5.1...v5.1.1