What's Changed
- Fix small llm readme by @pgmpablo157321 in #2220
- [DLRMv2] Updating DLRMv2 dataset size by @keithachorn-intel in #2222
- Remove incorrect automation commands by @anandhu-eng in #2218
- Fix Typo in Interactive Latencies by @mrmhodak in #2147
- Host speech2text artifacts in MLC storage bucket by @pgmpablo157321 in #2223
- Add missing interactive configurations by @pgmpablo157321 in #2224
- Update compliance table by @pgmpablo157321 in #2243
- Add deepseek dataset sources by @pgmpablo157321 in #2242
- Adding mlperf.conf setting for Whisper by @keithachorn-intel in #2238
- Fix Docs by @arjunsuresh in #2229
- Incorrect Regex for RougeLSum by @hvagadia in #2230
- update eval_accuracy.py and deepseek thresholds by @viraatc in #2233
- Add llama3.1-8b-edge as a separated benchmark by @pgmpablo157321 in #2231
- [Whisper] accuracy threshold by @wu6u3tw in #2259
- [Whisper] fix regex part by @wu6u3tw in #2260
- Update download path for llama3.1_8b dataset by @pgmpablo157321 in #2261
- Add FAQ addressing Whisper input padding. by @keithachorn-intel in #2255
- Update version generate_final_report.py by @pgmpablo157321 in #2269
- Add interactive scenario in the TEST06, bump loadgen version to 5.1 by @nvzhihanj in #2272
- Pinning vllm for speech-to-text reference by @keithachorn-intel in #2273
- Fix SingleStream llama3.1-8b typo by @pgmpablo157321 in #2274
- Update download path for DeepSeek-R1 Dataset by @pgmpablo157321 in #2275
- Update documentation by @anandhu-eng in #2279
- Address issue that logger.info not captured by stdout; remove redundant logging by @nvzhihanj in #2278
- Bugfix: Remove TEST01 for interactive scenario; add TEST06 for them by @nvzhihanj in #2281
- Fix list of workloads (LLMs) not requiring compliance TEST01 by @psyhtest in #2283
- Bugfix: Fix ds-r1 acc checker output format not captured by submission checker by @nvzhihanj in #2285
- Fixing 5.1 Submission Date by @mrmhodak in #2288
- fix: update ds-r1 truncate max-output-len to 20k (was 32k) by @viraatc in #2290
- Fix llama3.1-8b edge metrics and datasets by @pgmpablo157321 in #2300
- Add interactive scenario to final report to llama3.1 models by @pgmpablo157321 in #2299
- Allow more flexible datatypes in measurements file by @pgmpablo157321 in #2298
- Update evaluation.py by @taran2210 in #2303
- Only require Server or Interactive for closed by @pgmpablo157321 in #2304
- [Whisper] Adding n_token return for compliance fix by @keithachorn-intel in #2305
- Fix checking power directory by @anandhu-eng in #2306
- Only check for token latency requirements for server scenario by @pgmpablo157321 in #2313
- Use server SUT for SingleStream by @pgmpablo157321 in #2314
- Update the default value for repository arg by @anandhu-eng in #2317
- Update preprocess_submission.py | Skip inferring offline scenario if … by @arjunsuresh in #2316
- Fix: add llama3.1-8b-edge to generate_final_report by @pgmpablo157321 in #2319
- Allow lowercase 'interactive' as scenario name by @psyhtest in #2315
- Use sample latency as the metric for llama3.1_8b_edge SingleStream by @pgmpablo157321 in #2324
- Remove rclone references and update download instructions for DeepSeek-R1, Llama 3.1 8b, and Whisper by @anivar in #2289
- Hide long time untested implementations from docs by @anandhu-eng in #2328
- Initial draft for SCC 25 documentation by @anandhu-eng in #2331
- fix for fstring by @anandhu-eng in #2332
- Updation of automation run commands - v5.1_dev by @anandhu-eng in #2333
- Fixes for docs by @anandhu-eng in #2334
- Update submission_checker.py | Fixes #2325 by @arjunsuresh in #2326
- Fixes for scc doc by @anandhu-eng in #2339
- Update link to AMD readme for SCC by @anandhu-eng in #2340
- Add additional information for dataset and model downloads by @anandhu-eng in #2343
- Automation Docs: Provide correct syntax for NVIDIA batch size by @anandhu-eng in #2335
- Fix lookup of required accuracy delta by @psyhtest in #2337
- Llama3.1-405b: Add commands for model/dataset download using R2-downl… by @anandhu-eng in #2351
- Update commands to include R2-downloader for RGAT model download by @anandhu-eng in #2350
- Update minimum disk space required for PointPainting by @anandhu-eng in #2353
- Automation Docs: Add information about launching docker in privileged… by @anandhu-eng in #2352
- Remove Nvidia folder from compliance tree by @pgmpablo157321 in #2354
- Provide help - insufficient max locked memory error for Nvidia runs by @anandhu-eng in #2355
- fix: correct top-p and min-output-len for llama3.1-405b reference implementation by @viraatc in #2349
- Fix minor typos in reference_mlperf_perf.sh and reference_mlperf_accuracy.sh by @naveenmiriyaluredhat in #2327
- [LoadGen]
Time to Output Token->Time per Output Tokenby @wangshangsam in #2360 - [Whisper] Add labels' in the whisper output by @wu6u3tw in #2252
- Update gptj model download command to support R2 downloader by @anandhu-eng in #2368
- Update MLCFlow model and dataset download commands to support R2 downloader by @anandhu-eng in #2369
- Discard duplicate information about external model download by @anandhu-eng in #2370
- MIXTRAL - Update MLCFlow dataset and model download commands to suppo… by @anandhu-eng in #2371
- Update DLRM v2 assets download commands to support R2 downloader by @anandhu-eng in #2372
- SDXL: Update MLC commands to support model download through R2 by @anandhu-eng in #2373
- Updation for inference docs page: migration to R2 by @anandhu-eng in #2374
- Update MLC commands to support downloads through R2 by @anandhu-eng in #2367
- Replaced shell commands with Python for Windows compliance script compatibility by @sujik18 in #2344
New Contributors
- @hvagadia made their first contribution in #2230
- @wu6u3tw made their first contribution in #2259
- @taran2210 made their first contribution in #2303
- @anivar made their first contribution in #2289
- @naveenmiriyaluredhat made their first contribution in #2327
- @wangshangsam made their first contribution in #2360
- @sujik18 made their first contribution in #2344
Full Changelog: v5.1...v5.1.1