Skip to content

fix(kernel): use Triton FP8 quantization on SM120#646

Merged
lightseek-bot merged 1 commit into
lightseekorg:mainfrom
lucifer1004:agent/fix-sm120-mxfp8-quantization
Jul 12, 2026
Merged

fix(kernel): use Triton FP8 quantization on SM120#646
lightseek-bot merged 1 commit into
lightseekorg:mainfrom
lucifer1004:agent/fix-sm120-mxfp8-quantization

Conversation

@lucifer1004

@lucifer1004 lucifer1004 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • route SM120 online MXFP8 activation quantization for FlashInfer and Triton GEMMs through TokenSpeed's Triton token-group kernel
  • expose that Triton quantizer on NVIDIA while preventing unsupported TRT-LLM FP8 quantizers from being selected beyond SM10.x
  • reload quantization registrations in the existing kernel API test fixture now that GEMM uses the public quantization boundary

Root cause

The TRT-LLM dynamic FP8 operators bundled in the current environment do not have SM120 kernel images. The 1x128 path also exposes a flat padded scale tensor where the GEMM adapter expects a two-dimensional group-major layout. Selecting it for SM120 therefore fails before the FlashInfer GEMM can run and can leave a delayed cudaErrorNoKernelImageForDevice in the CUDA context.

The existing TokenSpeed Triton quantizer produces the row-major [tokens, K / 128] scales consumed by the SM120 FlashInfer GEMM. Other architectures keep their existing selection path and priorities.

Validation

  • inferlab run --environment tokenspeed -- pytest tokenspeed/tokenspeed-kernel/test/ops/test_quantization.py tokenspeed/tokenspeed-kernel/test/test_kernel_api_selection.py -q (31 passed, 50 skipped)
  • prek run --all-files

@lucifer1004
lucifer1004 marked this pull request as ready for review July 11, 2026 12:03
@lucifer1004
lucifer1004 requested review from a team as code owners July 11, 2026 12:03
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
@lucifer1004
lucifer1004 force-pushed the agent/fix-sm120-mxfp8-quantization branch from bb4fa08 to 725b1b0 Compare July 11, 2026 12:13
@lightseek-bot
lightseek-bot merged commit 721df0b into lightseekorg:main Jul 12, 2026
69 of 71 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants