Set is_first_microbatch when quant_recipe is configured#5642
Set is_first_microbatch when quant_recipe is configured#5642yezhengmao1 wants to merge 3 commits into
Conversation
Signed-off-by: yezhengmao <yezhengmaolove@gmail.com>
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
|
Thanks—the fix direction looks correct and is appropriately scoped. Before approval, please add a focused regression test covering |
Signed-off-by: yezhengmao <yezhengmaolove@gmail.com>
|
Thanks for the review! I've pushed a focused regression test (tests/unit_tests/transformer/test_module.py, class TestSetIsFirstMicrobatch) covering both cases you asked for. @Connor-XY |
What does this PR do?
Enable the
is_first_microbatchflag when quantization is driven by aquant_recipe(e.g. via--te-precision-config-file), not only for thefp8/fp4/ kitchen paths.Issue tracking
Linked issue: #5641
Problem
set_is_first_microbatch()only propagates the flag whenconfig.fp8,config.fp4, oruse_kitchenis set. When quantization is configuredsolely through
--te-precision-config-file(which setsconfig.quant_recipewhile leaving
fp8/fp4asNoneanduse_kitchenasFalse), the guardnever enters, so per-module
is_first_microbatchis never set toTrue.As a result TE modules never refresh their fp8 weight (transpose) cache on
the first micro-batch of each step under per-module quantization — the same
root cause as expert-only fp8.
Fix
Add
quant_recipe is not Noneto the guard. The flag is a no-op for modulesthat run in bf16 (TE ignores it outside the fp8 path), so this is safe for
mixed bf16/fp8 recipes and mirrors the existing global
fp8behavior.Contribution process
Pre-checks