Skip to content

Conversation

drisspg
Copy link
Contributor

@drisspg drisspg commented Oct 13, 2025

Summary

  # Float8 tensorwise (default)
 python benchmarks/float8/float8_inference_roofline.py benchmarks/data/output.csv --recipe_name=tensorwise --do_benchmarks=False

  # Float8 rowwise  
 python benchmarks/float8/float8_inference_roofline.py benchmarks/data/output.csv --recipe_name=rowwise --do_benchmarks=False

  # MX format
 python benchmarks/float8/float8_inference_roofline.py benchmarks/data/output.csv --recipe_name=mxfp8 --do_benchmarks=False
  
  # Nvfp4
 python benchmarks/float8/float8_inference_roofline.py benchmarks/data/output.csv --recipe_name=nvfp4 --do_benchmarks=False
  

Copy link

pytorch-bot bot commented Oct 13, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3167

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit 83ce813 with merge base 596da93 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 13, 2025
@drisspg drisspg force-pushed the inference-nums branch 2 times, most recently from d2a0328 to 39739d2 Compare October 13, 2025 17:53
@drisspg drisspg requested a review from vkuzo October 13, 2025 17:57
@drisspg drisspg added the topic: not user facing Use this tag if you don't want this PR to show up in release notes label Oct 13, 2025
@vkuzo
Copy link
Contributor

vkuzo commented Oct 13, 2025

@drisspg
Copy link
Contributor Author

drisspg commented Oct 13, 2025

whoops didnt see that one yeah I can move over

# kernel 1: x_bf16 -> x_nvfp4 (per-tensor scaling for inference)
kernel_1_rw = BYTES_PER_EL_BF16 * numel + BYTES_PER_EL_FLOAT4 * numel
# add minimal scaling overhead (per-tensor scale)
kernel_1_rw += BYTES_PER_EL_FLOAT32 # single scale factor
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the blockwise scaling write here too, since we are adding it for mxfp8? just to be consistent, I don't think it will change the numbers that much

Copy link
Contributor

@vkuzo vkuzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@drisspg drisspg force-pushed the inference-nums branch 2 times, most recently from 201de29 to e307ded Compare October 14, 2025 04:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. topic: not user facing Use this tag if you don't want this PR to show up in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants