You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: launcher/src/main.rs
+3Lines changed: 3 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -47,8 +47,11 @@ enum Quantization {
47
47
/// Bitsandbytes 4bit. nf4 should be preferred in most cases but maybe this one has better
48
48
/// perplexity performance for you model
49
49
BitsandbytesFP4,
50
+
/// [BETA]
50
51
/// [FP8](https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/) (e4m3) works on H100 and above
51
52
/// This dtype has native ops should be the fastest if available.
53
+
/// This is currently not the fastest because of local unpacking + padding to satisfy matrix
0 commit comments