NVFP4 vs MXFP4 Decode Benchmark
If you've been following the latest FP4 formats used by the GPU/TPU manufacturers, you've probably read a bunch of articles comparing NVFP4 and MXFP4: block sizes, scale formats, yada yada.
What I personally haven't seen much of is actual results on actual production inference workloads. Does it matter?!
So, Stas did measure that on a B200, NVFP4 gets ~9% more TFLOPS than MXFP4 on large GEMMs (read: matrix multiplys, GEneral Matrix Multiply). Well, that's a pretty big difference!
But it's still an isolated GEMM benchmark. Does the perf difference actually translate to production workloads?
Also, what about other non-GEMM-bound workloads, like decode? Also ... why?!
Well, we can test it! Perhaps we can sus it out. (which is the whole point of this post, and probably why you're still reading)
Ok. I tried to keep it simple by picking a realistic-ish dense LLM (Qwen3-32B), served with vLLM, on a B200 (via Modal). Both formats were quantized from scratch from the same BF16 weights, in this way we try to keep the confounders to a minimum.
My hypothesis was that MXFP4 should be faster as we saturate HBM throughput due to MXFP4's slightly smaller bits (4.25 vs 4.5). aka: this is the classic bandwidth-bound workload so why not?
TL;DR

The benchmark Claude mostly built & I ran suggests that if you're using a B200 GPU, you should probably always prefer NVFP4 over MXFP4, by a meaningful margin (up to ~8% faster decode at small batches)!
This holds for low-batch decode, and for GEMM-bound training NVFP4 looks better too with today's kernels (see below), though that's down to the kernels: with an identical kernel, MXFP4 is actually ~9% faster (see the update below). The difference disappears at larger batch sizes, which at first didn't surprise me: I assumed we were becoming HBM-bandwidth bound as the batch and KV-cache reads grow. I won't spoil it, but that turned out not to be the reason!
Additionally, quantizing from BF16 to the respective formats and measuring the loss indicates that NVFP4 should yield non-trivially better eval perf. I didn't run any evals to confirm this however.
Now, these findings may generalize but as Stas suggests, you should always strive to benchmark your actual production configuration.
Kernel implementations seem to dominate the perf differences
On a B200 (SM100) with vLLM v0.31, the NVFP4/MXFP4 decode perf difference mostly comes down to the kernel implementation, not to the 4.5 vs 4.25 bits difference. Note again that I tested this on only one dense model (Qwen3-32B, w/ BF16 KV cache).
Small-batch decode: NVFP4 is meaningfully faster
+7.1%, +7.6% and +4.2% decode throughput at batch 1, 8 and 32.
Worth highlighting that decode for this workload is neither bandwidth-bound (~40–50% of theoretical HBM bandwidth) nor compute-bound (the GEMMs are tiny), so kernel implementation differences dominate. NVFP4's superior perf here seems to come mostly from the kernels around the GEMMs: activation quantization, fewer kernels per step, etc.

Batch ≥ 64: there's no meaningful difference
So you might as well pick the format based on eval perf, portability or other concerns if you're in this regime.
Batch size dominates the perf difference & it's not due to HBM memory bandwidth saturation?
NVFP4's non-GEMM kernels are faster, but its GEMMs fall behind MXFP4's as they grow in size, and from batch 64 up the two differences cancel each other out.
So is it the batch size or the number of tokens that end up dominating? I kept the batch at 1 and grew the context instead, and separately held the tokens constant and grew the batch:

Turns out it's the batch size. Look @ batch 1, NVFP4 keeps saving the same ~0.4 ms per step all the way out to 127k tokens of context (the % only shrinks because each step gets longer). But at the same 128k tokens, bumping the batch to 128 wipes the saving out. More memory traffic didn't make a significant enough difference, but the bigger GEMMs did! Huh?
But why?!
Perhaps a longer context only adds attention work, i.e. more KV cache reads. Remember that I kept the KV cache at BF16 for both formats, so both get slower by the same amount and NVFP4 keeps its lead. So, a bigger batch, on the other hand, means bigger GEMMs, and vLLM autotunes MXFP4's GEMM but (I think?) runs NVFP4's on a fixed heuristic. So as the GEMMs grow, NVFP4's fall further behind MXFP4's, until that slowdown cancels out NVFP4's lead. Maybe!
Now, I'm still not 100% sure it isn't bandwidth: @ batch 128, MXFP4's small edge matches what the byte difference should theoreticaly be. But @ batch 1 w/ a 127k context, with the same memory traffic, and similar bandwidth use, NVFP4 keeps its lead, and the GEMM size is the only thing that changed between the runs.
This confuses me enough that I'll revisit it and read some metrics straight from the GPU to either confirm or rule it out.
Large GEMMs (prefill/training) depend heavily on the kernel implementation
In PyTorch, NVFP4 gets ~9% more TFLOPS (ml-engineering book), but there (at least in the PyTorch 2.13 I checked) NVFP4 runs on cuBLASLt and MXFP4 on a different library (MSLK). In vLLM's kernels (CuTe-DSL), NVFP4's largest GEMMs were from 4% to 20% slower at batch 512.
Update (Oct 15): it's definitely the kernels
Santosh challenged me on the finings here: if the
hardware roofline is about the same for both formats, then why how could NVFP4 be faster?
To convince ourselves we need to separate the format from the kernels. So I ran both through the same QuACK's GemmSm100 kernel on a B200.
Now, QuACK isn't as fast as the FlashInfer vLLM kernels, but it allows us to do an apples to apples comparison.
The findings from this experiement are consistent with the theory here as MXFP4 was
~9% faster on most large GEMMs, theoretically consistent with its smaller format size.

Blackwell Ultra
For even more evidence that kernel implementations dominate real life production performance, we can take a look at some Blackwell Ultra numbers. Blackwell Ultra's have more even more fp4 tensor cores (50% more in fact) than B200 (regular Blackwell). In addition, B300's are newer with even less mature kernel support: It seems in order to reach the theoretical 50% fp4 inference speedup vs B200's, you have to be on PyTorch 2.14.1 & CUDA 13.2 and use NVFP4. Also theoretical roofline perf difference also seems to differ based on the exact B300 config.
That perf difference widens even further when comparing GEMM perf on a B300 (Blackwell Ultra) between the formats, very likely due to the less polished mxfp4 kernels available for Blackwell Ultra. The NVFP4 kernels reach 7434 sustained TFLOPS vs 5847 for mxfp4 on PyTorch 2.14.1 & cu132. See here.
You should serve NVFP4 on vLLM's default kernel
That's CuTe-DSL (--linear-backend flashinfer_cutedsl). On cuDNN (flashinfer_cudnn) the same
weights are between 10 and 18% slower and lose to MXFP4 at every batch size. MXFP4 only has a single
W4A4 kernel in vLLM 0.31 (FlashInfer's mm_fp4 with the CuTe-DSL backend).
NVFP4 is more accurate
Its NLL penalty after quantization from BF16 is 2.7× smaller. While no evals were run, the lower NLL (negative log-likelihood) suggests that NVFP4 should yield better eval results.
Not tested here but to highlight this, NVIDIA's own paper found that an small LLM model trained with MXFP4 needed 36% more tokens (1.36T vs 1T) to match NVFP4's loss. Though they noted that both formats achieved similar raw throughput on both GB200 & GB300 (training is usually GEMM-bound).
Code & Weights
The code, quantized weights and full write-up are on GitHub and HF (MXFP4, NVFP4).