NVFP4 vs MXFP4 Decode Benchmark

Cezar Cocu·

If you've been following the latest FP4 formats used by the GPU/TPU manufacturers, you've probably read a bunch of articles comparing NVFP4 and MXFP4: block sizes, scale formats, yada yada.

What I personally haven't seen much of is actual results on actual production inference workloads. Does it matter?!

So, Stas did measure that on a B200, NVFP4 gets ~9% more TFLOPS than MXFP4 on large GEMMs (read: matrix multiplys, GEneral Matrix Multiply). Well, that's a pretty big difference!

But it's still an isolated GEMM benchmark. Does the perf difference actually translate to production workloads?

Also, what about other non-GEMM-bound workloads, like decode? Also ... why?!

Well, we can test it! Perhaps we can sus it out. (which is the whole point of this post, and probably why you're still reading)

Ok. I tried to keep it simple by picking a realistic-ish dense LLM (Qwen3-32B), served with vLLM, on a B200 (via Modal). Both formats were quantized from scratch from the same BF16 weights, in this way we try to keep the confounders to a minimum.

My hypothesis was that MXFP4 should be faster as we saturate HBM throughput due to MXFP4's slightly smaller bits (4.25 vs 4.5). aka: this is the classic bandwidth-bound workload so why not?

TL;DR

Left: NVFP4 decode throughput relative to MXFP4 by batch size, 7–8% faster at batch 1–8 and within ±2% from batch 64 to 512, next to a memory-bound prediction of MXFP4 being up to 5% faster. Right: the same NVFP4 weights on vLLM's CuTe-DSL and cuDNN kernels, relative to MXFP4.

The benchmark Claude mostly built & I ran suggests that if you're using a B200 GPU, you should probably always prefer NVFP4 over MXFP4, by a meaningful margin (up to ~8% faster decode at small batches)!

This holds for low-batch decode, and for GEMM-bound training NVFP4 looks better too with today's kernels (see below), though that's down to the kernels: with an identical kernel, MXFP4 is actually ~9% faster (see the update below). The difference disappears at larger batch sizes, which at first didn't surprise me: I assumed we were becoming HBM-bandwidth bound as the batch and KV-cache reads grow. I won't spoil it, but that turned out not to be the reason!

Additionally, quantizing from BF16 to the respective formats and measuring the loss indicates that NVFP4 should yield non-trivially better eval perf. I didn't run any evals to confirm this however.

Now, these findings may generalize but as Stas suggests, you should always strive to benchmark your actual production configuration.

Kernel implementations seem to dominate the perf differences

On a B200 (SM100) with vLLM v0.31, the NVFP4/MXFP4 decode perf difference mostly comes down to the kernel implementation, not to the 4.5 vs 4.25 bits difference. Note again that I tested this on only one dense model (Qwen3-32B, w/ BF16 KV cache).

Small-batch decode: NVFP4 is meaningfully faster

+7.1%, +7.6% and +4.2% decode throughput at batch 1, 8 and 32.

Worth highlighting that decode for this workload is neither bandwidth-bound (~40–50% of theoretical HBM bandwidth) nor compute-bound (the GEMMs are tiny), so kernel implementation differences dominate. NVFP4's superior perf here seems to come mostly from the kernels around the GEMMs: activation quantization, fewer kernels per step, etc.

Breakdown of the time NVFP4 saves per decode step vs MXFP4: at batch 1, 99 µs in the FP4 GEMMs, 130 µs in activation quantization and SiLU, 62 µs in other kernels and 103 µs in idle gaps, so 75% of the saving is outside the GEMMs; at batch 32, 85%.

Batch ≥ 64: there's no meaningful difference

So you might as well pick the format based on eval perf, portability or other concerns if you're in this regime.

Batch size dominates the perf difference & it's not due to HBM memory bandwidth saturation?

NVFP4's non-GEMM kernels are faster, but its GEMMs fall behind MXFP4's as they grow in size, and from batch 64 up the two differences cancel each other out.

So is it the batch size or the number of tokens that end up dominating? I kept the batch at 1 and grew the context instead, and separately held the tokens constant and grew the batch:

Two panels. Left: at batch 1, NVFP4 saves about 0.4 ms per decode step at every context length from 1k to 127k tokens. Right: at a constant 128k tokens of KV cache, the saving holds at batch 1, 8 and 32, then flips to −0.18 ms at batch 128.

Turns out it's the batch size. Look @ batch 1, NVFP4 keeps saving the same ~0.4 ms per step all the way out to 127k tokens of context (the % only shrinks because each step gets longer). But at the same 128k tokens, bumping the batch to 128 wipes the saving out. More memory traffic didn't make a significant enough difference, but the bigger GEMMs did! Huh?

But why?!

Perhaps a longer context only adds attention work, i.e. more KV cache reads. Remember that I kept the KV cache at BF16 for both formats, so both get slower by the same amount and NVFP4 keeps its lead. So, a bigger batch, on the other hand, means bigger GEMMs, and vLLM autotunes MXFP4's GEMM but (I think?) runs NVFP4's on a fixed heuristic. So as the GEMMs grow, NVFP4's fall further behind MXFP4's, until that slowdown cancels out NVFP4's lead. Maybe!

Now, I'm still not 100% sure it isn't bandwidth: @ batch 128, MXFP4's small edge matches what the byte difference should theoreticaly be. But @ batch 1 w/ a 127k context, with the same memory traffic, and similar bandwidth use, NVFP4 keeps its lead, and the GEMM size is the only thing that changed between the runs.

This confuses me enough that I'll revisit it and read some metrics straight from the GPU to either confirm or rule it out.

Large GEMMs (prefill/training) depend heavily on the kernel implementation

In PyTorch, NVFP4 gets ~9% more TFLOPS (ml-engineering book), but there (at least in the PyTorch 2.13 I checked) NVFP4 runs on cuBLASLt and MXFP4 on a different library (MSLK). In vLLM's kernels (CuTe-DSL), NVFP4's largest GEMMs were from 4% to 20% slower at batch 512.

Update (Oct 15): it's definitely the kernels

Santosh challenged me on the finings here: if the hardware roofline is about the same for both formats, then why how could NVFP4 be faster? To convince ourselves we need to separate the format from the kernels. So I ran both through the same QuACK's GemmSm100 kernel on a B200. Now, QuACK isn't as fast as the FlashInfer vLLM kernels, but it allows us to do an apples to apples comparison. The findings from this experiement are consistent with the theory here as MXFP4 was ~9% faster on most large GEMMs, theoretically consistent with its smaller format size.

Roofline on a B200: attained TFLOPS against FLOPs per byte for the gate_up_proj GEMM (51200×5120) through the same QuACK GemmSm100 kernel and 256×256 tile, M from 32 to 4096, under the spec (9 PFLOPS, 8 TB/s) and practical (6.6 PFLOPS, 6.66 TB/s) roofs. MXFP4 sits 9–11% above NVFP4 along the whole curve (1% at M = 32), with identical MMA instructions and cycles; NVFP4 copies twice as many block scales into tensor memory.

Blackwell Ultra

For even more evidence that kernel implementations dominate real life production performance, we can take a look at some Blackwell Ultra numbers. Blackwell Ultra's have more even more fp4 tensor cores (50% more in fact) than B200 (regular Blackwell). In addition, B300's are newer with even less mature kernel support: It seems in order to reach the theoretical 50% fp4 inference speedup vs B200's, you have to be on PyTorch 2.14.1 & CUDA 13.2 and use NVFP4. Also theoretical roofline perf difference also seems to differ based on the exact B300 config.

That perf difference widens even further when comparing GEMM perf on a B300 (Blackwell Ultra) between the formats, very likely due to the less polished mxfp4 kernels available for Blackwell Ultra. The NVFP4 kernels reach 7434 sustained TFLOPS vs 5847 for mxfp4 on PyTorch 2.14.1 & cu132. See here.

You should serve NVFP4 on vLLM's default kernel

That's CuTe-DSL (--linear-backend flashinfer_cutedsl). On cuDNN (flashinfer_cudnn) the same weights are between 10 and 18% slower and lose to MXFP4 at every batch size. MXFP4 only has a single W4A4 kernel in vLLM 0.31 (FlashInfer's mm_fp4 with the CuTe-DSL backend).

NVFP4 is more accurate

Its NLL penalty after quantization from BF16 is 2.7× smaller. While no evals were run, the lower NLL (negative log-likelihood) suggests that NVFP4 should yield better eval results.

Not tested here but to highlight this, NVIDIA's own paper found that an small LLM model trained with MXFP4 needed 36% more tokens (1.36T vs 1T) to match NVFP4's loss. Though they noted that both formats achieved similar raw throughput on both GB200 & GB300 (training is usually GEMM-bound).

Code & Weights

The code, quantized weights and full write-up are on GitHub and HF (MXFP4, NVFP4).