FP8, NVFP4, MXFP4, INT4 and GGUF explained: bits per weight, scales and which GPU computes what

FP8: E4M3 and E5M2 encodings and their scales

With their scales counted, FP8 takes 8 bits per weight, NVFP4 4.5 and MXFP4 4.25, while INT4 AWQ or GPTQ takes about 4.15 and a GGUF Q4_K_M file about 4.9. Ada and Hopper GPUs such as the L40S and H200 NVL compute FP8 but not FP4; NVFP4 and MXFP4 are computed natively from Blackwell on, for example on the RTX PRO Blackwell cards and DGX Spark. FP8 is the simplest of the five formats, and the OCP Microscaling specification lists two encodings for it: E4M3, whose largest normal value is ±448 and which has no infinities, and E5M2, which reaches ±57,344. In these names, E counts the exponent bits and M the mantissa bits. NVIDIA’s Transformer Engine documentation assigns E4M3 to weights and activations and E5M2 to gradients; inference checkpoints such as Meta’s Llama 3.1 405B FP8 store their quantised weights as E4M3. We supply a card for each of these formats, from the L4 to DGX Spark. Describe your model and your users, and our team will help you choose the format and the card.

Eight bits cannot cover a tensor’s range on their own, so FP8 tensors carry scales: one FP32 scale per tensor, as in Transformer Engine’s original FP8 recipe, or finer ones per row or per block, such as MXFP8’s power-of-two scale per 32 values. DeepSeek-V3 scales weights per 128 × 128 block and activations per 1 × 128 tile to “better accommodate outliers”; Qwen’s FP8 release of Qwen3-235B uses weight blocks of 128 × 128, according to its config.json. By our arithmetic, the scales add about 0.002 bits per weight for 128 × 128 blocks with FP32 scales, so FP8 takes one byte per weight, half as much as BF16. MXFP8’s 8-bit scales add 0.25 bits. According to its model card, NVIDIA’s FP8 Llama 3.3 70B scores 83.2 on the MMLU benchmark and 94.3 on GSM8K with chain of thought; the BF16 model scores 83.3 and 95.3.

NVFP4 vs MXFP4: block size and scale format

Both formats store each weight as FP4 E2M1: a sign, two exponent bits and one mantissa bit, which allows exactly 0, 0.5, 1, 1.5, 2, 3, 4 and 6, positive or negative. The two formats differ in the scale stored with each block of values.

MXFP4 comes from the OCP Microscaling specification v1.0 of September 2023: every 32 values share one 8-bit E8M0 scale, a power of two. That makes 4 + 8/32 = 4.25 bits per weight, the figure OpenAI’s model card gives for the MoE weights of gpt-oss, which it says lets the 117B model fit a single 80 GB GPU.

NVFP4 is NVIDIA’s variant: every 16 values share an FP8 E4M3 scale, and a second FP32 scale per tensor keeps the block scales in range. That makes 4 + 8/16 = 4.5 bits per weight, as NVIDIA’s technical blog states; the per-tensor scale adds only 32 bits to a tensor of millions of weights. Unlike MXFP4’s power-of-two scale, an E4M3 scale can also take values between powers of two. NVIDIA puts the saving at “approximately 3.5x relative to FP16, and approximately 1.8x compared to FP8”. It also reports that DeepSeek-R1-0528 lost “1% or less” moving from FP8 to NVFP4 across seven benchmarks, for example from 85 to 84 per cent on MMLU-Pro. NVIDIA’s NVFP4 Llama 3.3 70B scores 81.1 on MMLU and 92.6 on GSM8K; the BF16 model scores 83.3 and 95.3.

INT4 with AWQ or GPTQ: 4-bit weights, 16-bit or FP8 maths

AWQ and GPTQ turn weights into 4-bit integers and usually store a scale, often with a zero point, for every 128 weights. The AWQ paper uses 128 weights per scale “throughout the work”. The GPTQ paper, whose main results scale per row, puts the cost of a scale per 128 weights at about 0.15 extra bits, so roughly 4.15 bits per weight. Both quantise only the weights. As the AWQ paper explains, “the hardware does not provide multiplication instructions between INT4 and FP16”, so its kernels unpack the integers to FP16 inside the matrix multiplication: W4A16, 4-bit weights and 16-bit activations. W4A8 pairs 4-bit weights with FP8 activations, and NVIDIA’s Model Optimizer guide lists it for “Ada, Hopper and later”, the GPUs with FP8 Tensor Cores. W8A8 takes weights and activations to 8 bits, as FP8 or INT8.

The same guide says when each one is the better choice. At batch sizes up to four, inference is often “memory-bound” and weight-only INT4 gives the larger gain; for serving at 16 and more, it recommends quantising activations too and suggests “prioritizing using FP8 first”. It lists INT4 AWQ for “Ampere and later”.

GGUF: llama.cpp’s block formats

GGUF is a file format, not a number format: one file holding the tensors and their metadata for llama.cpp and the tools that use its files, such as Ollama, with llama.cpp’s own block types inside. Q8_0 stores 8-bit values with one scale per 32 weights: 8.5 bits per weight on Llama 3.1 8B in llama.cpp’s table. Q4_K packs 256 weights into a super-block of eight 32-weight blocks with 6-bit scales and minimums, 4.5 bits per weight. The popular Q4_K_M is a file type rather than a block type: it keeps more tensors in Q6_K, among them some of the attention value and feed-forward down projections, which is why the same table gives it 4.89 bits per weight, against 4.67 for Q4_K_S.

No Tensor Core computes these formats as such. llama.cpp’s build documentation says its own kernels for quantised models are the default “on GPUs with int8 tensor core support”, with FP16 cuBLAS as the alternative. vLLM loads GGUF files but calls its support “highly experimental and under-optimized”; for a multi-user vLLM service, start from FP8, NVFP4 or AWQ.

Bits per weight, scales included

NVIDIA NVFP4 technical blog (24 June 2025), OCP Microscaling Formats v1.0, NVIDIA’s Transformer Engine FP8 primer, OpenAI’s gpt-oss model card, AWQ (MLSys 2024) and GPTQ (ICLR 2023) papers, Hugging Face GGUF documentation, llama.cpp’s llama-quantize README, TensorRT-LLM quantisation matrix (1.3.0rc28; pre-release 1.4.0rc1 followed on 21 October 2026). The INT4 figure applies GPTQ’s estimate for a scale per 128 weights; the FP8 overhead is our arithmetic.

This site sizes NVFP4 at about 0.56 bytes per parameter, which is the per-weight figure: 4.5 bits are 0.5625 bytes. Whole checkpoints come out higher because not everything is quantised. NVIDIA’s NVFP4 Llama 3.3 70B quantises only the linear layers inside the transformer blocks and keeps the embeddings and the output layer in BF16. By our arithmetic from the checkpoint’s tensor types, its 68.45 billion NVFP4 weights take 38.5 GB: 34.2 GB of 4-bit values and 4.3 GB of FP8 scales. Its 2.1 billion BF16 parameters add 4.2 GB. Together that is 42.7 GB, the published size, or about 0.61 bytes per parameter overall. The same arithmetic works for Qwen3-32B-AWQ: 31.2 billion 4-bit weights at about 4.15 bits and 1.56 billion 16-bit embedding and output parameters make its 19.3 GB. Our VRAM guide turns weights like these into a memory budget.

Which Tensor Cores compute which format

TensorRT-LLM quantisation support matrix, documentation 1.3.0rc28, and NVIDIA’s compute capability list, both read on 23 September 2026; TensorRT-LLM 1.4.0rc1 followed on 21 October 2026. The FP8 scaling schemes and INT4 modes are those TensorRT-LLM supports on each architecture, not limits of the Tensor Cores. “In vLLM” marks vLLM’s weight-only Marlin kernels, which vLLM’s compatibility table lists for Ampere, Ada and Hopper.

The table shows two boundaries. FP8 arithmetic starts at compute capability 8.9: vLLM states that “FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell)” and runs FP8 models on Turing and Ampere “as weight-only W8A16”. FP4 arithmetic starts with Blackwell: NVIDIA’s blog says the “fifth-generation Tensor Core architecture implements NVFP4”, and TensorRT-LLM’s recipes state that “NVFP4 is only supported on NVIDIA Blackwell”. vLLM still loads NVFP4 checkpoints on Ampere, Ada and Hopper GPUs; its documentation, as of September 2026, states: “On GPUs without a supported native FP4 GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin and logs a warning; this may reduce throughput for compute-heavy workloads.” On those GPUs the checkpoint still saves memory, but the arithmetic runs in 16-bit, and NVIDIA’s own NVFP4 model card lists Blackwell as the only supported architecture.

Of the GPUs we supply, the RTX PRO Blackwell cards and DGX Spark compute FP4 natively; the H200 NVL, L40S and L4 compute FP8 but not FP4. Tell us the checkpoint you plan to run, and we quote a card that runs it natively.

FP8 KV cache: half the cache per token

The KV cache holds keys and values for every token of every open conversation and uses the memory that remains once the weights are loaded. Stored in FP8 instead of BF16 it takes half as much: for Llama 3.3 70B, 160 instead of 320 KiB per token, or 1.25 instead of 2.5 GiB for an 8,192-token conversation. Take one H200 NVL with FP8 weights and our users-per-card rule: 90 per cent of the 140.4 GiB the driver reports, less 3 GiB per card for activations and the runtime, less the weights. After 67.7 GiB of weights, 55.7 GiB remain for the cache, which by our arithmetic holds 22 such conversations in BF16 and 44 in FP8.

We do the same arithmetic for your case in a free sizing call and turn it into a server configuration you can compare with cloud GPUs. Write to us with the model and the number of concurrent users.

vLLM switches the FP8 KV cache on with --kv-cache-dtype fp8; without calibration its scales are 1.0, and its documentation recommends calibrating on a dataset through llm-compressor for maximum accuracy. SGLang accepts fp8_e4m3 or fp8_e5m2, and NVIDIA’s NVFP4 Llama 3.3 70B declares an FP8 KV cache in its hf_quant_config.json. TensorRT-LLM lists the FP8 KV cache for every architecture from Ampere to Blackwell, and a smaller NVFP4 KV cache for the B200 and B300 class only. vLLM 0.32.0 of 18 October 2026 states that its “NVFP4 KV cache now runs on SM8x and SM12x”, which includes the RTX PRO Blackwell cards. LMSYS published the first accuracy figures for SGLang on 16 September 2026, where the support is marked experimental: for Qwen3.8-27B, an NVFP4 cache scored 0.31 percentage points lower on GSM8K and 1.01 lower on GPQA-Diamond than an FP8 cache, and resolved 76.20 against 77.80 per cent of SWE-bench Verified. Two models are not a general rule, so our sizing keeps the FP8 cache, and you should test NVFP4 on your evaluation set before relying on it; our page on the NVFP4 KV cache on the RTX PRO 6000 Server Edition has the figures.

What to pick on L4, L40S, H200 NVL, RTX PRO and DGX Spark

L4 and L40S (Ada). Use FP8 for every model that fits: it is native, halves the weights and, on NVIDIA’s Llama 3.3 70B model cards, costs 0.1 points on MMLU. For 4-bit, use INT4 AWQ or GPTQ, W4A16 for a few users and W4A8 as batches grow; in vLLM, NVFP4 checkpoints load only through its weight-only fallback. Our L4 and L40S comparison sizes the models: 14B in FP8 on the L4 and 32B in FP8 on the L40S; a 70B model in 4-bit fits an L40S by weights but leaves almost no memory for the cache.

H200 NVL (Hopper). FP8 is the default, with every FP8 scaling scheme TensorRT-LLM offers, and INT4 AWQ or GPTQ when memory runs short; choose these rather than NVFP4 checkpoints. gpt-oss also runs on the H200 NVL. OpenAI states that it fits a single 80 GB GPU such as the H100, and in vLLM its MXFP4 weights run weight-only, as the table above shows. Card counts per model are in our guide to large models on H200 NVL.

RTX PRO Blackwell (12.0). NVFP4 and MXFP4 are the native 4-bit formats and, in TensorRT-LLM, the only ones: its matrix lists no INT4 AWQ or GPTQ for these cards, and FP8 only per tensor. NVFP4 weights need about 56 per cent of the memory of FP8, 4.5 against 8 bits, at the accuracy cost the model cards above show. For Llama 3.3 70B with an FP8 cache, that is 34 sessions of 8K per RTX PRO 6000 instead of 12 by our users-per-card rule, or 136 instead of 48 on a four-card RTX PRO 6000 Server Edition server like the inference node on our AI servers page.

DGX Spark (GB10, 12.1). At 273 GB/s of memory bandwidth, every byte saved per weight shows up in tokens per second. NVIDIA’s playbooks quantise to NVFP4 with Model Optimizer and serve with TensorRT-LLM or vLLM. In NVIDIA’s own benchmark of October 2025, gpt-oss-120b in MXFP4 runs through llama.cpp at 55.37 tokens per second of generation, with a 2,048-token prompt at batch 1. GGUF through llama.cpp or Ollama is the quickest start for one user; our DGX Spark sizing article shows what fits.

What we supply

We supply the professional NVIDIA GPUs from the section above: DGX Spark, the L4, L40S and H200 NVL, the RTX PRO 6000 in all three editions (Workstation, Max-Q and Server Edition) and the other RTX PRO Blackwell workstation cards. It all comes with manufacturer warranty on one EU contract and invoice. You can order the cards on their own, or in a server or workstation configured for the model and format you plan to run. Send us the model and we will match the card to it; quantities and lead time are confirmed in writing with the quote.

FAQ