FP8, NVFP4, MXFP4, INT4 and GGUF explained: bits per weight, scales and which GPU computes what

FP8: two encodings and a scale

A quantised format answers three questions: how many bits a weight takes once its scales are counted, which GPUs compute in it directly, and what it costs in accuracy. FP8 is the simplest case. The OCP Microscaling specification lists its two encodings: E4M3, whose largest normal value is ±448 and which has no infinities, and E5M2, which reaches ±57,344. NVIDIA’s Transformer Engine documentation assigns E4M3 to weights and activations and E5M2 to gradients; inference checkpoints such as Meta’s Llama 3.1 405B FP8 store their quantised weights as E4M3.

Eight bits cannot cover a tensor’s range on their own, so FP8 tensors carry scales: one FP32 scale per tensor, as in Transformer Engine’s original FP8 recipe, or finer ones per row or per block, such as MXFP8’s power-of-two scale per 32 values. DeepSeek-V3 scales weights per 128 × 128 block and activations per 1 × 128 tile to “better accommodate outliers”; Qwen’s FP8 release of Qwen3-235B uses weight blocks of 128 × 128, according to its config.json. The scales are close to free, about 0.002 extra bits per weight for 128 × 128 blocks with FP32 scales by our arithmetic, so FP8 is one byte per weight, half of BF16; MXFP8’s 8-bit scales add 0.25 bits. NVIDIA’s FP8 Llama 3.3 70B scores 83.2 on MMLU and 94.3 on GSM8K with chain of thought, against 83.3 and 95.3 in BF16, according to its model card.

NVFP4 and MXFP4: the same 4-bit values, different scales

Both formats store each weight as FP4 E2M1: a sign, two exponent bits and one mantissa bit, which allows exactly 0, 0.5, 1, 1.5, 2, 3, 4 and 6, positive or negative. What differs is the scale.

MXFP4 comes from the OCP Microscaling specification v1.0 of September 2023: every 32 values share one 8-bit E8M0 scale, a power of two. That makes 4 + 8/32 = 4.25 bits per weight, the figure OpenAI’s model card gives for the MoE weights of gpt-oss, which it says lets the 117B model fit a single 80 GB GPU.

NVFP4 is NVIDIA’s variant: every 16 values share an FP8 E4M3 scale, which, unlike MXFP4’s power-of-two scale, can also take values between powers of two, and a second FP32 scale per tensor keeps the block scales in range. That makes 4 + 8/16 = 4.5 bits per weight, as NVIDIA’s technical blog states; the per-tensor scale adds only 32 bits to a tensor of millions of weights. NVIDIA puts the saving at “approximately 3.5x relative to FP16, and approximately 1.8x compared to FP8”, and reports that DeepSeek-R1-0528 lost “1% or less” moving from FP8 to NVFP4 across seven benchmarks, for example from 85 to 84 per cent on MMLU-Pro. Its NVFP4 Llama 3.3 70B scores 81.1 on MMLU and 92.6 on GSM8K, against 83.3 and 95.3 in BF16.

INT4 with AWQ or GPTQ: 4-bit weights, 16-bit or FP8 maths

AWQ and GPTQ turn weights into 4-bit integers and usually store a scale, often with a zero point, for every 128 weights: the AWQ paper uses that size “throughout the work”, and the GPTQ paper, whose main results scale per row, puts its cost at about 0.15 extra bits, so roughly 4.15 bits per weight. Both quantise only the weights. As the AWQ paper explains, “the hardware does not provide multiplication instructions between INT4 and FP16”, so its kernels unpack the integers to FP16 inside the matrix multiplication: W4A16, 4-bit weights and 16-bit activations. W4A8 pairs 4-bit weights with FP8 activations, and NVIDIA’s Model Optimizer guide lists it for “Ada, Hopper and later”, the GPUs with FP8 Tensor Cores. W8A8 takes weights and activations to 8 bits, as FP8 or INT8.

The same guide says when each pays. At batch sizes up to four, inference is often “memory-bound” and weight-only INT4 gives the larger gain; for serving at 16 and more, it recommends quantising activations too and suggests “prioritizing using FP8 first”. It lists INT4 AWQ for “Ampere and later”.

GGUF: llama.cpp’s block formats

GGUF is a file format, not a number format: one file holding the tensors and their metadata for llama.cpp and the tools that use its files, such as Ollama, with llama.cpp’s own block types inside. Q8_0 stores 8-bit values with one scale per 32 weights: 8.5 bits per weight on Llama 3.1 8B in llama.cpp’s table. Q4_K packs 256 weights into a super-block of eight 32-weight blocks with 6-bit scales and minimums, 4.5 bits per weight. The popular Q4_K_M is a file type rather than a block type: it keeps more tensors in Q6_K, among them some of the attention value and feed-forward down projections, which is why the same table gives it 4.89 bits per weight, against 4.67 for Q4_K_S.

No Tensor Core computes these formats as such. llama.cpp’s build documentation says its own kernels for quantised models are the default “on GPUs with int8 tensor core support”, with FP16 cuBLAS as the alternative. vLLM loads GGUF files but calls its support “highly experimental and under-optimized”; for a multi-user vLLM service, start from FP8, NVFP4 or AWQ.

Bits per weight, scales included

NVIDIA NVFP4 technical blog (24 June 2025), OCP Microscaling Formats v1.0, NVIDIA’s Transformer Engine FP8 primer, OpenAI’s gpt-oss model card, AWQ (MLSys 2024) and GPTQ (ICLR 2023) papers, Hugging Face GGUF documentation, llama.cpp’s llama-quantize README, TensorRT-LLM quantisation matrix (1.3.0rc28). The INT4 figure applies GPTQ’s estimate for a scale per 128 weights; the FP8 overhead is our arithmetic.

This site sizes NVFP4 at about 0.56 bytes per parameter, which is the per-weight figure: 4.5 bits are 0.5625 bytes. Whole checkpoints come out higher because not everything is quantised. NVIDIA’s NVFP4 Llama 3.3 70B quantises only the linear layers inside the transformer blocks and keeps the embeddings and the output layer in BF16. By our arithmetic from the checkpoint’s tensor types, 68.45 billion NVFP4 weights take 38.5 GB, 34.2 GB of 4-bit values and 4.3 GB of FP8 scales, and 2.1 billion BF16 parameters take 4.2 GB: 42.7 GB, the published size, or about 0.61 bytes per parameter overall. Qwen3-32B-AWQ checks out the same way: 31.2 billion 4-bit weights at about 4.15 bits and 1.56 billion 16-bit embedding and output parameters make its 19.3 GB. Our VRAM guide turns weights like these into a memory budget.

Which Tensor Cores compute which format

TensorRT-LLM quantisation support matrix, documentation 1.3.0rc28, and NVIDIA’s compute capability list, both read on 23 September 2026. The FP8 scaling schemes and INT4 modes are those TensorRT-LLM supports on each architecture, not limits of the Tensor Cores. “In vLLM” marks vLLM’s weight-only Marlin kernels, which vLLM’s compatibility table lists for Ampere, Ada and Hopper.

Two boundaries matter. FP8 arithmetic starts at compute capability 8.9: vLLM states that “FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell)” and runs FP8 models on Turing and Ampere “as weight-only W8A16”. FP4 arithmetic is Blackwell’s: NVIDIA’s blog says the “fifth-generation Tensor Core architecture implements NVFP4”, and TensorRT-LLM’s recipes state that “NVFP4 is only supported on NVIDIA Blackwell”. vLLM still loads NVFP4 checkpoints on Ampere, Ada and Hopper GPUs; its documentation, as of September 2026, states: “On GPUs without a supported native FP4 GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin and logs a warning; this may reduce throughput for compute-heavy workloads.” The memory saving stays, the FP4 arithmetic does not, and NVIDIA’s own NVFP4 model card lists Blackwell as the only supported architecture.

FP8 KV cache: half the cache per token

The KV cache holds keys and values for every token of every open conversation and gets whatever memory the weights leave. Stored in FP8 instead of BF16 it takes half as much: for Llama 3.3 70B, 160 instead of 320 KiB per token, or 1.25 instead of 2.5 GiB for an 8,192-token conversation. On one H200 NVL with FP8 weights, a planning rule of 90 per cent of the 140.4 GiB the driver reports, minus 68 GiB of weights, leaves 58.4 GiB of cache: 23 such conversations in BF16 and 46 in FP8, by our arithmetic.

vLLM switches it on with --kv-cache-dtype fp8; without calibration its scales are 1.0, and its documentation recommends calibrating on a dataset through llm-compressor for maximum accuracy. SGLang accepts fp8_e4m3 or fp8_e5m2, and NVIDIA’s NVFP4 Llama 3.3 70B declares an FP8 KV cache in its hf_quant_config.json. TensorRT-LLM lists the FP8 KV cache for every architecture from Ampere to Blackwell, and a smaller NVFP4 KV cache for the B200 and B300 class only.

What to pick on L4, L40S, H200 NVL, RTX PRO and DGX Spark

L4 and L40S (Ada). FP8 for everything that fits: it is native, halves the weights and, on NVIDIA’s Llama 3.3 70B model cards, costs 0.1 points on MMLU. For 4-bit, use INT4 AWQ or GPTQ, W4A16 for a few users and W4A8 as batches grow; in vLLM, NVFP4 checkpoints load only through its weight-only fallback. Our L4 and L40S comparison sizes the models: 14B in FP8 on the L4, 32B in FP8 or 70B in 4-bit on the L40S.

H200 NVL (Hopper). FP8 is the default, with every FP8 scaling scheme TensorRT-LLM offers, and INT4 AWQ or GPTQ when memory runs short; NVFP4 checkpoints are the wrong download. gpt-oss runs: OpenAI states that it fits a single 80 GB GPU such as the H100, and in vLLM its MXFP4 weights run weight-only, as the table above shows. Card counts per model are in our guide to large models on H200 NVL.

RTX PRO Blackwell (12.0). NVFP4 and MXFP4 are the native 4-bit formats and, in TensorRT-LLM, the only ones: its matrix lists no INT4 AWQ or GPTQ for these cards, and FP8 only per tensor. NVFP4 weights need about 56 per cent of the memory of FP8, 4.5 against 8 bits, at the accuracy cost the model cards above show.

DGX Spark (GB10, 12.1). At 273 GB/s of memory bandwidth, every byte saved per weight shows up in tokens per second. NVIDIA’s playbooks quantise to NVFP4 with Model Optimizer and serve with TensorRT-LLM or vLLM; NVIDIA’s own benchmark of October 2025 runs gpt-oss-120b in MXFP4 through llama.cpp at 55.37 tokens per second of generation, with a 2,048-token prompt at batch 1. GGUF through llama.cpp or Ollama is the quickest start for one user; our DGX Spark sizing article shows what fits.

What we supply

Eurokommerz supplies DGX Spark and the L4, L40S, H200 NVL and RTX PRO Blackwell workstation cards EU-wide with manufacturer warranty, the cards singly or in servers and workstations configured for the model and format you plan to run. Our professional GPU range is the place to start; send us the model, and we will match the card to it.

FAQ