How much VRAM an LLM needs: formulas and tables

What the memory budget is made of

NVIDIA’s NIM engine claims a gpu_memory_utilization share of the card (0.9 by default) and lays out, inside it: weights, overhead, peak activations and the KV cache. The cache is allocated greedily: “it expands to fill all remaining space”. So the number of concurrent users is not a setting. It is a remainder.

Weights: a single multiplication

Official formula: weight memory per card = parameters × bytes per parameter / TP (tensor-parallel card count). Bytes per parameter: 2 for BF16/FP16, 1 for FP8, 0.5 for INT4 and NVFP4 (real NVFP4 checkpoints land nearer 0.56 once the block scales and the unquantised embeddings are counted: NVIDIA’s Llama 3.3 70B FP4 file is 42.7 GB).

KV cache, and why a 7B eats more than a 70B

The second half of the bill: the model stores keys and values for every token it has read. Formula: 2 × layers × KV heads × head dimension × bytes per value, per token (KV heads, not attention heads: Llama 3 70B has 64 attention heads but only 8 KV heads); volume grows linearly with sequence length and with parallel requests.

The counter-intuitive part: the 7B with old multi-head attention spends 512 KB per token, the 70B with grouped-query attention: 320 KB. Ten times the model, 38% less cache. “How much context fits” does not follow from parameter count. Add the tables up: a 70B in FP16 with a 128K context is 140 GB of weights + 40 GB of cache, 180 GB for one person.

The overhead you only see in logs

CUDA graph capture takes “1 to 5 GB depending on GPU architecture and model size” by NVIDIA’s own estimate. And a trap that never makes a slide: for some profiles “FP8 quantization happens on the fly, implying that BF16 weights must be loaded into memory first”. Eight bits save memory during inference, but startup demands BF16-sized room. Check the specific profile before you buy.

Card × model × context × users

Method: card capacity × 0.9, minus weights, minus ~2 GB for graphs and activations; divide the remainder by one session’s cache. These are memory ceilings; latency cuts them further, and if the seats are virtual desktops rather than one API, MIG and vGPU count differently.

8B-class model, FP8 weights (8 GB), FP16 KV cache at 128 KB/token (an FP8 cache halves it and doubles every count). Concurrent sessions:

70B-class model with GQA, cache 320 KB/token:

Look at the 48 GB row: the card technically “holds” a 70B at four bits, but only 6.2 GB survive the weights: two sessions of 8K tokens, and a single 32K context does not fit at all.

What real measurements add

Arithmetic gives the memory ceiling; latency rules the experience. NVIDIA’s own NIM benchmarking numbers (NIM 1.8.0) for Llama 3.1 8B on one H100 80 GB:

At a 20,000-token input the first token takes well over four minutes: the service is technically alive, and nobody will use it. The most direct lever is prompt length. NIM’s own log hands you the number: cutting context to 4,096 freed 15.1 GB in the documented example.

Cache quantisation: claim vs measurement

NVIDIA’s NVFP4 KV cache in TensorRT-LLM cuts cache memory by up to half against FP8 and, in NVIDIA’s long-context and cache-reuse scenarios, improves time to first token by up to 3×, at a small accuracy cost (MMLU-Pro on Qwen3-480B: 78.2% BF16 → 77.4% NVFP4); it arrived first on the B200 class, so confirm support for your card. Format alone guarantees nothing: the same idea in llama.cpp (a q4_0 KV cache) on unified memory collapsed prompt speed from 282.7 to 21.3 tok/s in one owner’s test because that path had no fused kernels. Hardware-accelerated NVFP4 in TensorRT-LLM wins; software unpacking loses. The engine and the silicon deliver the benefit, not the format.

What we supply

Eurokommerz supplies the full ladder across the EU: RTX PRO 4500 (32 GB) for models up to 13B, RTX PRO 5000 (48 / 72 GB) for 8–13B with long context, RTX PRO 6000 (96 GB) for a 70B in FP8 on one card, and H200 NVL (141 GB HBM3e) for long-context 70B serving at scale.

FAQ