How much VRAM an LLM needs: formulas and tables

The VRAM budget: weights, KV cache and overhead

An LLM needs VRAM for its weights, for the KV cache of every concurrent session and for 1 to 5 GB of CUDA graphs: a 70B model takes 70 GB in FP8 before any cache, and one 128K conversation on Llama 3.1 70B adds 40 GiB. NVIDIA’s NIM engine reserves a share of the card set by gpu_memory_utilization, 0.9 by default in NIM 2.0’s memory documentation of April 2026 (vLLM itself has defaulted to 0.92 since its release 0.20.0 of the same month). Inside that share it places the weights, overhead, peak activations and the KV cache. The cache is allocated greedily: “it expands to fill all remaining space”. The number of concurrent users therefore depends on the memory left after the weights and the overhead. We supply the NVIDIA cards sized in this guide and build AI servers to order with them. Tell us your model and number of users and our team will help.

Weight memory: parameters × bytes per parameter

The formula in NVIDIA’s NIM documentation is weight memory per card = parameters × bytes per parameter / TP, where TP is the number of cards in tensor parallelism. A parameter takes 2 bytes in BF16/FP16, 1 in FP8, and in 4-bit formats half a byte plus its scales, which makes 0.5625 bytes per NVFP4 weight including the block scales. Whole checkpoints come out higher, because the embeddings and output layer stay at 16-bit precision: NVIDIA’s NVFP4 checkpoint of Llama 3.3 70B is 42.7 GB, about 0.61 bytes per parameter, and an INT4 AWQ checkpoint of Llama 3.1 70B is about 39.8 GB. NVIDIA’s FP8 checkpoint of the same 70B model is 72.7 GB for the same reason.

For a mixture-of-experts model, count all parameters, not only the active ones: every expert stays in GPU memory. gpt-oss-120b has 117 billion parameters with 5.1 billion active per token, and its checkpoint, with the experts in MXFP4, is 65.3 GB. The active parameters decide how many bytes each token reads and therefore the speed, as our article on MoE total and active parameters explains.

KV cache per token for 7B, 8B and 70B models

The second part of the memory budget is the KV cache, in which the model stores keys and values for every token it has read. Per token it takes 2 × layers × KV heads × head dimension × bytes per value. The formula uses KV heads rather than attention heads: Llama 3 70B has 64 attention heads but only 8 KV heads. The cache grows linearly with sequence length and with the number of parallel requests.

Llama 2 7B supports 4,096 tokens, so its 32K and 128K cells are arithmetic only.

The 7B model, with the older multi-head attention, needs 512 KiB of cache per token; the 70B model, with grouped-query attention, needs 320 KiB. The 70B model is ten times larger and needs 38% less cache, so the context that fits cannot be read from the parameter count. Taking both tables together, a 70B model in FP16 with a 128K context needs 131 GiB of weights plus 40 GiB of cache, about 171 GiB for one user.

Overhead: CUDA graphs and FP8 profiles that load BF16 weights

CUDA graph capture takes “1 to 5 GB depending on GPU architecture and model size” by NVIDIA’s estimate in the NIM 2.0 memory documentation. Some profiles also need more memory at startup than while serving. In a vLLM FP8 profile of NVIDIA’s vision-language NIM, for example, “FP8 quantization happens on the fly, implying that BF16 weights must be loaded into memory first”. Such a profile saves memory during inference but needs room for BF16 weights at startup, so check the specific profile before you buy.

Concurrent users per card for 8B and 70B models

The tables use our users-per-card rule, as does our guide to users per RTX PRO 6000. They take 90% of card memory, subtract the weights and about 3 GiB per card for CUDA graphs and activations, and divide the remainder by the cache of one session. Card memory is what the driver reports, 95.6 GiB on a 96 GB card and 140.4 GiB on the H200 NVL, and for the other cards the nominal capacity taken as GiB, slightly more than their drivers report; weights are the published checkpoints. NVIDIA’s CUDA Best Practices Guide states that ECC on GDDR memory takes 6.25 per cent of capacity; NVIDIA publishes no figure for the GDDR7 RTX PRO Blackwell cards. We size with the memory the driver reports (nvidia-smi, FB Memory Total) where we know it; check it on your card in the ECC mode you will run. These are memory ceilings; latency cuts them further, and if the seats are virtual desktops rather than one API, MIG and vGPU count differently.

8B-class model, FP8 weights of 8.5 GiB (NVIDIA’s 9.1 GB checkpoint), FP16 KV cache at 128 KiB per token (an FP8 cache halves it and doubles every count). Concurrent sessions:

70B-class model with GQA, cache 320 KiB per token; weights from NVIDIA’s Llama 3.3 70B checkpoints, 39.8 GiB (42.7 GB) in NVFP4 and 67.7 GiB (72.7 GB) in FP8:

On the 48 GB RTX PRO 5000 a 70B model leaves no room for a single session. NVIDIA’s four-bit checkpoint is 42.7 GB, or 39.8 GiB, which only just fits inside the 43.2 GiB that a 0.9 share of the card allows. After about 3 GiB for CUDA graphs and activations, 0.4 GiB remain, less than the 2.5 GiB of one 8K session.

With several cards in tensor parallelism the weights split across the cards and the cache adds up, because Llama 3.3 70B has eight KV heads. By the same rule, two RTX PRO 6000 hold one FP8 copy of the 70B model with 98.4 GiB free for cache, 39 sessions of 8K with a 16-bit cache. The four-card inference node among our AI server configurations holds two such pairs, 78 sessions, or one four-way copy with 264.5 GiB free, 105 sessions (26 at 32K); an FP8 cache about doubles each count. The cards have no NVLink, so a split copy exchanges data over PCIe in every layer, and vLLM disables its custom all-reduce for more than two GPUs without NVLink, which favours two TP=2 pairs over one TP=4 copy. Throughput at peak concurrency is not measured for these layouts, so ask for a load test on your model.

If your model or number of users is not covered above, send them to us and we will run the same calculation for your case before you order.

Measured latency: NIM benchmarks for Llama 3.1 8B

The arithmetic shows what fits in memory; response time is a separate limit. NVIDIA has published these NIM benchmark figures for Llama 3.1 8B on one H100 80 GB, measured with NIM for LLMs 1.8.0, an earlier release (2.0.13 is current in October 2026); they show how input length, precision and the number of requests in flight change latency on one card:

With 250 requests of 20,000 tokens arriving together, the first token takes well over four minutes, because each request waits for the prefills queued ahead of it; the table gives no single-user figure. Long inputs therefore cost time twice, in the prefill itself and in the queue behind it. NIM’s memory warning shows what a shorter maximum context saves: in NVIDIA’s documented example, a limit of 4,096 tokens lowers the estimated memory need from 45.2 to 30.1 GB.

KV cache quantisation: FP8 and NVFP4

NVIDIA’s NVFP4 KV cache in TensorRT-LLM cuts cache memory by up to half against FP8. In NVIDIA’s long-context and cache-reuse scenarios it also improves time to first token by up to 3×, at a small accuracy cost: MMLU-Pro on Qwen3-Coder-480B-A35B is 78.1% with an FP8 cache and 77.4% with NVFP4. NVIDIA introduced it for Blackwell GPUs; SGLang offers an NVFP4 cache as well, and vLLM 0.32.0 of 18 October 2026 states that its NVFP4 KV cache “now runs on SM8x and SM12x”, which includes the RTX PRO Blackwell cards and DGX Spark. Confirm support for your card and engine. A quantised KV cache saves memory, but its speed depends on your engine’s kernels for that format on your GPU, so measure it before you count on it.

What we supply

We supply every card in the sizing tables across the EU, on one contract and invoice and with manufacturer warranty: the RTX PRO 4500 (32 GB) for models up to 13B, the RTX PRO 5000 (48 or 72 GB) for 8B to 13B models with long context, the RTX PRO 6000 (96 GB; Workstation, Max-Q or Server Edition) for a 70B model in FP8 on one card, and the H200 NVL (141 GB HBM3e) for a 70B model with more users or longer contexts per card. They come as cards or in AI servers built to order.

FAQ