How much GPU memory fine-tuning needs: full fine-tuning, LoRA and QLoRA for 8B, 32B and 70B models

What fills the memory when you train

A full fine-tune with mixed-precision Adam needs about 16 bytes of GPU memory per parameter before activations, LoRA on a BF16 base about 2 and QLoRA under 1. For Llama 3.3 70B that comes to 1,129 GB, 144.4 GB and 42.8 GB of model states. By our arithmetic an 8B LoRA (16.7 GB) fits a 32 GB card, a 70B QLoRA fits one 72 or 96 GB card, or a 48 GB card with about 5 GB to spare, and a 70B LoRA on a BF16 base needs two cards. For each of these runs we name the card that fits or size a server built to order: describe the run you plan, and our team will help.

Serving a model needs memory for the weights and the KV cache, as our VRAM guide sets out; training needs more. Hugging Face’s documentation lists the weights, a gradient for every weight, the optimiser states, the forward activations, which “are computed in the forward pass and cached for the backward pass”, and temporary tensors. Weights, gradients and optimiser states are the model states: they depend only on the parameter count and the method, so they can be calculated in advance. Activations depend on the batch.

The standard accounting is in the ZeRO paper (arXiv 1910.02054). Mixed-precision training with Adam, it says, “requires enough memory to hold an fp16 copy of the parameters and the gradients, with memory requirements of 2Ψ and 2Ψ bytes respectively. In addition, it needs to hold the optimizer states: an fp32 copy of the parameters, momentum and variance, with memory requirements of 4Ψ, 4Ψ, and 4Ψ bytes, respectively.” That is 2 + 2 + 12 bytes, in the paper’s words “2Ψ + 2Ψ + KΨ = 16Ψ bytes of memory requirement”, where Ψ is the number of parameters. BF16 is also 2 bytes, so the count holds for BF16.

Full fine-tuning: 16 bytes per parameter

A full fine-tune trains every weight, so every weight carries all 16 bytes. With the parameter counts Hugging Face reports for the three checkpoints, 8.03, 32.76 and 70.55 billion, that is 128, 524 and 1,129 GB of model states for Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B by our arithmetic. Hugging Face’s documentation counts the gradients in FP32 and lists 6 bytes of weights, 4 of gradients and 8 of Adam states, 18 in all: 145, 590 and 1,270 GB. Our fine-tuning or RAG comparison prints both counts as a range.

With 32-bit Adam, only the 8B model’s states fit on a single card we supply, the H200 NVL, with about 12 GB to spare. Hugging Face documents one way to reduce this: a quantised Adam from bitsandbytes, optim="adamw_bnb_8bit" in its Trainer, stores the optimiser states in 2 bytes per parameter instead of 8, which saves 48 GB on the 8B model. Its states then come to about 80 GB by the ZeRO count, which fits one RTX PRO 6000 with about 16 GB left, or 96 GB by Hugging Face’s, which would take the card’s whole listed 96 GB. The other way, sharding, is covered below.

Activations, and what gradient checkpointing trades

Activations grow with batch size and sequence length. In the ZeRO paper’s example, a 1.5B-parameter GPT-2 “trained with sequence length of 1K and batch size of 32 requires about 60 GB of memory” for activations, two and a half times the 24 GB of its model states.

Gradient checkpointing, which the paper calls activation checkpointing, trades computation for that memory: it stores only some activations and recomputes the rest during the backward pass. By the ZeRO paper’s estimate it cuts the activations of that example from 60 GB to about 8 GB, “at the expense of 33% re-computation overhead”. Hugging Face’s training guide quotes “slower training speed (~20%)”; in its Trainer the switch is gradient_checkpointing=True. The model states stay the same.

With adapters, activations outweigh the adapter itself. The QLoRA paper estimated, for a 7B model at batch size 1, that “the LoRA input gradients have a memory footprint of 567 MB while the LoRA parameters take up only 26 MB”, against 5,048 MB for the 4-bit base; gradient checkpointing brought those gradients down to about 18 MB per sequence. Activations depend on batch size and sequence length, so our tables leave them out: what the model states leave free is the budget for batch size times sequence length.

LoRA: freeze the base, train an adapter

LoRA (arXiv 2106.09685) “freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture”. The frozen weights need neither gradients nor optimiser states: “we do not need to calculate the gradients or maintain the optimizer states for most parameters”.

An adapter of rank r on a layer with m inputs and n outputs adds r × (m + n) parameters. Hugging Face’s PEFT library puts adapters on every linear layer except the output layer with target_modules="all-linear", and the QLoRA paper found that “LoRA on all linear transformer block layers are required to match full finetuning performance”. At rank 16, an example value, our arithmetic from each model’s layer dimensions gives 41.9 million trainable parameters for Llama 3.1 8B, 134.2 million for Qwen3-32B and 207.1 million for Llama 3.3 70B: 0.52, 0.41 and 0.29 per cent of the weights. At 16 bytes per parameter they need 0.7, 2.1 and 3.3 GB; the frozen BF16 base, at 2 bytes per parameter, accounts for nearly all of the memory: 16, 66 and 141 GB. At rank 64, the setting the QLoRA paper used for its Guanaco models, the adapter figures quadruple.

QLoRA: a 4-bit base

QLoRA (arXiv 2305.14314) keeps the adapters and stores the frozen base in 4 bits. The paper states: “We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” Three mechanisms do the work. 4-bit NormalFloat (NF4) is a data type designed for normally distributed weights. Double quantisation stores the quantisation constants in 8 bits, cutting them “from 32/64 = 0.5 bits, to 8/64 + 32/(64 · 256) = 0.127 bits” per parameter, so a quantised weight costs about 4.13 bits, or 0.516 bytes. In Hugging Face Transformers, NF4 and double quantisation both have to be requested (bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True); without double quantisation a weight costs 4.5 bits and the 70B states come to about 46.0 GB. Paged optimisers use NVIDIA unified memory to move optimiser states to CPU memory when the GPU runs out and back for the update step; the paper calls them “critical to do 33B/65B QLORA tuning on a single 24/48GB GPU”.

Only linear layers are quantised, and for causal language models Hugging Face notes that “the last lm_head is kept in its original dtype”; the input embedding is not a linear layer either, so in these three models 1.05, 1.56 and 2.10 billion parameters stay in BF16. The 4-bit weights are dequantised to the compute type for every matrix multiplication, BF16 once bnb_4bit_compute_dtype is set to it (the default is FP32), and gradients pass through them but are kept only for the adapters. Hugging Face’s bitsandbytes guide is explicit that “8 and 4-bit training is only supported for training extra parameters”: a 4-bit model is no route to a full fine-tune.

Model states only, no activations. Our arithmetic, GB = 10⁹ bytes: 16 bytes per parameter for full fine-tuning (ZeRO paper); a frozen base at 2 bytes (BF16), or, for QLoRA, linear layers at 0.516 bytes (NF4 with double quantisation, QLoRA paper) with the embeddings, output layer and normalisation layers at 2 bytes; plus a rank-16 adapter on all linear layers at 16 bytes per trainable parameter. Parameter counts from Hugging Face. Real footprints are higher; for its 4-bit 7B base, the QLoRA paper gives 5,048 MB, where this arithmetic gives about 3.9 GB.

Which GPU fits which fine-tuning run

What each card has left after the model states is the budget for activations, the CUDA context and temporary buffers.

Memory left after the model states in the table above, by our arithmetic. All cards at their listed capacity of 32, 48, 72, 96 and 141 GB, which keeps the figures on the safe side: the driver of an RTX PRO 6000 reports 95.6 GiB and that of an H200 NVL 140.4 GiB.

The RTX PRO 6000 takes LoRA up to 32B, with more than a quarter of the card free, and QLoRA up to 70B with about 53 GB left. The 72 GB RTX PRO 5000 holds a 70B QLoRA with room to spare, but a 32B LoRA on a BF16 base leaves it only 4 GB. The 48 GB card leaves about 5 GB beside a 70B QLoRA, before the CUDA context and activations; the QLoRA paper’s own 65B model on 48 GB needed paged optimisers. On the 32 GB RTX PRO 4500, QLoRA brings the 32B within reach with 11 GB left, and an 8B LoRA leaves 15 GB, where batch size decides. NVIDIA states that none of its DGX Spark fine-tuning examples “can run on a 32 GB consumer GPU”; one of them is an 8B LoRA at batch size 4 with 2,048-token sequences. A 70B LoRA on a BF16 base, at 144.4 GB, exceeds even the 141 GB of an H200 NVL, so we would split it over two cards. Which card to choose for each method is set out in our guide to the best GPU for fine-tuning.

DGX Spark. NVIDIA’s product page states: “Fine-tune AI models up to 70 billion parameters.” Its PyTorch fine-tuning playbook, as of September 2026, shows the configurations: a full fine-tune of Llama 3.2 3B, LoRA on Llama 3.1 8B and QLoRA on Llama 3.1 70B on one Spark, and LoRA on the 70B with FSDP under the heading “Run on two Sparks”. Our arithmetic is consistent with this. The 144.4 GB of a 70B LoRA exceed the unified memory of one 128 GB DGX Spark, which the operating system shares, and the model states of an 8B full fine-tune with 32-bit Adam alone come to about all of it. DGX Spark systems with 64 GB, available from partners since 23 October 2026 (NVIDIA blog, 21 October 2026), have half that memory: a 70B QLoRA, at 42.8 GB of model states, leaves them about 21 GB for the operating system and activations. Our DGX Spark memory article has the details.

For the 70B LoRA set-up in NVIDIA’s playbook, we quote two 128 GB DGX Sparks. Tell us the model and the method you have in mind, and we say whether one Spark, two or a GPU server suits it.

More than one card: FSDP and ZeRO-3

When the states do not fit one card, shard them. PyTorch’s FSDP “reduces GPU memory footprint by sharding model parameters, gradients, and optimizer states”; DeepSpeed’s ZeRO partitions the optimiser states in stage 1, the gradients as well in stage 2 and the 16-bit parameters in stage 3, where, in the ZeRO paper’s words, “Memory reduction is linear with DP degree”. By our arithmetic the 8B full fine-tune then needs 64.2 GB of model states per card on two cards, the 32B 65.5 GB per card on eight, and the 70B LoRA on a BF16 base 72.2 GB per card on two, each plus activations and the parameters being gathered. A 70B full fine-tune still needs 141 GB per card across eight, which exceeds an RTX PRO 6000 and the 141 GB of an H200 NVL as well, so it calls for more cards or for offloading: ZeRO-Offload moves optimiser and gradient states to CPU memory, ZeRO-Infinity to NVMe as well.

Sharding costs traffic. FSDP all-gathers the sharded parameters before the forward and the backward pass, and for a full fine-tune the ZeRO paper puts the total at 1.5 times the traffic of plain data parallelism. Whether that traffic runs over NVLink or PCIe is the subject of our H200 NVL and RTX PRO 6000 comparison.

On a server with four RTX PRO 6000 Server Edition, FSDP brings the 8B full fine-tune to 32.1 GB of model states per card, the 70B LoRA on a BF16 base to 36.1 GB and, with 8-bit Adam, a 32B full fine-tune to about 81.9 GB, by our arithmetic. A 70B full fine-tune, at 282 GB per card, does not fit, and the same server can instead run four QLoRA jobs of up to 70B side by side, one per card. We supply AI servers built to order as multi-GPU nodes for fine-tuning, with H200 NVL and NVLink bridges or RTX PRO 6000 Server Edition. Send us the model size and the method you plan to train with; the configuration and quote follow within one business day.

FP8 and the GPU generation

NVIDIA’s Transformer Engine, which runs FP8 training, lists in its README of September 2026 “Support for FP8 on NVIDIA Hopper, Ada, and Blackwell GPUs” and states that “FP8 features require Compute Capability 8.9+”, which covers every card here: the H200 NVL at 9.0, the RTX PRO Blackwell cards at 12.0 and DGX Spark at 12.1. MXFP8 and NVFP4 are listed for Blackwell only, but Transformer Engine’s own support check in release 2.19 (September 2026) refuses MXFP8 on compute capability 12.0 and above, “not supported on 12.0+ architectures yet”: the RTX PRO Blackwell cards and DGX Spark get NVFP4, not MXFP8. NVIDIA’s FP8 primer presents FP8 as a route to “higher throughput of matrix multiplies”; our tables count no FP8 saving.

What we supply

We supply every card in the tables above (the RTX PRO 4500 as a workstation card, the RTX PRO 5000 in 48 and 72 GB, the RTX PRO 6000 in all three editions including the Server Edition, and the H200 NVL) and the DGX Spark Founders Edition with 128 GB across the EU with manufacturer warranty, on one EU contract and invoice. The cards come on their own or in workstations and servers built to order for training on one GPU or several. If you also want the platform around them, MLOps on Kubernetes (KServe, Kubeflow) is part of our Private AI/ML service, built by our engineering partner Vixen.UNO under the same contract.

FAQ