How much GPU memory fine-tuning needs: full fine-tuning, LoRA and QLoRA for 8B, 32B and 70B models

What fills the memory when you train

Serving a model needs memory for the weights and the KV cache, as our VRAM guide sets out. Training adds more. Hugging Face’s documentation lists the weights, a gradient for every weight, the optimiser states, the forward activations, which “are computed in the forward pass and cached for the backward pass”, and temporary tensors. Weights, gradients and optimiser states are the model states: they depend only on the parameter count and the method, so they can be calculated in advance. Activations depend on the batch.

The standard accounting is in the ZeRO paper (arXiv 1910.02054). Mixed-precision training with Adam, it says, “requires enough memory to hold an fp16 copy of the parameters and the gradients, with memory requirements of 2Ψ and 2Ψ bytes respectively. In addition, it needs to hold the optimizer states: an fp32 copy of the parameters, momentum and variance, with memory requirements of 4Ψ, 4Ψ, and 4Ψ bytes, respectively.” That is 2 + 2 + 12 bytes, in the paper’s words “2Ψ + 2Ψ + KΨ = 16Ψ bytes of memory requirement”, where Ψ is the number of parameters. BF16 is also 2 bytes, so the count holds for BF16.

Full fine-tuning: 16 bytes per parameter

A full fine-tune trains every weight, so every weight carries all 16 bytes. With the parameter counts Hugging Face reports for the three checkpoints, 8.03, 32.76 and 70.55 billion, that is 128, 524 and 1,129 GB of model states for Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B by our arithmetic. Hugging Face’s documentation counts the gradients in FP32 and lists 6 bytes of weights, 4 of gradients and 8 of Adam states, 18 in all: 145, 590 and 1,270 GB. Our fine-tuning or RAG comparison prints both counts as a range.

With 32-bit Adam, only the 8B model’s states fit on a single card we supply, the H200 NVL, with 22 GB to spare. Hugging Face documents one lever: a quantised Adam from bitsandbytes, optim="adamw_bnb_8bit" in its Trainer, stores the optimiser states in 2 bytes per parameter instead of 8, which saves 48 GB on the 8B model. Its states then come to about 80 GB by the ZeRO count, which fits one RTX PRO 6000 with about 22 GB left, or 96 GB by Hugging Face’s, which leaves about 6 GB. The other lever is sharding, below.

Activations, and what gradient checkpointing trades

Activations grow with batch size and sequence length. The ZeRO paper gives the scale: a 1.5B-parameter GPT-2 “trained with sequence length of 1K and batch size of 32 requires about 60 GB of memory” for activations, two and a half times the 24 GB of its model states.

Gradient checkpointing, which the paper calls activation checkpointing, trades computation for that memory: it stores only some activations and recomputes the rest during the backward pass. By the ZeRO paper’s estimate it cuts the activations of that example from 60 GB to about 8 GB, “at the expense of 33% re-computation overhead”. Hugging Face’s training guide quotes “slower training speed (~20%)”; in its Trainer the switch is gradient_checkpointing=True. The model states stay the same.

With adapters, activations outweigh the adapter itself. The QLoRA paper estimated, for a 7B model at batch size 1, that “the LoRA input gradients have a memory footprint of 567 MB while the LoRA parameters take up only 26 MB”, against 5,048 MB for the 4-bit base; gradient checkpointing brought those gradients down to about 18 MB per sequence. Activations depend on batch size and sequence length, so our tables leave them out: what the model states leave free is the budget for batch size times sequence length.

LoRA: freeze the base, train an adapter

LoRA (arXiv 2106.09685) “freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture”. The frozen weights need neither gradients nor optimiser states: “we do not need to calculate the gradients or maintain the optimizer states for most parameters”.

An adapter of rank r on a layer with m inputs and n outputs adds r × (m + n) parameters. Hugging Face’s PEFT library puts adapters on every linear layer except the output layer with target_modules="all-linear", and the QLoRA paper found that “LoRA on all linear transformer block layers are required to match full finetuning performance”. At rank 16, an example value, our arithmetic from each model’s layer dimensions gives 41.9 million trainable parameters for Llama 3.1 8B, 134.2 million for Qwen3-32B and 207.1 million for Llama 3.3 70B: 0.52, 0.41 and 0.29 per cent of the weights. At 16 bytes per parameter they need 0.7, 2.1 and 3.3 GB; the frozen BF16 base, at 2 bytes per parameter, is nearly all of the bill: 16, 66 and 141 GB. At rank 64, the setting the QLoRA paper used for its Guanaco models, the adapter figures quadruple.

QLoRA: a 4-bit base

QLoRA (arXiv 2305.14314) keeps the adapters and stores the frozen base in 4 bits. Its claim: “We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” Three mechanisms do the work. 4-bit NormalFloat (NF4) is a data type designed for normally distributed weights. Double quantisation stores the quantisation constants in 8 bits, cutting them “from 32/64 = 0.5 bits, to 8/64 + 32/(64 · 256) = 0.127 bits” per parameter, so a quantised weight costs about 4.13 bits, or 0.516 bytes. Paged optimisers use NVIDIA unified memory to move optimiser states to CPU memory when the GPU runs out and back for the update step; the paper calls them “critical to do 33B/65B QLORA tuning on a single 24/48GB GPU”.

Only linear layers are quantised, and for causal language models Hugging Face notes that “the last lm_head is kept in its original dtype”; the input embedding is not a linear layer either, so in these three models 1.05, 1.56 and 2.10 billion parameters stay in BF16. The 4-bit weights are dequantised to BF16 for every matrix multiplication, and gradients pass through them but are kept only for the adapters. Hugging Face’s bitsandbytes guide is explicit that “8 and 4-bit training is only supported for training extra parameters”: a 4-bit model is no route to a full fine-tune.

Model states only, no activations. Our arithmetic, GB = 10⁹ bytes: 16 bytes per parameter for full fine-tuning (ZeRO paper); a frozen base at 2 bytes (BF16), or, for QLoRA, linear layers at 0.516 bytes (NF4 with double quantisation, QLoRA paper) with the embeddings, output layer and normalisation layers at 2 bytes; plus a rank-16 adapter on all linear layers at 16 bytes per trainable parameter. Parameter counts from Hugging Face. Real footprints run higher: for its 4-bit 7B base the QLoRA paper gives 5,048 MB, where this arithmetic gives about 3.9 GB.

Which card holds which run

What each card has left after the model states is the budget for activations, the CUDA context and temporary buffers.

Memory left after the model states in the table above, by our arithmetic. RTX PRO 6000 and H200 NVL at their driver-visible 95.6 and 140.4 GiB, which are 102.6 and 150.8 GB; RTX PRO 4500 and 5000 at nominal capacity.

The RTX PRO 6000 takes LoRA up to 32B, with a third of the card free, and QLoRA up to 70B. The 72 GB RTX PRO 5000 holds a 70B QLoRA with room to spare, but a 32B LoRA on a BF16 base leaves it only 4 GB. The 48 GB card leaves about 5 GB beside a 70B QLoRA, before the CUDA context and activations; the QLoRA paper’s own 65B model on 48 GB needed paged optimisers. On the 32 GB RTX PRO 4500, QLoRA brings the 32B within reach with 11 GB left, and an 8B LoRA leaves 15 GB, where batch size decides: NVIDIA states that none of its DGX Spark fine-tuning examples, an 8B LoRA at batch size 4 with 2,048-token sequences among them, “can run on a 32 GB consumer GPU”. A 70B LoRA on a BF16 base leaves about 6 GB even on the H200 NVL; we would split it over two cards.

DGX Spark. NVIDIA’s product page promises: “Fine-tune AI models up to 70 billion parameters.” Its PyTorch fine-tuning playbook, as of September 2026, shows how: a full fine-tune of Llama 3.2 3B, LoRA on Llama 3.1 8B and QLoRA on Llama 3.1 70B on one unit, and LoRA on the 70B with FSDP under the heading “Run on two Sparks”. The arithmetic agrees: the 144.4 GB of a 70B LoRA exceed the unit’s 128 GB of unified memory, which the operating system shares, and the model states of an 8B full fine-tune with 32-bit Adam alone come to about all of it. Our DGX Spark memory article has the details.

More than one card: FSDP and ZeRO-3

When the states do not fit one card, shard them. PyTorch’s FSDP “reduces GPU memory footprint by sharding model parameters, gradients, and optimizer states”; DeepSpeed’s ZeRO partitions the optimiser states in stage 1, the gradients as well in stage 2 and the 16-bit parameters in stage 3, where, in the ZeRO paper’s words, “Memory reduction is linear with DP degree”. By our arithmetic the 8B full fine-tune then needs 64.2 GB of model states per card on two cards, the 32B 65.5 GB per card on eight, and the 70B LoRA on a BF16 base 72.2 GB per card on two, each plus activations and the parameters being gathered. A 70B full fine-tune still needs 141 GB per card across eight, which exceeds an RTX PRO 6000 and leaves under 10 GB on an H200 NVL, so it calls for more cards or for offloading: ZeRO-Offload moves optimiser and gradient states to CPU memory, ZeRO-Infinity to NVMe as well.

Sharding costs traffic. FSDP all-gathers the sharded parameters before the forward and the backward pass, and for a full fine-tune the ZeRO paper puts the total at 1.5 times the traffic of plain data parallelism. Whether that traffic runs over NVLink or PCIe is the subject of our H200 NVL and RTX PRO 6000 comparison.

FP8 and the GPU generation

NVIDIA’s Transformer Engine, which runs FP8 training, lists in its README of September 2026 “Support for FP8 on NVIDIA Hopper, Ada, and Blackwell GPUs” and states that “FP8 features require Compute Capability 8.9+”, which covers every card here: the H200 NVL at 9.0, the RTX PRO Blackwell cards at 12.0 and DGX Spark at 12.1. MXFP8 and NVFP4 are listed for Blackwell only, but Transformer Engine’s own support check, as of September 2026, refuses MXFP8 on compute capability 12.0 and above, “not supported on 12.0+ architectures yet”: the RTX PRO Blackwell cards and DGX Spark get NVFP4, not MXFP8. NVIDIA’s FP8 primer presents FP8 as a route to “higher throughput of matrix multiplies”; our tables count no FP8 saving.

What we supply

Eurokommerz supplies DGX Spark and the RTX PRO 6000, RTX PRO 5000, RTX PRO 4500 and H200 NVL EU-wide with manufacturer warranty, the cards singly or in workstations and servers configured for training on one GPU or several. Our GPU range is the place to start; send us the model and the method, and we will match the card to the run.

FAQ