GRPO hardware requirements: GPUs and memory for reinforcement learning on LLMs

GRPO GPU requirements: rollouts plus training

GRPO needs more GPU memory than a LoRA fine-tune of the same model because every training step also generates text: the GPUs hold the training run, a second copy of the weights in an inference engine such as vLLM, and the KV cache of all completions being sampled. By our sizing, an 8B LoRA run fits two NVIDIA RTX PRO 6000 Blackwell Server Edition cards, one generating and one training; a 32B LoRA run fits four; a 70B LoRA run fits eight H200 NVL. Completion length and the number of completions per prompt decide the rollout memory, more than the model does.

The memory rules for supervised LoRA, QLoRA and full fine-tuning are in our guide to GPU memory for fine-tuning, and the servers for those runs in our fine-tuning server guide. This page adds what reinforcement learning puts on top.

What one GRPO step does

GRPO was introduced in the DeepSeekMath paper (arXiv 2402.03300) as "a variant of Proximal Policy Optimization (PPO)" that improves reasoning "while concurrently optimizing the memory usage of PPO". For each prompt the current model samples several completions, a reward function scores them, and each completion is compared with the others for the same prompt; the training step then makes the better ones more likely.

In practice a step has two phases with different hardware profiles. Generation is inference: the model decodes thousands of tokens per completion, limited by memory bandwidth and the KV cache. Training is a forward and backward pass over the prompts and completions, limited by the tensor rate. Hugging Face's TRL library runs the generation phase in vLLM when use_vllm is set, and the trainer streams the updated weights into vLLM after each optimiser step over NCCL, as TRL's vLLM integration guide describes.

The defaults in TRL 1.16.0, the stable release of 15 October 2026, set the starting point. By the GRPOConfig docstring: num_generations is 8 completions per prompt, max_completion_length 512 tokens, beta 0.0 ("If 0.0 (default), the reference model is not loaded"), learning_rate 1e-6, loss_type "dapo", and gradient checkpointing and BF16 are switched on. Reasoning runs usually raise the completion length well beyond 512, and that is where the memory goes.

The memory budget of a GRPO run

Our arithmetic. Training states for a rank-16 LoRA on Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B from our GPU memory guide (8.03, 32.76 and 70.55 billion parameters). KV cache in BF16 from each model's config.json (read 24 October 2026): 32, 64 and 80 layers with 8 KV heads of dimension 128, so 128, 256 and 320 KiB per token, for a rollout of a 512-token prompt and a 4,096-token completion. Activations come on top.

The rollout cache scales with three numbers you choose: prompts per step, completions per prompt and the completion length. Sixteen prompts with eight completions each are 128 rollouts; at 4,096 tokens of completion each, one step decodes up to 524,288 tokens. vLLM does not need to hold all of them at once, since it works through the batch as cache becomes free, but the fewer it holds, the more passes the generation phase needs and the longer each step takes.

The reference model needs memory only when β is above zero, because TRL then computes a KL penalty against the starting model. For a full-parameter run that can mean one more frozen 16-bit copy of the weights; at TRL's default of 0.0 it is not loaded. Our page on reasoning models covers how long reasoning outputs get once the model is trained.

TRL colocate or server mode

TRL's documentation describes two ways to run vLLM. In colocate mode, the default, "vLLM runs inside the trainer process and shares GPU memory with the training model", which "can improve GPU utilization, but may lead to memory contention on the training GPUs". In server mode, "vLLM runs as a separate process on dedicated GPUs and communicates with the trainer via HTTP", which TRL calls "ideal if you have GPUs dedicated to inference"; the server and the trainer "must run on separate CUDA devices". TRL's example starts the server on GPUs 0 to 3 with --tensor-parallel-size 4 and the trainer on GPUs 4 to 7.

Colocate mode gives vLLM a share of each card through vllm_gpu_memory_utilization, 0.3 by default. On one 96 GB RTX PRO 6000 Server Edition and an 8B model, by our arithmetic, that is 28.8 GB, of which 16.1 GB are weights, leaving at most about 12.7 GB for the cache: about 21 rollouts of 4,608 tokens at a time. Raising the share to 0.5 gives vLLM 48 GB and room for about 52 rollouts, and leaves 48 GB for the 16.7 GB of LoRA states and the activations of training. If the run still runs out of memory, TRL recommends vllm_enable_sleep_mode, under which "the vllm parameters and cache will be offloaded during the optimization step".

Server mode spends a whole card on generation. On a second RTX PRO 6000 Server Edition, vLLM's own default of 0.92 gives 88.3 GB, which after the weights holds about 119 rollouts of 4,608 tokens, nearly all 128 of the example step at once, while the trainer has the first card to itself. For a team that runs GRPO regularly we prefer this layout on a server with two cards: no contention, and each card does one kind of work. These counts are upper bounds by our estimate, against listed capacity; vLLM also keeps activations and CUDA graphs inside its share.

We supply servers with two, four or eight cards for this kind of work; send us the model, the completion length and the completions per prompt you plan, and our team replies with the layout and the memory per card.

Three configurations by model size

Our arithmetic, server mode: vLLM at its default 0.92 of the listed 96 or 141 GB per card, minus BF16 weights, divided by the KV cache of a 4,608-token rollout; trainer states from our GPU memory guide, data parallel unless FSDP is named. Upper bounds; activations and vLLM's own overhead come off.

8B. Two cards as above, or one card in colocate mode for experiments. Our entry server with two RTX PRO 6000 Server Edition cards is described on our page to buy a 2× RTX PRO 6000 Server Edition server.

32B. Four RTX PRO 6000 Server Edition cards: vLLM splits the 65.5 GB of weights over two cards with tensor parallelism, and the trainer runs data parallel on the other two, one copy of the 67.7 GB of LoRA states per card with about 28 GB left for activations, or FSDP at about 34 GB per card when sequences get long. Four H200 NVL give more cache and more room per card.

70B. Eight H200 NVL with two 4-way bridges: vLLM runs with tensor parallelism inside one NVLink domain, and the trainer shards the 144.4 GB of LoRA states over the other domain at 36.1 GB per card. Eight RTX PRO 6000 Server Edition cards hold the same layout by memory, but both halves then work over PCIe: the trainer's sharded weights, about 212 GB per card per forward and backward pass by the arithmetic in our fine-tuning server guide, and vLLM's tensor-parallel all-reduce. vLLM disables its custom all-reduce by default on more than two PCIe GPUs, and vLLM 0.30.0 (22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink.

Links between cards and the weight sync

Data-parallel LoRA training tolerates PCIe well: only the adapter gradients are all-reduced, about 0.83 GB per card per step for a rank-16 LoRA on Llama 3.3 70B by our guide's arithmetic. GRPO adds one transfer that supervised fine-tuning does not have: after each optimiser step the trainer sends updated weights to vLLM over NCCL. If the full 16-bit weights travel, as they must for a full-parameter run, that is 141.1 GB per step for a 70B model, at least 2.2 seconds at 64 GB/s over PCIe Gen5 by our arithmetic. On an eight-card H200 NVL server with generation in one NVLink domain and training in the other, that transfer crosses PCIe as well.

Against a generation phase that decodes hundreds of thousands of tokens, a few seconds per step are a small share. What matters more is that NCCL works between all cards: NVIDIA's NCCL documentation states that "CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer", and our P2P guide for the RTX PRO 6000 turns that into an acceptance check. TRL's guide also notes that since vLLM 0.14.0 offline data-parallel scaling is not supported for dense models, so generation for them scales with --tensor-parallel-size.

Full-parameter GRPO

A full-parameter run trains every weight, at 16 bytes per parameter with mixed-precision Adam by the ZeRO paper's count. For Llama 3.1 8B that is 128.5 GB of states, which two RTX PRO 6000 Server Edition cards hold with FSDP at 64.2 GB per card. With a third card for vLLM and a fourth for a second vLLM instance or a reference model, a four-card server fits an 8B full-parameter GRPO run, by our sizing. If β is above zero, the reference model adds 16.1 GB in BF16.

From 32B upwards full-parameter GRPO is a multi-node job. TRL's own SLURM example trains Qwen2.5-72B on four nodes with eight GPUs each under DeepSpeed ZeRO-3, and uses a fifth node with eight GPUs for vLLM with --tensor-parallel-size 8. TRL's GRPO page states that "using ZeRO Stage 3 is required for training such models". For most company projects a LoRA run on one server is the realistic scope; our 8× H200 NVL server page has the training node with 400G for a second node.

Frameworks, run times and where to start

TRL's GRPOTrainer is the most direct route, and our comparison of Unsloth, Axolotl and TRL on several GPUs covers multi-GPU support in each. verl, under the Apache-2.0 licence, describes itself as "a flexible, efficient and production-ready RL training library", trains with "FSDP, FSDP2 and Megatron-LM", generates with "vLLM, SGLang and HF Transformers", lists GRPO among its algorithms and offers "Multi-gpu LoRA RL support to save memory". Unsloth's Qwen3.8 page states that the model can be trained with RL "on 24GB VRAM".

Published run times give a sense of scale. TRL's GRPO example trains Qwen2.5-0.5B-Instruct on DeepMath-103K and states: "Distributed across 8 GPUs, the training takes approximately 1 day." Unsloth reports 1,000 steps and about 4 hours of RL training for gpt-oss-20b on one DGX Spark, as our DGX Spark fine-tuning article quotes. Reward design and completion length change these figures more than the hardware does, so a short pilot on the target server is the way to fix the step time.

Both phases keep the cards busy, so plan power for training load: about 1.8 kW at the wall for two RTX PRO 6000 Server Edition cards with two AMD EPYC 9135, about 3.4 kW for four with two EPYC 9355, and about 6.3 kW for eight H200 NVL with two EPYC 9555, by our estimate (cards at 600 W, processors, plus 15 per cent); a measurement on the delivered system confirms the figure.

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. The server is assembled and tested before delivery and delivered in the EU on one contract and invoice, with manufacturer warranty on every component; quantities and lead time are confirmed in writing with the quote. Setting up TRL or verl with vLLM, the reward functions and the serving of the trained model are part of our Private AI/ML service with our engineering partner Vixen.UNO. Tell us about the run you plan in the form below.

FAQ