Best GPU for fine-tuning in 2026: RTX PRO 6000 editions, H200 NVL or RTX PRO 5000 72 GB

Which card for which fine-tuning method

For most company fine-tuning in 2026, which means LoRA or QLoRA on models from 8B to 70B parameters, one NVIDIA RTX PRO 6000 Blackwell Server Edition in a rack server or one RTX PRO 6000 Max-Q in a workstation is enough. Choose the H200 NVL when the plan includes full fine-tuning or a 70B LoRA on a 16-bit base spread over several cards, because it has 141 GB per card and joins cards over NVLink.

The 96 GB of one RTX PRO 6000 hold a 32B LoRA and a 70B QLoRA with room for activations. The RTX PRO 5000 72 GB is the 300 W option for QLoRA up to 70B and LoRA on 8B models.

NVIDIA H200 product page (1,671 TFLOPS with sparsity, halved by us); NVIDIA RTX PRO Blackwell GPU architecture whitepaper v1.0, Table 4, for the Workstation Edition and Max-Q; RTX PRO 6000 Server Edition product page, which lists 1 PFLOP of BF16 without a sparsity note; RTX PRO 5000 datasheet, which gives no BF16 rate. Read for our fine-tuning articles and re-checked against FACTS on 24 October 2026.

The table leaves out two things on purpose. Sparse rates count only for weights pruned to NVIDIA's 2:4 pattern, and a fine-tune produces dense weights, so we compare dense rates. And bandwidth is in the memory column, because training's large matrix multiplications are limited by the tensor rate, a point our H200 NVL vs RTX PRO 6000 comparison works through with NVIDIA's arithmetic-intensity formula.

Bytes per parameter: the rules behind every sizing

Every memory figure in this article comes from a handful of per-parameter rules, each with a published source. They cover the model states: weights, gradients and optimiser states. Activations come on top and grow with batch size and sequence length.

Papers and documentation as quoted in our GPU memory guide and, for ZeRO-Offload, read on 24 October 2026. QLoRA's 0.516 bytes is 4 bits plus 0.127 bits of quantisation constants; embeddings and the output layer stay in BF16.

Applied to Llama 3.3 70B, with its 70.55 billion parameters, the rules give 1,129 GB of model states for a full fine-tune, 144.4 GB for a rank-16 LoRA on all linear layers and 42.8 GB for the same LoRA on an NF4 base. The 8B and 32B figures, the adapter arithmetic and the effect of gradient checkpointing are in our guide to GPU memory for fine-tuning; here we use the results to choose a card.

The rules decide most purchases before tensor rates do. A run whose model states do not fit has to be sharded over several cards or offloaded, and from that point the link between cards and the host memory matter as much as the card.

RTX PRO 6000: which of the three editions to train on

The three editions carry the same GB202 chip and 96 GB of GDDR7 with ECC. For fine-tuning they differ in three ways: power, cooling and where they may be installed.

Power sets the training rate. The Workstation Edition runs at 600 W, the Max-Q at 300 W, and the Server Edition is configurable up to 600 W. Training behaves like the prompt phase of serving: thousands of tokens pass through each weight that is read, so the tensor rate, and with it power, sets the pace. A Level1Techs forum thread of 5 December 2025 shows the effect on a Workstation Edition limited to 300 W with nvidia-smi -pl 300: llama-bench prompt processing on Llama 3.3 70B in Q8_0 fell from 1,734.59 to 906.64 tokens per second, while token generation moved only from 20.54 to 19.29. The Max-Q is designed for 300 W, and NVIDIA's whitepaper puts its dense BF16 peak at 438.9 TFLOPS, 87 per cent of the Workstation Edition's 503.8. By our reading, a fine-tuning step on a Max-Q takes somewhat longer than on a 600 W card; we found no published fine-tuning benchmark comparing the two.

The Server Edition's clocks are lower. Its product page lists 120 TFLOPS of FP32 against 125 on the Workstation Edition, and 234 TFLOPS of TF32, close to the Workstation Edition's dense 251.9. Both point to a lower clock; NVIDIA publishes no dense BF16 figure for it. Some 2U servers also cap it: Lenovo's ThinkSystem SR650a V4 takes four cards only at 450 W.

Cooling decides the place. The Server Edition is passive and needs a server listed for it. For multi-day runs in continuous operation in a rack, it is the edition we recommend. At a desk, NVIDIA states that the Max-Q "enables up to four GPUs in a single system", while the 600 W Workstation Edition stops at one or two per tower in the makers' specifications. Puget Systems limits its desktops to three Max-Q with a slot gap, and a Level1Techs build log with four Max-Q recorded 91 °C on a card between two others, so four cards need a chassis built for them.

All three editions split into up to four MIG instances of 24 GB. An 8B QLoRA, at 6.4 GB of model states, fits in one instance, so one card can run four small experiments at once; NVIDIA's MIG guide states that "NCCL is currently not supported with MIG", so an instance never joins a multi-card run. Our edition comparison lists the remaining differences.

We supply all three editions as cards, in workstations and in rack servers; tell us the models and the fine-tuning method, and our team replies with the edition and the number of cards that fit.

H200 NVL: when 141 GB and NVLink pay off

The H200 NVL wins on three kinds of run. The first is a full fine-tune that fits one card only with 141 GB: an 8B model with 32-bit Adam needs 128.5 GB of model states, which leaves about 12 GB of the 141 GB on an H200 NVL and does not fit the 96 GB of an RTX PRO 6000 of any edition. With 8-bit Adam the same run drops to about 80 GB and fits an RTX PRO 6000 too.

The second is any run sharded over several cards. A 70B LoRA on a BF16 base needs 72.2 GB per card on two cards with FSDP, and each card then receives about 141 GB of weights per forward and backward pass. Over a 2-way NVLink bridge at 450 GB/s in each direction that takes at least 0.3 seconds; over PCIe Gen5 at 64 GB/s at least 2.2 seconds, by our arithmetic from NVIDIA's link rates. A bridge joins at most four cards, so an eight-card server has two NVLink domains with PCIe between them.

The third is throughput on long queues of runs. NVIDIA's dense BF16 peak is 1.66 times the Workstation Edition's and 1.90 times the Max-Q's, and the H200 NVL's 4.8 TB/s help the memory-bound parts of a step, such as the optimiser update and QLoRA's dequantisation. In MLPerf® Training v6.0, two Dell servers with eight H200 NVL each fine-tuned Llama 2 70B with LoRA to the benchmark's target.

MLPerf® Training v6.0, closed division, available on-premise, Llama 2 70B LoRA, entries 6.0-0046 and 6.0-0047, retrieved from mlcommons.org on 24 September 2026. Results verified by MLCommons Association. Wall-clock time to the benchmark's quality target. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The benchmark fine-tunes Llama 2 70B on the SCROLLS GovReport data until the loss reaches 0.925. The same card model and software took 35 per cent longer in the second server, which had different processors and less host memory, so a published time describes the whole system. We found no LoRA result for the RTX PRO 6000 that we could cite with its entry ID.

The H200 NVL has limits too. It has no FP4 arithmetic, it is a passive card of up to 600 W for servers only, and its five-year NVIDIA AI Enterprise subscription starts 90 days after the card is shipped to the server maker. MIG splits it into up to seven 1g.18gb instances, each large enough for an 8B QLoRA.

RTX PRO 5000 72 GB: QLoRA at 300 W

The RTX PRO 5000 72 GB is an actively cooled 300 W workstation card with 1,344 GB/s. For fine-tuning, its 72 GB sit between the methods. A 70B QLoRA, at 42.8 GB of model states, leaves 29 GB for activations, enough for moderate batch sizes and sequence lengths. A LoRA on an 8B model leaves 55 GB. A LoRA on a 32B model with a BF16 base needs 67.7 GB and leaves only 4 GB, too little to plan on.

Its tensor rate is about half that of the 600 W RTX PRO 6000 Workstation Edition: its datasheet lists 2,064 AI TOPS against 4,000 for the Workstation Edition and 3,511 for the Max-Q, all FP4 with sparsity, and 65 TFLOPS of FP32 against 125. The same run therefore takes longer on it.

The 48 GB version has the same chip and speed. Beside a 70B QLoRA it leaves about 5 GB, and the QLoRA paper's own 65B model on a 48 GB card needed paged optimisers, which move optimiser states to CPU memory when the GPU runs out. For a 70B QLoRA, the 72 GB card is the one to buy.

On paper the Max-Q gives more for the same 300 W: 96 GB against 72 and 1,792 GB/s against 1,344. The RTX PRO 5000 72 GB has one thing the Max-Q lacks, a place on NVIDIA's vGPU list from release 20.2, for Red Hat Enterprise Linux with KVM only. Its MIG mode splits it into two instances of 36 GB according to its datasheet; NVIDIA's MIG guide does not list the 72 GB card yet. Our sibling page on two RTX PRO 5000 72 GB or one RTX PRO 6000 Max-Q compares the two for a desk.

Several cards: data parallel over PCIe, sharding over NVLink

How much the link matters depends on whether the model states fit one card. If they do, each card trains its own replica of the run and only the gradients of the trained weights are exchanged. For a rank-16 LoRA on Llama 3.3 70B those are the 207.1 million adapter weights: 0.41 GB of BF16 gradients per step, and about 0.83 GB of traffic per card by the ZeRO paper's count of 2Ψ for an all-reduce, or around 13 ms at 64 GB/s, by our arithmetic. PCIe carries that without trouble, which is why RTX PRO 6000 Server Edition servers suit data-parallel LoRA and QLoRA.

If the states do not fit, they are sharded, and the weights cross the link on every pass. PyTorch's FSDP in its fully sharded mode all-gathers the parameters "before the forward" and again "before the backward computation", and the ZeRO paper puts the total at 1.5 times the traffic of plain data parallelism. On four RTX PRO 6000 Server Edition cards, a 70B LoRA holds 36.1 GB of model states per card, but each card receives about 212 GB per forward and backward pass, at least 3.3 seconds over PCIe by our arithmetic. Two H200 NVL with a bridge hold the same run at 72.2 GB per card and move their 141 GB over NVLink.

A QLoRA run can also be sharded. Hugging Face's bitsandbytes documentation describes FSDP-QLoRA, which trains "LLMs up to 70B parameters on a dual 24GB GPU system", provided bnb_4bit_quant_storage matches the data type of the other layers. On two RTX PRO 5000 48 GB cards, a 70B QLoRA would then hold 21.4 GB of model states per card by our arithmetic. Our sibling page on the fine-tuning server with 2, 4 or 8 GPUs sets out which configuration runs which model and method, including host memory and NVMe.

FP8 and FP4 training on these cards

All cards in this comparison train in FP8 through NVIDIA's Transformer Engine, which lists FP8 for Hopper, Ada and Blackwell GPUs from compute capability 8.9; the H200 NVL has 9.0 and the RTX PRO Blackwell cards 12.0. NVFP4 is listed for Blackwell only, so the H200 NVL cannot use it. Transformer Engine 2.19, released on 11 September 2026, added NVFP4 stochastic rounding on SM120 and SM121 GPUs, which include the RTX PRO Blackwell cards. MXFP8 is a different matter: the support check in the source code of 2.19 and of 2.20.2, released on 5 October 2026, rejects compute capability 12.0 and above with the message that MXFP8 "is not supported on 12.0+ architectures yet". None of our memory figures assume an FP8 or FP4 saving.

Our recommendation by use case

• LoRA on 8B to 32B models for one department. One or two RTX PRO 6000 Server Edition cards in a rack server, or a Max-Q workstation for a team that wants the hardware at hand.
• QLoRA on a 70B model. Any RTX PRO 6000 edition, about 53 GB left after the model states, or one RTX PRO 5000 72 GB at 300 W with 29 GB left.
• LoRA on a 70B model with a BF16 base. Two H200 NVL with a 2-way bridge, at 72.2 GB per card over NVLink. Four RTX PRO 6000 Server Edition cards work too, at 36.1 GB per card, with slower steps over PCIe.
• Full fine-tuning of an 8B model. One H200 NVL. With an RTX PRO 6000, use 8-bit Adam on one card or FSDP on two, at 64.2 GB per card.
• Full fine-tuning of 32B and 70B models. Eight H200 NVL: a 32B model takes 65.5 GB per card. A 70B model takes 141.1 GB per card, as much as the 141 GB of the card before any activation, so it needs 8-bit Adam (88.2 GB per card by the ZeRO count) or optimiser offload to host memory, about 988 GB at 14 bytes per parameter by the ZeRO-Offload paper's accounting.
• Many small experiments. MIG: four instances of 24 GB on any RTX PRO 6000, seven of 18 GB on an H200 NVL, each for one single-GPU job.

How to buy the card or the server from us

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the RTX PRO 6000 in all three editions, the H200 NVL with its 2-way and 4-way NVLink bridges and the RTX PRO 5000 in 48 and 72 GB, as cards or in AI servers built to order, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Quantities and lead time are confirmed in writing with the quote. If you also want the training platform on top, MLOps on Kubernetes is part of our Private AI/ML service, built by our engineering partner Vixen.UNO.

For a fine-tuning plan we need three things: the base models, the method for each and the sequence length you train at. Send us those details, and we answer with the card, the card count and a written quote.

FAQ