H200 NVL or RTX PRO 6000 for fine-tuning and training: memory, bandwidth, tensor rates and the link between cards

Memory, power and cooling of the two cards

For fine-tuning, LoRA up to 32B and QLoRA up to 70B fit both cards, while the model states of a full 8B fine-tune with 32-bit Adam fit one H200 NVL but not one RTX PRO 6000. In dense BF16 the H200 NVL reaches 835.5 TFLOPS (NVIDIA lists 1,671 with sparsity), 1.66 times the 503.8 of the RTX PRO 6000 Workstation Edition. Of the two cards, only the H200 NVL has NVLink, which matters once a run is sharded over several cards.

The H200 NVL and the air-cooled RTX PRO 6000 Server Edition are passive cards that depend on the server’s fans; NVIDIA also lists a single-slot, liquid-cooled Server Edition. The Workstation Edition carries the same chip and 96 GB with its own fans, for a desk, and the Max-Q version is rated at 300 W. For training, they differ in memory, tensor throughput and the link between cards.

NVIDIA H200 product page; RTX PRO 6000 datasheets and Server Edition product page; NVIDIA MIG user guide, updated 4 November 2026. Visible memory as reported by nvidia-smi. The NVLink rate is NVIDIA’s total for both directions.

Which runs fit in 141 and 96 GB

Our fine-tuning memory guide works out the model states, meaning weights, gradients and optimiser states, for Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B. Set against the listed capacity, 141 GB on the H200 NVL and 96 GB on the RTX PRO 6000, they leave this much for activations.

Our arithmetic, GB = 10⁹ bytes: rank-16 adapters on all linear layers, 16 bytes per trained parameter, a BF16 base for LoRA and, for QLoRA, NF4 linear layers with the embeddings and output layer in BF16; full fine-tuning at 16 bytes per parameter with 32-bit Adam and 10 with 8-bit Adam, by the ZeRO count. What is left, counted from the listed 141 and 96 GB, must hold activations, the CUDA context and temporary buffers.

LoRA up to 32B and QLoRA up to 70B fit both cards with room for activations. For these runs, the H200 NVL’s extra 48 GB allow longer sequences and larger batches, but no run that the RTX PRO 6000 cannot do. Capacity makes the difference in two cases. The model states of a full 8B fine-tune with 32-bit Adam fit one H200 NVL but not one RTX PRO 6000, unless 8-bit Adam cuts them to about 80 GB. A 70B LoRA on a BF16 base, at 144.4 GB, does not fit even one H200 NVL, so it needs two cards of either kind, and then the link between the cards matters.

We supply both cards, and for a two-card run on the H200 NVL the two-way NVLink bridge comes in the same quote. Tell us the model and the method, and we reply within one business day.

Why bandwidth counts for less in training

In serving, bandwidth limits speed, because generating a token for one user reads every weight of a dense model. The H200 NVL’s 4.8 TB/s against 1,792 GB/s on the Workstation Edition and 1,597 GB/s on the Server Edition, 2.7 and 3.0 times, therefore sets the pace, as our inference comparison shows. Training behaves like the prompt phase instead. NVIDIA’s performance guide defines arithmetic intensity as operations per byte of memory traffic and says an algorithm “is math limited on a given processor if the algorithm’s arithmetic intensity is higher than the processor’s ops:byte ratio”. In its own examples a linear layer at batch size 512 reaches 315 FLOPS per byte and is limited by arithmetic; the same layer at batch size 1 reaches 1 FLOPS per byte and is limited by memory.

A training step pushes thousands of tokens through each layer at once. By the same formula, the up-projection of Llama 3.1 8B, from 4,096 to 14,336, reaches about 1,250 FLOPS per byte on one 2,048-token sequence in BF16, by our arithmetic. The ops:byte ratio is 174 on the H200 NVL (835.5 dense BF16 TFLOPS over 4.8 TB/s) and 281 on the RTX PRO 6000 Workstation Edition (503.8 over 1.792 TB/s). The large matrix multiplications are therefore compute-bound on both cards, so for them the tensor rate decides, and it favours the H200 NVL by 1.66 times rather than 2.7. Bandwidth still matters for the operations NVIDIA lists as limited by memory, such as activation functions and layer normalisation, and, by our reasoning, for three more: the optimiser step, which touches every state once with little arithmetic; the rank-16 adapter multiplications, at about 16 FLOPS per byte by the same formula; and, in QLoRA, the dequantisation of every 4-bit weight before each multiplication. These parts favour the H200 NVL by the bandwidth ratio, not the tensor ratio. We found no published fine-tuning benchmark that runs both cards under the same conditions, so we give no measured ratio.

Peak tensor rates: BF16, FP8, FP4 and TF32

TFLOPS unless marked. NVIDIA H200 product page, rates “With sparsity”, halved by us for dense; NVIDIA RTX PRO Blackwell GPU architecture whitepaper v1.0, Table 4, Workstation Edition at its 2,617 MHz boost clock; RTX PRO 6000 Server Edition product page, which does not say whether its figures include sparsity.

Sparse rates apply only to weights pruned to NVIDIA’s 2:4 pattern, where “In each contiguous block of four values, two values must be zero”. A normal fine-tune produces dense weights, so compare the dense rates. The RTX PRO 6000’s BF16 and FP8 figures are those NVIDIA prints with FP32 accumulation; NVIDIA’s H200 page does not state the accumulate mode, but its Hopper architecture blog gives Hopper’s BF16 Tensor rates with FP32 accumulate. In peak dense rates, the H200 NVL leads the Workstation Edition 835.5 to 503.8 TFLOPS in BF16 and 1,670.5 to 1,007.6 in FP8, 1.66 times in both. Against the 300 W Max-Q, which NVIDIA’s RTX PRO whitepaper puts at 438.9 dense BF16 TFLOPS, the ratio is 1.90; for the Server Edition we found no dense rate from NVIDIA. The Server Edition’s 1, 2 and 4 PFLOPS match the Workstation Edition’s sparse figures, rounded, but its TF32 figure of 234 TFLOPS is close to the Workstation Edition’s dense 251.9, and its 120 TFLOPS of FP32 sit below the Workstation Edition’s 125, which points to lower clocks. By our reading, the H200 NVL therefore leads the Server Edition by somewhat more than 1.66 times. QLoRA dequantises its 4-bit weights to BF16 for every multiplication, so the BF16 row applies to it too. The H200’s BF16 TFLOPS and compute capability beside those of other NVIDIA cards are in our comparison of GPU TFLOPS in FP4, FP8 and BF16.

FP8 and FP4 training. Both cards train in FP8 through NVIDIA’s Transformer Engine, whose README, as of September 2026, lists “Support for FP8 on NVIDIA Hopper, Ada, and Blackwell GPUs”. MXFP8 and NVFP4 are listed for Blackwell GPUs only, so neither is an option on the H200 NVL; on the RTX PRO 6000 only NVFP4 is, because Transformer Engine’s own support check in release 2.19 refuses MXFP8 on compute capability 12.0 and above, “not supported on 12.0+ architectures yet”. Transformer Engine 2.19, released on 11 September 2026, added NVFP4 stochastic rounding for SM120 and SM121 GPUs, which include the RTX PRO Blackwell cards and DGX Spark; 2.20.2 followed on 5 October 2026.

More than one card: NVLink against PCIe

In data-parallel training every card holds a copy of the model, and the gradients of what is trained are all-reduced at every step; for a model of Ψ parameters, all trained, the ZeRO paper counts “2Ψ data movement during each training step”. The method therefore sets the traffic. A rank-16 LoRA on Llama 3.1 8B trains 41.9 million parameters, about 190 times fewer than the 8.03 billion of a full fine-tune, and its all-reduce is small on any link. A full fine-tune all-reduces 16 GB of BF16 gradients per step on the same model, by our arithmetic. Sharding adds to that: FSDP all-gathers the sharded weights before the forward and the backward pass, and for a full fine-tune the ZeRO paper puts the total at 1.5 times the traffic of plain data parallelism.

In a 70B LoRA sharded over two cards, FSDP gathers the frozen base: each card holds half of the base’s 141 GB and receives the other half, layer by layer, in the forward pass and again in the backward pass, which needs the weights to pass gradients back to the adapters of earlier layers. By our arithmetic that is about 141 GB per card for every forward and backward pass, so for every micro-batch, whatever its size. With more cards each holds a smaller share and receives more: about 212 GB on four and 247 GB on eight. At the nominal link rates alone, again our arithmetic, 141 GB take at least 2.2 seconds over PCIe Gen5 at 64 GB/s each way and at least 0.3 seconds over a 2-way NVLink bridge at 450 GB/s each way; FSDP hides part of that behind computation.

Here the two cards differ most. The H200 NVL takes a 2- or 4-way NVLink bridge at 900 GB/s per GPU, against 128 GB/s for PCIe Gen5 in NVIDIA’s figures, both totals for the two directions, so 450 and 64 GB/s each way. A bridge joins at most four cards, so a bridged eight-card server has at least two NVLink domains, with PCIe between them. The RTX PRO 6000 has no NVLink and exchanges data over PCIe, at best peer to peer. NVIDIA’s NCCL documentation warns that on bare-metal Linux “CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer”, and that ACS on PCIe switches has to be disabled, on some systems in the BIOS: check both before blaming the cards. Google Cloud describes an “enhanced PCIe-based P2P data path” that speeds up all-reduce in its G4 machines with the Server Edition, with gains quoted for tensor-parallel serving (October 2025).

MIG: several experiments on one card

MIG splits the H200 NVL into up to seven 1g.18gb instances, 16.5 GB each by NVIDIA’s product page, or four of 35 GB, and the RTX PRO 6000 into four of 24 GB or two of 48 GB. An 8B QLoRA, at 6.4 GB of model states, fits any of these slices; an 8B LoRA on a BF16 base, at 16.7 GB, leaves about 7 GB in a 24 GB slice and does not fit a 1g.18gb one at 16.5 GB. NVIDIA’s MIG guide (updated 4 November 2026) gives NCCL support for MIG only from NCCL 2.31 and as experimental (one rank per MIG instance, CUDA driver 13.0 or later, NCCL_MNNVL_ENABLE=0), so plan an instance for a single-GPU job, not as part of a multi-card run. On the Workstation Edition and Max-Q, NVIDIA requires the display mode to be switched to compute before MIG can be enabled, which turns the card’s display outputs off.

Which card for which run

LoRA on 8B to 32B models. The RTX PRO 6000 holds both with 79 and 28 GB left, the Workstation Edition at a desk and the Server Edition in a rack. The H200 NVL runs the same jobs at a higher peak tensor rate, which matters when runs queue up.

QLoRA on a 70B model. Either card works, with 53 GB left on the RTX PRO 6000 and 98 GB on the H200 NVL.

Full fine-tuning of an 8B model. With 32-bit Adam, one H200 NVL holds the 128.5 GB of model states with about 12 GB left; the RTX PRO 6000 needs a second card and FSDP, at 64.2 GB per card, or 8-bit Adam, at about 80 GB on one card.

Several cards. Sharded full fine-tuning and a sharded 70B LoRA move tens to hundreds of GB per forward and backward pass, and there the H200 NVL’s NVLink domains of two or four cards are the stronger platform. Beyond four cards, traffic between domains crosses PCIe on both. Where a LoRA or QLoRA run fits one card, one job per card avoids the link question altogether.

A four-card server. Four RTX PRO 6000 Server Edition hold 410 GB as the driver reports it: four LoRA or QLoRA runs side by side, one per card, or one 70B LoRA under FSDP at 36.1 GB of model states per card. Each card then receives about 212 GB per forward and backward pass over PCIe, at least 3.3 seconds at 64 GB/s, while two H200 NVL with a bridge hold the same run at 72.2 GB per card and move 141 GB per card over NVLink in at least 0.3 seconds. We build training nodes to order with RTX PRO 6000 Server Edition or with bridged H200 NVL; send us the largest model you plan to tune, and we compare both builds for it.

What we supply

For fine-tuning, we supply the H200 NVL with its two-way and four-way NVLink bridges, and the RTX PRO 6000 in all three editions: Workstation Edition and Max-Q for workstations, Server Edition for rack servers. They come as cards, in GPU workstations or in AI servers built to order, on one EU contract and invoice with manufacturer warranty.

FAQ