H200 NVL or RTX PRO 6000 for fine-tuning and training: memory, bandwidth, tensor rates and the link between cards

Two 600 W cards with different memory

The H200 NVL and the air-cooled RTX PRO 6000 Server Edition are passive cards that depend on the server’s fans; NVIDIA also lists a single-slot, liquid-cooled Server Edition. The Workstation Edition carries the same chip and 96 GB with its own fans, for a desk, and the Max-Q version is rated at 300 W. What separates them for training is memory, tensor throughput and the link between cards.

NVIDIA H200 product page; RTX PRO 6000 datasheets and Server Edition product page; NVIDIA MIG user guide of 11 September 2026. Visible memory as reported by nvidia-smi. The NVLink rate is NVIDIA’s total for both directions.

Which runs fit in 141 and 96 GB

Our fine-tuning memory guide works out the model states, meaning weights, gradients and optimiser states, for Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B. Set against the memory the driver shows, 150.8 GB on the H200 NVL and 102.6 GB on the RTX PRO 6000, they leave this much for activations.

Our arithmetic, GB = 10⁹ bytes: rank-16 adapters on all linear layers, 16 bytes per trained parameter, a BF16 base for LoRA and, for QLoRA, NF4 linear layers with the embeddings and output layer in BF16; full fine-tuning at 16 bytes per parameter with 32-bit Adam and 10 with 8-bit Adam, by the ZeRO count. What is left must hold activations, the CUDA context and temporary buffers.

LoRA up to 32B and QLoRA up to 70B fit both cards with room for activations, so for these runs the H200 NVL’s extra 48 GB buy longer sequences and larger batches, not a run the RTX PRO 6000 cannot do. Capacity decides in two cases. The model states of a full 8B fine-tune with 32-bit Adam fit one H200 NVL but not one RTX PRO 6000, unless 8-bit Adam cuts them to about 80 GB. A 70B LoRA on a BF16 base leaves about 6 GB even on the H200 NVL, too little to plan on, and needs two cards of either kind. That moves the question to the link between them.

Why bandwidth counts for less in training

For serving, bandwidth rules: generating a token for one user reads every weight of a dense model, so the H200 NVL’s 4.8 TB/s against 1,792 GB/s on the Workstation Edition and 1,597 GB/s on the Server Edition, 2.7 and 3.0 times, sets the pace, as our inference comparison shows. Training behaves like the prompt phase instead. NVIDIA’s performance guide defines arithmetic intensity as operations per byte of memory traffic and says an algorithm “is math limited on a given processor if the algorithm’s arithmetic intensity is higher than the processor’s ops:byte ratio”. In its own examples a linear layer at batch size 512 reaches 315 FLOPS per byte and is limited by arithmetic; the same layer at batch size 1 reaches 1 FLOPS per byte and is limited by memory.

A training step pushes thousands of tokens through each layer at once. By the same formula, the up-projection of Llama 3.1 8B, from 4,096 to 14,336, reaches about 1,250 FLOPS per byte on one 2,048-token sequence in BF16, by our arithmetic. The ops:byte ratio is 174 on the H200 NVL, 835.5 dense BF16 TFLOPS over 4.8 TB/s, and 281 on the RTX PRO 6000 Workstation Edition, 503.8 over 1.792 TB/s. The large matrix multiplications are therefore compute-bound on both cards, and for them the tensor rate counts: 1.66 times in the H200 NVL’s favour, not 2.7. Bandwidth still matters for the operations NVIDIA lists as limited by memory, such as activation functions and layer normalisation, and, by our reasoning, for three more: the optimiser step, which touches every state once with little arithmetic; the rank-16 adapter multiplications, at about 16 FLOPS per byte by the same formula; and, in QLoRA, the dequantisation of every 4-bit weight before each multiplication. These parts favour the H200 NVL by the bandwidth ratio, not the tensor ratio. We found no published fine-tuning benchmark that runs both cards under the same conditions, so we give no measured ratio.

Tensor rates, precision by precision

TFLOPS unless marked. NVIDIA H200 product page, rates “With sparsity”, halved by us for dense; NVIDIA RTX PRO Blackwell GPU architecture whitepaper v1.0, Table 4, Workstation Edition at its 2,617 MHz boost clock; RTX PRO 6000 Server Edition product page, which does not say whether its figures include sparsity.

Sparse rates apply only to weights pruned to NVIDIA’s 2:4 pattern, where “In each contiguous block of four values, two values must be zero”. A normal fine-tune produces dense weights, so compare the dense rates. The RTX PRO 6000’s BF16 and FP8 figures are those NVIDIA prints with FP32 accumulation; NVIDIA’s H200 page does not state the accumulate mode, but its Hopper architecture blog gives Hopper’s BF16 Tensor rates with FP32 accumulate. In peak dense rates, the H200 NVL leads the Workstation Edition 835.5 to 503.8 TFLOPS in BF16 and 1,670.5 to 1,007.6 in FP8, 1.66 times in both. Against the 300 W Max-Q, which NVIDIA’s RTX PRO whitepaper puts at 438.9 dense BF16 TFLOPS, the ratio is 1.90; for the Server Edition we found no dense rate from NVIDIA. The Server Edition’s 1, 2 and 4 PFLOPS match the Workstation Edition’s sparse figures, rounded. QLoRA dequantises its 4-bit weights to BF16 for every multiplication, so the BF16 row applies to it too.

FP8 and FP4 training. Both cards train in FP8 through NVIDIA’s Transformer Engine, whose README, as of September 2026, lists “Support for FP8 on NVIDIA Hopper, Ada, and Blackwell GPUs”. MXFP8 and NVFP4 are listed for Blackwell GPUs only, so neither is an option on the H200 NVL; on the RTX PRO 6000 only NVFP4 is, because Transformer Engine’s own support check still refuses MXFP8 on compute capability 12.0 and above, “not supported on 12.0+ architectures yet”. Transformer Engine 2.19, released on 11 September 2026, added NVFP4 stochastic rounding for SM120 and SM121 GPUs, which include the RTX PRO Blackwell cards and DGX Spark.

More than one card: NVLink against PCIe

In data-parallel training every card holds a copy of the model, and the gradients of what is trained are all-reduced at every step; for a model of Ψ parameters, all trained, the ZeRO paper counts “2Ψ data movement during each training step”. The method therefore sets the traffic. A rank-16 LoRA on Llama 3.1 8B trains 41.9 million parameters, about 190 times fewer than the 8.03 billion of a full fine-tune, and its all-reduce is small on any link. A full fine-tune all-reduces 16 GB of BF16 gradients per step on the same model, by our arithmetic. Sharding adds to that: FSDP all-gathers the sharded weights before the forward and the backward pass, and for a full fine-tune the ZeRO paper puts the total at 1.5 times the traffic of plain data parallelism.

In a 70B LoRA sharded over two cards, what is gathered is the frozen base: each card holds half of the base’s 141 GB and receives the other half, layer by layer, in the forward pass and again in the backward pass, which needs the weights to pass gradients back to the adapters of earlier layers. By our arithmetic that is about 141 GB per card for every forward and backward pass, so for every micro-batch, whatever its size. With more cards each holds a smaller share and receives more: about 212 GB on four and 247 GB on eight. At the nominal link rates alone, again our arithmetic, 141 GB take at least 2.2 seconds over PCIe Gen5 at 64 GB/s each way and at least 0.3 seconds over a 2-way NVLink bridge at 450 GB/s each way; FSDP hides part of that behind computation.

Here the two cards differ most. The H200 NVL takes a 2- or 4-way NVLink bridge at 900 GB/s per GPU, against 128 GB/s for PCIe Gen5 in NVIDIA’s figures, both totals for the two directions, so 450 and 64 GB/s each way. A bridge joins at most four cards, so a bridged eight-card server has at least two NVLink domains, with PCIe between them. The RTX PRO 6000 has no NVLink and exchanges data over PCIe, at best peer to peer. NVIDIA’s NCCL documentation warns that on bare-metal Linux “CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer”, and that ACS on PCIe switches has to be disabled, on some systems in the BIOS: check both before blaming the cards. Google Cloud describes an “enhanced PCIe-based P2P data path” that speeds up all-reduce in its G4 machines with the Server Edition, with gains quoted for tensor-parallel serving (October 2025).

MIG: several experiments on one card

MIG splits the H200 NVL into up to seven 1g.18gb instances, 16.5 GB each by NVIDIA’s product page, or four of 35 GB, and the RTX PRO 6000 into four of 24 GB or two of 48 GB. An 8B QLoRA, at 6.4 GB of model states, fits any of these slices; an 8B LoRA on a BF16 base, at 16.7 GB, leaves about 7 GB in a 24 GB slice and no working room in a 1g.18gb one. NVIDIA’s MIG guide states that “NCCL is currently not supported with MIG”, so an instance runs a single-GPU job, never part of a multi-card run. On the Workstation Edition and Max-Q, NVIDIA requires the display mode to be switched to compute before MIG can be enabled, which turns the card’s display outputs off.

Which card for which run

LoRA on 8B to 32B models. The RTX PRO 6000 holds both with 86 and 35 GB left, the Workstation Edition at a desk and the Server Edition in a rack. The H200 NVL runs the same jobs at a higher peak tensor rate, which matters when runs queue up.

QLoRA on a 70B model. Either card, with 60 GB left on the RTX PRO 6000 and 108 GB on the H200 NVL.

Full fine-tuning of an 8B model. With 32-bit Adam, one H200 NVL holds the 128 GB of model states with 22 GB left; the RTX PRO 6000 needs a second card and FSDP, at 64.2 GB per card, or 8-bit Adam, at about 80 GB on one card.

Several cards. Sharded full fine-tuning and a sharded 70B LoRA move tens to hundreds of GB per forward and backward pass, and there the H200 NVL’s NVLink domains of two or four cards are the stronger platform. Beyond four cards, traffic between domains crosses PCIe on both. Where a LoRA or QLoRA run fits one card, one job per card avoids the link question altogether.

What we supply

Eurokommerz supplies the H200 NVL and the RTX PRO 6000 Workstation Edition and Max-Q EU-wide with manufacturer warranty, as cards or in workstations configured for training, and builds AI servers to order. Tell us the runs you plan, and we will size the cards and their number with you.

FAQ