RTX PRO 6000 Blackwell vs H200 NVL for LLM inference: when 96 GB is enough

The same slot, two different memories

On paper the cards look like neighbours: dual-slot, passive, PCIe Gen5, up to 600 W configurable, both sold for inference in ordinary rack servers. What separates them is the memory system. The RTX PRO 6000 Blackwell Server Edition carries 96 GB of GDDR7 on a 512-bit bus for 1,597 GB/s. The H200 NVL carries 141 GB of HBM3e for 4,800 GB/s. Three times the bandwidth and half again the capacity, at the same power.

Everything in this article follows from those two numbers and from one fact of LLM inference: producing a token means reading every weight of the model once. Generation speed for a single user is a bandwidth problem, not a compute problem.

Specifications that matter for inference

Sources: NVIDIA product pages for both cards, TensorRT-LLM quantisation support matrix, NVIDIA AI Enterprise licensing guide. H200 NVL Tensor figures are NVIDIA’s sparsity numbers halved.

How big is the model, really

Take Llama 3.3 70B, the model most of our sizing requests start from. The published NVIDIA FP8 checkpoint is 72.7 GB on disk, about 68 GiB in memory. The NVIDIA NVFP4 checkpoint is 42.7 GB, about 40 GiB, and the model card reports MMLU falling from 83.3 to 81.1 and GSM8K from 95.3 to 92.6 for the compression. For most business tasks that is a trade worth making; for some it is not, and the decision belongs to whoever owns the evaluation set.

Then the KV cache. Llama 3 70B has 80 layers, 8 key-value heads and a head dimension of 128, so every token in context costs 320 KiB in FP16 or 160 KiB in FP8. A 32k-token context is therefore 10 GiB in FP16, 5 GiB in FP8, per concurrent request at full length.

gpt-oss-120b is the other frequent request: 117B parameters, 5.1B active, MXFP4 weights, 65 GB on disk. It fits either card. Qwen3-235B-A22B does not: even in NVFP4 it is around 134 GB of weights, which exceeds 96 GB outright and leaves the H200 NVL with almost no room for context.

The arithmetic behind tokens per second

NVIDIA’s own inference-optimisation guide puts it plainly: in the decode phase, the speed at which weights are transferred from memory dominates latency, not the arithmetic. A useful ceiling for one user is bandwidth divided by the bytes of weights read per token.

Single-stream ceilings from bandwidth ÷ weight bytes, not measurements. Real numbers land below the ceiling; the ranking does not change.

This is why the H200 NVL feels three times faster to one person typing at any precision: 4-bit weights help both cards, and the bandwidth ratio stays. It is also why batching changes the picture: with many concurrent requests, the same weights serve many tokens per read, the workload becomes compute-bound, and the Blackwell card’s FP4 Tensor Cores start to count.

What the published benchmarks say

NVIDIA’s TensorRT-LLM performance table lists Llama 3.3 70B in FP4 on one RTX PRO 6000 Server Edition at 1,724 output tokens per second per GPU under full load with 1,000-token prompts, falling to 296 with 8,192-token prompts. That is the batched, throughput-first regime.

MLPerf Inference v6.0 gives the only like-for-like public numbers. On the Llama 2 70B interactive scenario, which caps time-to-first-token at 450 ms and time-per-output-token at 40 ms, a Dell XE7740 with eight H200 NVL cards reached 16,343.8 tokens per second; an eight-card RTX PRO 6000 system reached 6,238.1. On the offline scenario, where latency does not count, the same chassis reached 32,004.0 tokens per second with eight H200 NVL in FP8 and 27,034.8 with eight RTX PRO 6000 in FP4; in the server scenario 29,084.9 against 25,524.1. A two-card HPE system with RTX PRO 6000 reached 6,901.9 offline.

Read together: when every user is waiting, the H200 NVL delivers about 2.6 times the interactive throughput per card. When the work is a queue of documents nobody is watching, the gap shrinks to about 18 per cent, and the Blackwell card in FP4 is a strong throughput engine at the same power, at a fraction of the memory cost.

When 96 GB is enough

The RTX PRO 6000 Server Edition is the right card when the model fits in FP4 or FP8 with the context and concurrency you actually run; when the traffic is batch or asynchronous, so throughput matters more than latency per user; when four isolated MIG tenants per card is the requirement; when the same host also needs rendering, video or virtual workstations; and when the fleet has no NVLink and never will.

The H200 NVL is the right card when users are interactive and the per-token latency is a promise; when contexts are long and concurrency high, so the KV cache, not the weights, fills the memory; when the model is above roughly 100B dense parameters and has to stay on one host, where two or four bridged cards share 282 or 564 GB over NVLink at 900 GB/s per GPU; and when the five-year AI Enterprise subscription is part of the budget anyway, because the H200 NVL brings it and the RTX PRO 6000 does not.

Our engineering partner Vixen.UNO runs the sizing from the model, the precision the evaluation set tolerates, the concurrency and the context length, in that order. The card is the output of that worksheet, and a surprising number of 70B requests end on one Blackwell card.

FAQ