RTX PRO 6000 Blackwell vs H200 NVL for LLM inference: when 96 GB is enough

The same slot, two different memories

On paper the cards look like neighbours: dual-slot, passive, PCIe Gen5, up to 600 W configurable, both sold for inference in ordinary rack servers. What separates them is the memory system. The RTX PRO 6000 Blackwell Server Edition carries 96 GB of GDDR7 on a 512-bit bus for 1,597 GB/s. The H200 NVL carries 141 GB of HBM3e for 4,800 GB/s. Three times the bandwidth and half again the capacity, at the same power.

Everything in this article follows from those two numbers and from one fact of LLM inference: producing a token means reading every weight of a dense model once, and every active weight of a mixture-of-experts model. Generation speed for a single user is a bandwidth problem, not a compute problem.

Specifications that matter for inference

Sources: NVIDIA product pages for both cards, TensorRT-LLM quantisation support matrix, NVIDIA AI Enterprise licensing guide. H200 NVL Tensor figures are NVIDIA’s sparsity numbers halved.

How big is the model, really

Take Llama 3.3 70B, the model most of our sizing requests start from. The published NVIDIA FP8 checkpoint is 72.7 GB on disk, about 68 GiB in memory. The NVIDIA NVFP4 checkpoint is 42.7 GB, about 40 GiB, and the model card reports MMLU falling from 83.3 to 81.1 and GSM8K from 95.3 to 92.6 for the compression. For most business tasks that is a trade worth making; for some it is not, and the decision belongs to whoever owns the evaluation set.

Then the KV cache. Llama 3 70B has 80 layers, 8 key-value heads and a head dimension of 128, so every token in context costs 320 KiB in FP16 or 160 KiB in FP8. A 32k-token context is therefore 10 GiB in FP16, 5 GiB in FP8, per concurrent request at full length.

Cache room by our sizing rule: 90 per cent of the driver-visible memory (95.6 GiB and 140.4 GiB) minus the weights.

gpt-oss-120b is the other frequent request: 117B parameters, 5.1B active, MXFP4 weights, 65 GB on disk. It fits either card. Qwen3-235B-A22B does not: even in NVFP4 it is around 134 GB of weights, which exceeds 96 GB outright. On the H200 NVL, NVFP4 has no native path: TensorRT-LLM supports it only on Blackwell, and vLLM runs NVFP4 checkpoints there only weight-only, with 16-bit arithmetic, and 134 GB would leave almost no room for context. The FP8 version, at 236 GB, needs two H200 NVL cards.

The arithmetic behind tokens per second

NVIDIA’s own inference-optimisation guide puts it plainly: in the decode phase, the speed at which weights are transferred from memory dominates latency, not the arithmetic. A useful ceiling for one user is bandwidth divided by the bytes of weights read per token.

Single-stream ceilings from bandwidth ÷ weight bytes read per token, not measurements. gpt-oss-120b is a mixture-of-experts model: per token it reads only its four active experts in MXFP4, the attention layers and the output layer in BF16, about 5 GB of its 65 GB by our arithmetic from OpenAI’s model card. Real numbers land below the ceiling; the ranking does not change.

This is why the H200 NVL feels three times faster to one person typing at any precision: 4-bit weights help both cards, and the bandwidth ratio stays. It is also why batching changes the picture: with many concurrent requests, the same weights serve many tokens per read, the workload becomes compute-bound, and the Blackwell card’s FP4 Tensor Cores start to count.

What the published benchmarks say

NVIDIA’s TensorRT-LLM performance table lists Llama 3.3 70B in FP4 on one RTX PRO 6000 Server Edition at 1,724 output tokens per second per GPU under full load with 1,000-token prompts, falling to 296 with 8,192-token prompts. That is the batched, throughput-first regime.

MLPerf Inference v6.0 (April 2026) gives like-for-like public numbers; the v6.1 round published on 16 September 2026 is not included here. On the Llama 2 70B interactive scenario, which caps time-to-first-token at 450 ms and time-per-output-token at 40 ms, a Dell PowerEdge XE7740 with eight H200 NVL cards reached 16,343.8 tokens per second; an eight-card RTX PRO 6000 system in another maker’s server reached 6,238.1 (entry 6.0-0004). On the offline scenario, where latency does not count, the same Dell chassis reached 32,004.0 tokens per second with eight H200 NVL in FP8 and 27,034.8 with eight RTX PRO 6000 in FP4; in the server scenario 29,084.9 against 25,524.1. A two-card HPE ProLiant DL385 Gen11 with RTX PRO 6000 reached 6,901.9 offline.

MLPerf Inference v6.0, Datacenter, closed division, available systems, benchmark llama2-70b-99. Dell PowerEdge XE7740 with eight H200 NVL: entry 6.0-0021 (Interactive, Offline, Server); with eight RTX PRO 6000: entry 6.0-0022 (Offline, Server); eight RTX PRO 6000, Interactive: entry 6.0-0004; HPE ProLiant DL385 Gen11 with two RTX PRO 6000: entry 6.0-0052 (Offline). Retrieved from mlcommons.org on 24 September 2026; results verified by MLCommons Association. Per-card comparisons are our division of each system result by its number of GPUs, not an MLPerf metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Read together: when every user is waiting, the H200 NVL delivers about 2.6 times the interactive throughput per card. When the work is a queue of documents nobody is watching, the gap shrinks to about 18 per cent, and the Blackwell card in FP4 is a strong throughput engine at the same 600 W board power.

When 96 GB is enough

The RTX PRO 6000 Server Edition is the right card when the model fits in FP4 or FP8 with the context and concurrency you actually run; when the traffic is batch or asynchronous, so throughput matters more than latency per user; when four isolated MIG tenants per card is the requirement; when the same host also needs rendering, video or virtual workstations; and when the fleet has no NVLink and never will.

The H200 NVL is the right card when users are interactive and the per-token latency is a promise; when contexts are long and concurrency high, so the KV cache, not the weights, fills the memory; when the model is above roughly 100B dense parameters and has to stay on one host, where two or four bridged cards share 282 or 564 GB over NVLink at 900 GB/s per GPU; and when the five-year AI Enterprise subscription is part of the budget anyway, because the H200 NVL brings it and the RTX PRO 6000 does not.

Our engineering partner Vixen.UNO sizes the server from the workload: the model, the precision the evaluation set tolerates, the concurrency and the context length, as our AI servers page describes. The card is the output of that worksheet, not its input.

FAQ