RTX PRO 6000 Blackwell vs H200 NVL for LLM inference: when 96 GB is enough

Memory: 96 GB GDDR7 against 141 GB HBM3e

On paper the two cards are similar: dual-slot, passive, PCIe Gen5, configurable up to 600 W, and both sold for inference in standard rack servers. They differ in the memory system. The RTX PRO 6000 Blackwell Server Edition carries 96 GB of GDDR7 on a 512-bit bus for 1,597 GB/s. The H200 NVL carries 141 GB of HBM3e for 4,800 GB/s, three times the bandwidth and half as much capacity again, at the same power.

The rest of this comparison follows from those two numbers. Producing a token means reading every weight of a dense model once, and every active weight of a mixture-of-experts model, so for a single user, generation speed is limited by memory bandwidth rather than by compute.

Specifications that matter for inference

Sources: NVIDIA product pages for both cards, TensorRT-LLM quantisation support matrix, NVIDIA AI Enterprise licensing guide. H200 NVL Tensor figures are NVIDIA’s sparsity numbers halved; NVIDIA lists the RTX PRO 6000 Server Edition at 2 PFLOPS FP8 without stating sparsity, and ~1 PFLOPS is the Workstation Edition’s dense rate.

Model sizes and KV cache: what fits in 96 GB and 141 GB

Take Llama 3.3 70B, a common starting point for on-premise sizing. The published NVIDIA FP8 checkpoint is 72.7 GB on disk, about 68 GiB in memory. The NVIDIA NVFP4 checkpoint is 42.7 GB, about 40 GiB, and the model card reports MMLU falling from 83.3 to 81.1 and GSM8K from 95.3 to 92.6 for the compression. For most business tasks that loss is acceptable, but not for all, so the decision belongs to whoever owns the evaluation set.

The KV cache comes on top of the weights. Llama 3 70B has 80 layers, 8 key-value heads and a head dimension of 128, so every token in context costs 320 KiB in FP16 or 160 KiB in FP8. A 32k-token context is therefore 10 GiB in FP16, 5 GiB in FP8, per concurrent request at full length.

Cache room by a planning rule: 90 per cent of the driver-visible memory (95.6 GiB and 140.4 GiB) minus the weights, without the 3 GiB per card our guide to concurrent users also subtracts.

gpt-oss-120b is the second model in this comparison: 117B parameters, 5.1B active, MXFP4 weights and 65 GB on disk, so it fits either card. Qwen3-235B-A22B does not fit either card on its own: even in NVFP4 it has around 134 GB of weights, more than the 96 GB of the RTX PRO 6000. On the H200 NVL, NVFP4 has no native path, because TensorRT-LLM supports it only on Blackwell and vLLM runs NVFP4 checkpoints there only weight-only, with 16-bit arithmetic; 134 GB would also leave almost no room for context. The FP8 version, at 236 GB, needs two H200 NVL cards.

For a two-card setup, we supply the H200 NVL together with the two-way NVLink bridge, and the four-way bridge for four cards. Tell us the model and the server the cards go into, and you get a written quote.

The arithmetic behind tokens per second

NVIDIA’s own inference-optimisation guide states that in the decode phase, latency is dominated by the speed at which weights are transferred from memory, not by the arithmetic. A useful ceiling for one user is bandwidth divided by the bytes of weights read per token.

Single-stream ceilings from bandwidth ÷ weight bytes read per token, not measurements. gpt-oss-120b is a mixture-of-experts model: per token it reads only its four active experts in MXFP4, the attention layers and the output layer in BF16, about 5 GB of its 65 GB by our arithmetic from OpenAI’s model card. Real numbers land below the ceiling; the ranking does not change.

This is why, for one person typing, the H200 NVL is about three times as fast at any precision: 4-bit weights help both cards, and the bandwidth ratio stays the same. Batching changes the balance, because with many concurrent requests the same weights serve many tokens per read; the workload becomes compute-bound, and the FP4 Tensor Cores of the Blackwell card make a difference.

Published benchmarks: TensorRT-LLM and MLPerf® Inference v6.0

NVIDIA’s TensorRT-LLM performance overview (documentation for 1.4.0rc1, updated on 19 October 2026) lists Llama 3.3 70B in FP4 on one RTX PRO 6000 Server Edition at 1,724 output tokens per second per GPU under full load with 1,000 input and 1,000 output tokens, falling to 296 with 8,192 input and 1,024 output tokens. That is the batched, throughput-first regime.

MLPerf® Inference v6.0 (April 2026) gives like-for-like public numbers; it is the latest round with results for both cards, since the closed division of v6.1, published on 16 September 2026, has no H200 NVL result. On the Llama 2 70B interactive scenario, which caps time-to-first-token at 450 ms and time-per-output-token at 40 ms, a Dell PowerEdge XE7740 with eight H200 NVL cards reached 16,343.8 tokens per second; an eight-card RTX PRO 6000 system in another maker’s server reached 6,238.1 (entry 6.0-0004). In the offline scenario, where latency does not count, the same Dell chassis reached 32,004.0 tokens per second with eight H200 NVL in FP8 and 27,034.8 with eight RTX PRO 6000 in FP4. In the server scenario the figures were 29,084.9 and 25,524.1. HPE’s ProLiant Compute DL380a Gen12 with eight RTX PRO 6000, also in FP4, reached 29,908.4 offline and 28,727.8 in the server scenario. A two-card HPE ProLiant DL385 Gen11 with RTX PRO 6000 reached 6,901.9 offline.

MLPerf® Inference v6.0, Datacenter, closed division, available systems, benchmark llama2-70b-99. Dell PowerEdge XE7740 with eight H200 NVL: entry 6.0-0021 (Interactive, Offline, Server); with eight RTX PRO 6000: entry 6.0-0022 (Offline, Server); eight RTX PRO 6000, Interactive: entry 6.0-0004; HPE ProLiant Compute DL380a Gen12 with eight RTX PRO 6000: entry 6.0-0047 (Offline, Server); HPE ProLiant DL385 Gen11 with two RTX PRO 6000: entry 6.0-0052 (Offline). Retrieved from mlcommons.org on 24 September 2026 (entry 6.0-0047 on 5 October 2026); results verified by MLCommons Association. Per-card comparisons are our division of each system result by its number of GPUs, not an MLPerf® metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Taken together, the H200 NVL delivers about 2.6 times the interactive throughput per card when every user is waiting for an answer. Against the eight-card RTX PRO 6000 systems cited here, its lead shrinks to 7 to 18 per cent offline, the scenario closest to a queue of documents that nobody is waiting for, and to 1 to 14 per cent in the server scenario. For such batch work, the Blackwell card in FP4 delivers high throughput at the same 600 W board power.

When 96 GB is enough, and when the H200 NVL is needed

The RTX PRO 6000 Server Edition is the right card when the model fits in FP4 or FP8 with the context and concurrency you run; when the traffic is batch or asynchronous, so throughput matters more than latency per user; when four isolated MIG tenants per card are required; when the same host also needs rendering, video or virtual workstations; and when your servers have no NVLink and none is planned.

The H200 NVL is the right card when users work interactively and per-token latency is part of a service level; when contexts are long and concurrency high, so the KV cache, not the weights, fills the memory; when the model is above roughly 100B dense parameters and has to stay on one host, where two or four bridged cards share 282 or 564 GB over NVLink at 900 GB/s per GPU; and when the five-year AI Enterprise subscription is part of the budget anyway, because the H200 NVL includes it and the RTX PRO 6000 does not.

What we supply

We supply the RTX PRO 6000 Server Edition and the H200 NVL with its two-way and four-way NVLink bridges, as cards for a server you already run (see the RTX PRO 6000 compatible servers and the H200 NVL checklist for an existing server) or in an AI server built to order, on one EU contract and invoice with manufacturer warranty. We size the choice from the workload: the model, the precision your evaluation set tolerates, the concurrency and the context length. The H200 NVL comes with its five-year NVIDIA AI Enterprise subscription; for the RTX PRO 6000, an NVIDIA AI Enterprise licence goes into the same quote where your setup needs one.

FAQ