H200 NVL vs H100 NVL: when the upgrade pays off

Compute specifications are identical

The H200 NVL and the H100 NVL have the same compute figures; they differ in memory (141 GB of HBM3e against 94 GB of HBM3), bandwidth (4.8 against 3.9 TB/s), power limit (up to 600 W against 400 W) and NVLink, which joins up to four H200 NVL but only two H100 NVL. Side by side, the compute sections of the two spec sheets match to the last digit. We supply the H200 NVL and the NVLink bridges it needs; tell us your model and context length, and our team will help you judge whether the upgrade pays off.

Tensor figures with structured sparsity, per NVIDIA’s own footnotes

For a compute-bound task, the two cards therefore perform the same on paper. Under sustained load, however, the older card can clock down earlier, because the H100 NVL defaults to a 400 W cap and the H200 NVL to 600 W. The identical peak figures come from the spec sheets, and we found no measurement of sustained clocks on the two cards. That said, NVIDIA’s technical blog on its MLPerf® Inference v4.0 results of March 2024 states that after TensorRT-LLM optimisations, Llama 2 70B on H200 is limited by compute, not memory bandwidth.

Memory: 141 GB HBM3e against 94 GB HBM3

Both NVL cards run a wider bus than the H100 SXM: six HBM stacks instead of five, so even the older H100 NVL delivers 3.9 TB/s against 3.35 on SXM. We found no NVIDIA explanation of why the capacity is 94 GB rather than a round 96.

Published gains by workload

The gain ranges from zero to 3.4×, depending on how memory-bound the task is.

Baseten measured SXM cards (8×H200 vs 8×H100, 700 W); the NVL cards run 600 W against 400 W, so treat the ratios as indicative.

NVIDIA published a separate figure from its own MLPerf® Inference v4.0 submissions of March 2024: at the same 700 W, the H200 SXM delivered up to 28% more Llama 2 70B throughput than the H100 SXM (unverified). It is NVIDIA’s figure for the SXM cards and does not compare with Baseten’s measurements. Neither 2026 round, MLPerf® Inference v6.0 or v6.1, has results for both NVL cards in the same benchmark, so neither gives a like-for-like comparison. Result not verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Baseten concludes that outside these situations, H200 gives a minimal gain over H100, and that many inference workloads are more cost-effective on H100. For the NVL pair NVIDIA quotes up to 1.7× on language models and 1.3× on HPC; we found no published test conditions for these figures.

NVIDIA’s headline figures and their footnotes

The largest figure in the H200 materials is “110× on HPC”. According to the footnote, it compares four GPUs with two Sapphire Rapids 8480 CPUs, so the comparison is with processors, not with the H100.

The second figure is 1.9× on Llama 2 70B. NVIDIA’s footnote gives the conditions: H100 ran batch 8 and H200 ran batch 32. The larger batch is possible because of the extra memory (see the KV-cache arithmetic), which is the point of the H200. It does not mean that the same workload runs twice as fast: if your model already fits in 94 GB with the batch you need, that gain will not appear.

NVLink: the biggest generational difference

Bridges of the two generations are not interchangeable

The rated 900 GB/s counts both directions: 18 links at 25 GB/s per direction, 450 GB/s one-way. A user report of a two-card H200 NVL setup in an HPE DL385 Gen11 shows all 18 links active, reported by nvidia-smi at a raw rate of 26.6 GB/s per link. Cards outside an NVLink bridge talk over PCIe Gen5 at 128 GB/s, roughly seven times slower.

Power, temperature, driver and licence

The first two rows matter for planning. The newer card’s default power limit is 200 W higher, and its rated ambient range is narrower at both ends. The server’s limit is the tighter one, because the servers certified for the H200 NVL cap the room at 30 or 35 °C, as our checklist for existing servers shows. At full load all year, the extra 200 W add up to 1,752 kWh per card, and the rack power budget becomes a constraint before the energy cost does. The driver row gives first-support versions only: R535 reached end of life in June 2026, so both cards belong on a current branch, R580 with long-term support to June 2028 or R595 to March 2027.

When the upgrade pays, and when an existing H100 NVL is enough

An upgrade pays when the gain on your workload profile outweighs the difference in cost per useful token; the cost side can only be calculated from a quote for your configuration. In Baseten’s tests on SXM systems a chat workload with short prompts gained about 11% in FP8, while 32K-token inputs gained 3.4×.

It matters even more to compare whole configurations rather than single cards, as the table below does.

According to a reseller’s end-of-life notice, the last time buy for the H100 NVL was 1 November 2025. The comparison below is for servers that already run H100 NVL cards.

Where the table points to the H200 NVL, we supply it with the two-way or four-way NVLink bridge. Write to us with the machine type of your current servers, and we check their power and airflow before you order.

For spiky, one-off workloads, the cloud is the better choice. Both NVL cards, however, ship with a five-year NVIDIA AI Enterprise subscription (an add-on for SXM boards outside NVIDIA’s DGX systems), and on your own servers your data stays in your network. In medical, financial and public-sector projects this is often a precondition, settled before cost is discussed.

What we supply

For an upgrade from H100 NVL, we supply the NVIDIA H200 NVL (141 GB HBM3e, PCIe) with the new NVLink bridges it needs, on one EU contract and invoice with manufacturer warranty. Where 96 GB of GDDR7 per card is enough and no H100 NVL estate has to be matched, we supply the RTX PRO 6000 Blackwell Server Edition and build servers to order with it, among them the inference node with four cards and 384 GB in our AI server configurations; it has no NVLink, so a model split across its cards communicates over PCIe. We do not supply the H100 NVL.

FAQ