H200 NVL vs H100 NVL: when the premium pays off

First, the fact that breaks most comparisons

Put the two spec sheets side by side and you will see something reviews rarely mention: the compute sections match to the last digit. Not approximately: exactly.

Tensor figures with structured sparsity, per NVIDIA’s own footnotes

The consequence is simple: if your task is compute-bound, the two cards perform the same on paper. One caveat an engineer will raise: the H100 NVL defaults to a 400 W cap and the H200 NVL to 600 W, so under sustained load the older card can clock down earlier; identical peak specs are a spec-sheet statement, not a measured one. That said, NVIDIA’s own MLPerf report states that after TensorRT-LLM optimisations, Llama 2 70B on H200 is limited by compute, not memory bandwidth.

All the difference lives in memory

Both NVL cards run a wider bus than the H100 SXM: six HBM stacks instead of five, so even the older H100 NVL delivers 3.9 TB/s against 3.35 on SXM. Why the capacity is 94 GB rather than a round 96, NVIDIA explains nowhere.

What it buys on specific workloads

There is no single number. The spread runs from zero to 3.4×, depending on how memory-bound the task is.

Baseten and MLPerf measured SXM cards (8×H200 vs 8×H100, 700 W); the NVL cards run 600 W against 400 W, so treat the ratios as indicative

Baseten’s conclusion is blunter than any slogan: outside these situations, H200 gives a minimal gain over H100, and many inference workloads are more cost-effective on H100. For the NVL pair NVIDIA itself promises up to 1.7× on language models and 1.3× on HPC, without publishing the test conditions.

How to read the presentation numbers

The loudest figure in H200 materials is “110× on HPC”. The footnote reveals the baseline: four GPUs against a pair of Sapphire Rapids 8480 CPUs. It is a comparison with processors, not with H100.

The second figure is 1.9× on Llama 2 70B. NVIDIA’s footnote shows the conditions: H100 ran batch 8, H200 ran batch 32. That is not cheating: the bigger batch is possible precisely because of the bigger memory, the KV-cache arithmetic, and that is the point of the product. But it does not mean the same workload runs twice as fast. If your model already fits in 94 GB with the batch you need, that gain will not appear.

NVLink: the biggest generational difference

Bridges of the two generations are not interchangeable

The rated 900 GB/s counts both directions: 18 links at 25 GB/s per direction, 450 GB/s one-way. A documented two-card H200 NVL setup in an HPE DL385 Gen11 shows all 18 links active, reported by nvidia-smi at the raw lane rate of 26.6 GB/s each. Cards outside a bridge group talk over PCIe Gen5 at 128 GB/s, roughly seven times slower.

What changes besides speed

Note the second row: the newer card draws 200 W more and tolerates a narrower temperature window on both ends. In a machine room, a detail; in an office server room that heats past 45 °C in summer, a reason not to buy. And the electricity: 200 W of difference around the clock is 1,752 kWh per card per year; the rack maths feels it before the invoice does.

When the premium pays, and when H100 NVL is enough

The rule is one line: the premium is justified when the gain on your workload profile, in percent, exceeds the price difference in percent. A chat service with short prompts and an 11% gain on a card that costs 40% more makes every token more expensive. Batch-processing 30K-token documents with a 3.4× gain repays the premium in months.

The second calculation matters more, and almost nobody does it: compare configurations, not cards.

On buying versus renting: the cloud wins on spiky, one-off workloads. But both NVL cards ship with a five-year NVIDIA AI Enterprise licence (bought separately for SXM), the data stays inside your perimeter; for medical, financial and public-sector projects that is a question of admission, not money, and Hopper is winding down: some server platforms with these GPUs have already closed ordering.

What we supply

Eurokommerz supplies the NVIDIA H200 NVL (141 GB HBM3e, PCIe) and the H100 NVL (both available to order, along with the RTX PRO 6000 Blackwell Server Edition when 96 GB of GDDR7 is enough and HBM is overkill), across the EU, with manufacturer warranty and EU invoicing.

FAQ