NVIDIA H200 NVL benchmarks: MLPerf® results per card and per user, and limits

The card, and where the numbers come from

The H200 NVL is NVIDIA’s Hopper GPU on a dual-slot, full-length, passively cooled PCIe card, with 141 GB of HBM3e at 4.8 TB/s and a board power configurable up to 600 W. NVIDIA’s table gives 3,341 TFLOPS of FP8, a figure that counts sparsity (1,670.5 dense), and the card has one connector for a two-way or four-way NVLink bridge at 900 GB/s per GPU. NVIDIA announced the card on 18 November 2024, with partner servers to follow from December 2024. This review collects published measurements; we have not tested the card ourselves.

The richest source is the MLPerf® Inference and Training benchmark results: MLCommons publishes every result with its system description and run logs. Eight-card H200 NVL servers from Dell, Cisco and others appear in MLPerf® Inference v5.0 (April 2025), v5.1 (September 2025) and v6.0 (1 April 2026), and two Dell servers in MLPerf® Training v6.0 (June 2026). NVIDIA’s TensorRT-LLM performance overview, last updated on 19 October 2026, lists the H200 SXM but not the NVL card, and its inference performance page has no H200 NVL results. Independent measurements are scarce: in a review published on 23 September 2026, StorageReview included one H200 NVL in a Dell PowerEdge R770 as a comparison system for smaller models and measured output tokens per second with vLLM, at 512 input and 512 output tokens and at 8,192 and 1,024, from one to 128 concurrent streams. MLPerf® Inference v6.1 was published on 16 September 2026; we found no H200 NVL result in its closed division, so this review cites results up to v6.0.

Llama 2 70B in MLPerf® Inference

The Llama 2 70B benchmark of MLPerf® Inference answers questions from the OpenOrca dataset, about 290 generated tokens per answer by our arithmetic from the logs, in three scenarios. Offline hands the system all its samples at once and measures throughput with no latency limit. Server sends requests at random intervals and reports the throughput sustained while the 99th percentile of the time to first token (TTFT) stays within 2 s and that of the time per output token (TPOT), the average interval between generated tokens, within 200 ms. Interactive tightens the limits to 450 ms and 40 ms. The v6.0 result comes from Dell’s PowerEdge XE7740: eight cards at a 600 W board power, two Xeon 6747P processors and FP8 weights.

MLPerf® Inference v6.0, Datacenter, closed division, available systems: Llama 2 70B (99 per cent of the reference accuracy), offline, server and interactive scenarios, result 6.0-0021, Dell PowerEdge XE7740 with eight H200 NVL; the latency limits apply to the 99th percentile. Every MLPerf® result on this page: retrieved from mlcommons.org on 24 September 2026 (Inference result 6.0-0047 on 5 October 2026), result verified by MLCommons Association. Software as Dell lists it: TensorRT 10.13 and 10.14, CUDA 13.0, driver 580.126.09. Per card is our division of the system result by its number of GPUs, not an MLPerf® metric; mean TPOT is taken from the run logs. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The v5.0 round gave similar figures. In MLPerf® Inference v5.0, closed division, Dell’s PowerEdge XE7745 with eight cards (result 5.0-0017) reached 31,149.9 tokens per second offline on Llama 2 70B and 16,134.4 under the interactive limits, which v5.0 ran as a separate benchmark, Llama 2 70B Interactive, in the server scenario. Both are within 3 per cent of v6.0, across two rounds, two chassis and TensorRT 10.8 to 10.14; the v5.0 files we read state no weight precision. These are stable reference figures for the card with NVIDIA’s TensorRT stack, which every H200 NVL result we read used; results with another engine may differ. The system descriptions name NVLink as the link between the cards, but none states how the bridges were fitted.

Tokens per second for one request

By our arithmetic from the mean TPOT in the run logs (not MLPerf® metrics), each request received about 6.5 tokens per second in the server run and about 28 in the interactive run. Dividing the throughput per card by those rates gives some 560 and 72 requests in progress per card at once, on average. The server row is a heavily batched service in which each answer streams at roughly reading pace; the interactive row gives each request more than four times that speed for 56 per cent of the throughput.

On one card, memory bandwidth sets the ceiling for a single request, because every generated token reads all the weights once. For a dense 70B model in FP8, about 70 GB, 4.8 TB/s allows about 68 tokens per second; this is a ceiling, not a measurement. The interactive run gave each of its 72 requests per card about 40 per cent of that. With many requests in flight, the weights are read once per step for all of them, and the work becomes more compute-bound. NVIDIA wrote in March 2024 that after its TensorRT-LLM optimisations, Llama 2 70B on the H200 (SXM) “is compute performance bound rather than limited by memory bandwidth or communication bottlenecks”. The H200 NVL has the same peak compute specifications as the H100 NVL, so its gains over that card come mainly from memory; the older card’s lower power limit, 350 to 400 W against up to 600 W, can also cost it clock speed under sustained load, as our comparison of the two sets out.

On capacity, the driver reports 143,771 MiB, 140.4 GiB, the figure Dell also enters in its MLPerf® Training system descriptions. By the sizing rule of our guide to concurrent users, take 90 per cent of that and subtract 3 GiB of overhead and 67.7 GiB of Llama 3.3 70B weights in FP8. That leaves 55.7 GiB of cache on one card, enough for 44 sessions of 8,192 tokens with an FP8 cache at 1.25 GiB each.

Against the RTX PRO 6000 in the same server

For v6.0, Dell also ran the XE7740 with eight RTX PRO 6000 Blackwell Server Edition cards, likewise at 600 W with TensorRT on CUDA 13.0, but with FP4 weights, a format Hopper cannot compute.

MLPerf® Inference v6.0, Datacenter, closed division, available systems, Llama 2 70B (99 per cent of the reference accuracy), offline and server scenarios, tokens per second for eight cards in Dell PowerEdge XE7740 systems: result 6.0-0021, H200 NVL with FP8 weights, and result 6.0-0022, RTX PRO 6000 Server Edition with FP4 weights, linked through PCIe switches; the processors, host memory and driver builds also differ. HPE ProLiant Compute DL380a Gen12 with eight RTX PRO 6000 Server Edition cards and FP4 weights, cited below the table: MLPerf® Inference result 6.0-0047 (not the MLPerf® Training result with the same number), in the same round, division, category, benchmark and scenarios. Percentages are our arithmetic, not MLPerf® metrics.

In the two throughput scenarios the H200 NVL leads by only 18 and 14 per cent. By NVIDIA’s account of this benchmark, batched work depends mainly on arithmetic, and the Blackwell card computes in FP4. The XE7740 with RTX PRO 6000 has no interactive entry. Result 6.0-0004 is another maker’s server with eight RTX PRO 6000 Server Edition cards and FP4 weights, with its own processors and software. In the interactive scenario of the same round and division it reached 6,238.1 tokens per second, which puts the H200 NVL at about 2.6 times per card by our arithmetic. The same system reached 27,812.9 tokens per second offline and 28,038.5 in the server scenario, so against it the H200 NVL led by 15 and 4 per cent in those scenarios. Against the highest eight-card RTX PRO 6000 results in the same round and division, HPE’s result 6.0-0047 with 29,908.4 tokens per second offline and 28,727.8 in the server scenario, the lead was 7 and 1 per cent. Under a 40 ms limit per token, the speed of each request counts, and the H200 NVL reads its memory three times as fast: 4.8 TB/s against 1,597 GB/s. Our comparison of the two cards turns this into a buying rule.

We supply both cards. For your own load, the Private AI/ML assessment (price fixed before work begins) covers model and GPU selection and a pilot plan with metrics. Describe the model and the expected load; the first call is free of charge.

Llama 3.1 8B inference and MLPerf® Training results

MLPerf® Inference v6.0, Datacenter, closed division, available systems, Llama 3.1 8B, offline and server scenarios, result 6.0-0021 (FP8 weights, TensorRT, CUDA 13.0); MLPerf® Training v6.0, closed division, available on-premise, Llama 2 70B LoRA and Llama 3.1 8B, results 6.0-0046 (XE7740) and 6.0-0047 (XE7745), NVIDIA NGC PyTorch 26.04. Retrieved from mlcommons.org on 24 September 2026, results verified by MLCommons Association. Every system has eight H200 NVL at 600 W. Training results are wall-clock minutes to the benchmark’s quality target.

On Llama 3.1 8B, which fits one card many times over, that is about 6,650 tokens per second per card offline and 5,630 under the server limits of 2 s to the first token and 100 ms per token at the 99th percentile, by our division of the system result, not an MLPerf® metric.

MLPerf® Training v6.0 lists two Dell servers with eight H200 NVL. The LoRA benchmark fine-tunes Llama 2 70B on SCROLLS GovReport until the cross-entropy loss falls to 0.925; the Llama 3.1 8B benchmark pretrains on C4 to a log perplexity of 3.3. Dell describes its runs as BF16 mixed precision with FP8 acceleration through NVIDIA’s Transformer Engine. With the same card model and software but different processors and host memory, the XE7745 (AMD EPYC 9655, 1.5 TB) took 35 per cent longer than the XE7740 (Intel Xeon 6767P, 3.072 TB) on LoRA and 14 per cent longer on pretraining. The results give no reason; with the same card model and software, the difference has to come from the rest of the system. Our fine-tuning comparison compares the H200 NVL with the RTX PRO 6000 by memory, tensor rates and the link between cards.

What NVIDIA claims

NVIDIA’s figures for this card are ratios. Its product page says that with up to four cards on NVLink and 1.5 times the memory, LLM inference runs up to 1.7 times and HPC applications up to 1.3 times faster than on the H100 NVL; its launch blog of 18 November 2024 repeats both and adds 2.5 times over the Ampere generation for HPC. We found no test conditions in either source. The benchmark charts on the product page are H200 SXM measurements: the 1.9 times on Llama 2 70B compares an H100 SXM at batch size 8 with an H200 SXM at batch size 32, per NVIDIA’s footnote. For absolute numbers on eight-card systems with the NVL card, MLPerf® Inference and Training results are the main source we found.

H200 NVL limits: power, NVLink, FP4 and vGPU

Power and air. The card draws up to 600 W, configurable down to 200 W, through a 16-pin cable whose sense pins must declare the 451 to 600 W class, or the card does not boot. It is passive, relies on the server’s fans, and NVIDIA’s product brief gives no airflow figure for it, only the direction of flow. Every system description in the MLPerf® results above that states a board power gives 600 W; none shows the card in a server whose maker caps it lower. Eight cards draw up to 4.8 kW before processors and fans. Our checklist for existing servers covers cables, fans, ducts and caps per chassis.

If the cards are to go into a server you already run, we check its power and airflow before you order. Tell us the server model and which slots the cards would use, and we say in advance if something will not work.

NVLink domains. A bridge joins two or four adjacent cards at 900 GB/s per GPU, against 128 GB/s over a PCIe Gen5 x16 link, both counting the two directions together. An eight-card server therefore holds at most two NVLink domains of four, by our reading, with PCIe between them, and tensor parallelism beyond four cards crosses PCIe, as our multi-GPU guide works through.

No FP4. Hopper computes FP8, and 4-bit weights run as INT4 through AWQ or GPTQ; TensorRT-LLM’s quantisation support matrix (TensorRT-LLM 1.4.0rc1, updated 19 October 2026) lists NVFP4 for Blackwell but not for Hopper. vLLM loads NVFP4 checkpoints on Hopper weight-only: the weights keep their 4-bit size in memory, but the arithmetic runs in 16-bit, which vLLM’s documentation says “may reduce throughput for compute-heavy workloads”. A model published natively in FP4 has no native path here, and we found no MLPerf® Inference result for the H200 NVL on Llama 3.1 405B, DeepSeek-R1 or gpt-oss-120b. Card counts for such models are in our large-model guide.

Sharing and software. MIG splits the card into up to seven 1g.18gb instances of 16.5 GB each; NVIDIA’s MIG user guide (updated 4 November 2026) describes the MIG support in NCCL 2.31 and later only as experimental (one rank per MIG instance, CUDA driver 13.0 or later, NCCL_MNNVL_ENABLE=0), so each instance runs single-GPU jobs. For virtual GPUs, NVIDIA lists only compute (C-series) vGPU types for this card, time-sliced or MIG-backed, from vGPU 18.1 and licensed through NVIDIA AI Enterprise. NVIDIA’s brief requires driver R565 TRD1 or later and adds CUDA 12.7, the CUDA version of that driver (no 12.7 toolkit exists; toolkits from 11.8 support Hopper). For video, NVIDIA lists decoders only: 7 NVDEC and 7 JPEG.

It is therefore not suited to virtual workstations and graphics, to FP4-native models, or to servers that cannot supply 600 W and the matching airflow to a slot. It suits interactive serving of 70B-class models, models that need two or four bridged cards, and double-precision work at 30 TFLOPS of FP64 and 60 on its Tensor Cores; our A100 replacement guide recommends it for such simulation work.

What we supply

We supply the NVIDIA H200 NVL with its two-way and four-way NVLink bridges, and the RTX PRO 6000 Server Edition compared above, on one EU contract and invoice with manufacturer warranty; each H200 NVL includes a five-year NVIDIA AI Enterprise subscription, which starts 90 days after the card ships to the server maker. We build AI servers to order around the cards, including a training node with eight H200 NVL and NVLink bridges, and check rack power and airflow before we quote. If you also need the software, our engineering partner Vixen.UNO delivers Private AI/ML: private LLMs on vLLM or NVIDIA AI Enterprise in your infrastructure, under the contract with Eurokommerz.

FAQ