NVIDIA H200 NVL: what MLPerf® Inference and Training measured, what it means per card and per user, and the limits the spec sheet leaves out

The card, and where the numbers come from

The H200 NVL is NVIDIA’s Hopper GPU on a dual-slot, full-length, passively cooled PCIe card: 141 GB of HBM3e at 4.8 TB/s, a board power configurable up to 600 W, 3,341 TFLOPS of FP8 in NVIDIA’s table, which counts sparsity (1,670.5 dense), and one connector for a two-way or four-way NVLink bridge at 900 GB/s per GPU. This review collects published measurements; we have not tested the card ourselves.

The richest source is the MLPerf® Inference and Training benchmark results: MLCommons publishes every result with its system description and run logs. Eight-card H200 NVL servers from Dell, Cisco and others appear in MLPerf Inference v5.0 (April 2025), v5.1 (September 2025) and v6.0 (1 April 2026), and two Dell servers in MLPerf Training v6.0 (June 2026). NVIDIA’s TensorRT-LLM performance overview of 21 September 2026 lists the H200 SXM but not the NVL card, and its inference performance page has no H200 NVL results. Independent measurements are scarce: in a review published on 23 September 2026, StorageReview included one H200 NVL in a Dell PowerEdge R770 as a comparison system for smaller models and measured output tokens per second with vLLM, at 512 input and 512 output tokens and at 8,192 and 1,024, from one to 128 concurrent streams. MLPerf Inference v6.1 was published on 16 September 2026; this review cites results up to v6.0.

Llama 2 70B in MLPerf Inference

The Llama 2 70B benchmark of MLPerf Inference answers questions from the OpenOrca dataset, about 290 generated tokens per answer by our arithmetic from the logs, in three scenarios. Offline hands the system all its samples at once and measures throughput with no latency limit. Server sends requests at random intervals and reports the throughput sustained while the 99th percentile of the time to first token (TTFT) stays within 2 s and that of the time per output token (TPOT), the average interval between generated tokens, within 200 ms. Interactive tightens the limits to 450 ms and 40 ms. The v6.0 result comes from Dell’s PowerEdge XE7740: eight cards at a 600 W board power, two Xeon 6747P processors and FP8 weights.

MLPerf Inference v6.0, Datacenter, closed division, available systems: Llama 2 70B (99 per cent of the reference accuracy), offline, server and interactive scenarios, result 6.0-0021, Dell PowerEdge XE7740 with eight H200 NVL; the latency limits apply to the 99th percentile. Every MLPerf result on this page: retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association. Software as Dell lists it: TensorRT 10.13 and 10.14, CUDA 13.0, driver 580.126.09. Per card is our division of the system result by its number of GPUs, not an MLPerf metric; mean TPOT is taken from the run logs. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The earlier round agrees. In MLPerf Inference v5.0, closed division, Dell’s PowerEdge XE7745 with eight cards (result 5.0-0017) reached 31,149.9 tokens per second offline on Llama 2 70B and 16,134.4 under the interactive limits, which v5.0 ran as a separate benchmark, Llama 2 70B Interactive, in the server scenario: within 3 per cent of v6.0 across two rounds, two chassis and TensorRT 10.8 to 10.14; the v5.0 files we read state no weight precision. These are stable reference figures for the card with NVIDIA’s TensorRT stack, which every H200 NVL result we read used; results with another engine may differ. The system descriptions name NVLink as the link between the cards, but none states how the bridges were fitted.

What one request gets

By our arithmetic from the mean TPOT in the run logs, not MLPerf metrics, each request received about 6.5 tokens per second in the server run and about 28 in the interactive run; dividing the throughput per card by those rates, some 560 and 72 requests per card were in progress at once on average. The server row is a heavily batched service in which each answer streams at roughly reading pace; the interactive row gives each request more than four times that speed for 56 per cent of the throughput.

On one card, the ceiling for a single request is set by memory bandwidth, because every generated token reads all the weights once: for a dense 70B model in FP8, about 70 GB, 4.8 TB/s allows about 68 tokens per second, a ceiling rather than a measurement. The interactive run gave each of its 72 requests per card about 40 per cent of that. With many requests in flight, the weights are read once per step for all of them and the work shifts towards arithmetic: NVIDIA wrote in March 2024 that after its TensorRT-LLM optimisations, Llama 2 70B on the H200 (SXM) “is compute performance bound rather than limited by memory bandwidth or communication bottlenecks”. The H200 NVL has the same peak compute specifications as the H100 NVL, so its gains over that card come mainly from memory; the older card’s lower power limit, 350 to 400 W against up to 600 W, can also cost it clock speed under sustained load, as our comparison of the two sets out.

On capacity, the driver reports 143,771 MiB, 140.4 GiB, the figure Dell also enters in its MLPerf Training system descriptions. By the sizing rule of our guide to concurrent users, 90 per cent of that, less 3 GiB of overhead and 67.7 GiB of Llama 3.3 70B weights in FP8, leaves 55.7 GiB of cache on one card: 44 sessions of 8,192 tokens with an FP8 cache at 1.25 GiB each.

Against the RTX PRO 6000 in the same server

For v6.0, Dell also ran the XE7740 with eight RTX PRO 6000 Blackwell Server Edition cards, likewise at 600 W with TensorRT on CUDA 13.0, but with FP4 weights, a format Hopper cannot compute.

MLPerf Inference v6.0, Datacenter, closed division, available systems, Llama 2 70B (99 per cent of the reference accuracy), offline and server scenarios, tokens per second for eight cards in Dell PowerEdge XE7740 systems: result 6.0-0021, H200 NVL with FP8 weights, and result 6.0-0022, RTX PRO 6000 Server Edition with FP4 weights, linked through PCIe switches; the processors, host memory and driver builds also differ. Percentages are our arithmetic, not MLPerf metrics.

In the two throughput scenarios the H200 NVL leads by only 18 and 14 per cent: by NVIDIA’s account of this benchmark, batched work leans on arithmetic, and the Blackwell card computes in FP4. The XE7740 with RTX PRO 6000 has no interactive entry. Result 6.0-0004, eight RTX PRO 6000 Server Edition cards with FP4 weights in another maker’s server, with its own processors and software, reached 6,238.1 tokens per second in the interactive scenario of the same round and division, which puts the H200 NVL at about 2.6 times per card by our arithmetic. The same system reached 27,812.9 tokens per second offline and 28,038.5 in the server scenario, so against it the H200 NVL led by 15 and 4 per cent in those scenarios. Under a 40 ms limit per token, speed per request counts, and the H200 NVL reads its memory three times as fast, 4.8 TB/s against 1,597 GB/s. Our comparison of the two cards turns this into a buying rule. The load that decides is still your own: the technical assessment of our AI/ML Integration service, delivered by our engineering partner Vixen.UNO, covers model and GPU selection and a pilot plan with metrics.

Smaller models and training

MLPerf Inference v6.0, Datacenter, closed division, available systems, Llama 3.1 8B, offline and server scenarios, result 6.0-0021 (FP8 weights, TensorRT, CUDA 13.0); MLPerf Training v6.0, closed division, available on-premise, Llama 2 70B LoRA and Llama 3.1 8B, results 6.0-0046 (XE7740) and 6.0-0047 (XE7745), NVIDIA NGC PyTorch 26.04. Retrieved from mlcommons.org on 24 September 2026, results verified by MLCommons Association. Every system has eight H200 NVL at 600 W. Training results are wall-clock minutes to the benchmark’s quality target.

On Llama 3.1 8B, which fits one card many times over, that is about 6,650 tokens per second per card offline and 5,630 under the server limits of 2 s to the first token and 100 ms per token at the 99th percentile, by our division of the system result, not an MLPerf metric.

MLPerf Training v6.0 lists two Dell servers with eight H200 NVL. The LoRA benchmark fine-tunes Llama 2 70B on SCROLLS GovReport until the cross-entropy loss falls to 0.925; the Llama 3.1 8B benchmark pretrains on C4 to a log perplexity of 3.3. Dell describes its runs as BF16 mixed precision with FP8 acceleration through NVIDIA’s Transformer Engine. With the same card model and software but different processors and host memory, the XE7745 (AMD EPYC 9655, 1.5 TB) took 35 per cent longer than the XE7740 (Intel Xeon 6767P, 3.072 TB) on LoRA and 14 per cent longer on pretraining; the results do not say why, but a published figure clearly belongs to a whole system, not to the card alone. Our fine-tuning comparison compares the H200 NVL with the RTX PRO 6000 by memory, tensor rates and the link between cards.

What NVIDIA claims

NVIDIA’s figures for this card are ratios. Its product page says that with up to four cards on NVLink and 1.5 times the memory, LLM inference runs up to 1.7 times and HPC applications up to 1.3 times faster than on the H100 NVL; its launch blog of 18 November 2024 repeats both and adds 2.5 times over the Ampere generation for HPC. Neither gives test conditions. The benchmark charts on the product page are H200 SXM measurements: the 1.9 times on Llama 2 70B compares an H100 SXM at batch size 8 with an H200 SXM at batch size 32, per NVIDIA’s footnote. For absolute numbers on eight-card systems with the NVL card, MLPerf Inference and Training results are the main source we found.

Limits the spec sheet does not show

Power and air. The card draws up to 600 W, configurable down to 200 W, through a 16-pin cable whose sense pins must declare the 451 to 600 W class, or the card does not boot. It is passive, relies on the server’s fans, and NVIDIA’s product brief gives no airflow figure for it, only the direction of flow. Every system description in the MLPerf results above that states a board power gives 600 W; none shows the card in a server whose maker caps it lower. Eight cards draw up to 4.8 kW before processors and fans. Our checklist for existing servers covers cables, fans, ducts and caps per chassis.

NVLink domains. A bridge joins two or four adjacent cards at 900 GB/s per GPU, against 128 GB/s over PCIe Gen5. An eight-card server therefore holds at most two NVLink domains of four, by our reading, with PCIe between them, and tensor parallelism beyond four cards crosses PCIe, as our multi-GPU guide works through.

No FP4. Hopper computes FP8, and 4-bit weights run as INT4 through AWQ or GPTQ; TensorRT-LLM supports NVFP4 only on Blackwell. A model published natively in FP4 has no native path here, and we found no MLPerf Inference result for the H200 NVL on Llama 3.1 405B, DeepSeek-R1 or gpt-oss-120b. Card counts for such models are in our large-model guide.

Sharing and software. MIG splits the card into up to seven 1g.18gb instances of 16.5 GB each; NCCL is not supported with MIG, so each instance runs single-GPU jobs. For virtual GPUs, NVIDIA lists only compute (C-series) vGPU types for this card, time-sliced or MIG-backed, from vGPU 18.1 and licensed through NVIDIA AI Enterprise. NVIDIA’s brief requires driver R565 TRD1 or later and adds CUDA 12.7, the CUDA version of that driver (no 12.7 toolkit exists; toolkits from 11.8 support Hopper). For video, NVIDIA lists decoders only: 7 NVDEC and 7 JPEG.

That makes it the wrong card for virtual workstations and graphics, for FP4-native models, and for servers that cannot feed 600 W and the airflow to a slot. It is the right one for interactive serving of 70B-class models, for models that need two or four bridged cards, and for double precision, at 30 TFLOPS of FP64 and 60 on its Tensor Cores, which is why our A100 replacement guide sends such simulation work to it.

What we supply

Eurokommerz supplies the NVIDIA H200 NVL EU-wide with manufacturer warranty, on one EU contract and invoice, together with its two-way and four-way NVLink bridges; each card comes with its five-year NVIDIA AI Enterprise subscription. We build AI servers to order, including a training node with eight H200 NVL and NVLink bridges, and check rack power and airflow before we quote. Our engineering partner Vixen.UNO delivers AI/ML Integration on top: a technical assessment with model and GPU selection and a pilot plan with metrics, then private LLMs on vLLM or NVIDIA AI Enterprise in your infrastructure, under the contract with Eurokommerz.

FAQ