H200 NVL for large LLMs: how many cards each model needs, and how they connect

NVLink bridges join up to four H200 NVL cards

The H200 NVL is a dual-slot, air-cooled PCIe card with 141 GB of HBM3e at 4.8 TB/s and a configurable power limit of up to 600 W. Its NVLink bridges let one model span up to four cards at more than PCIe speed. Two- or four-way bridges connect the cards at 900 GB/s per GPU, against 128 GB/s for PCIe Gen5, which NVIDIA summarises as GPU-to-GPU communication “7x faster than fifth-generation PCIe”. NVIDIA’s reference architecture states the size of that domain, “up to four GPUs connected by NVIDIA NVLink”, and the memory it pools: “the combined H200 GPU memory is 564 GB”.

NVIDIA lists partner servers with up to eight cards. The server makers document how the bridges are arranged. Dell’s technical guide for the PowerEdge XE7740 (Rev. A04, April 2026) states: “For four GPUs, one 4-way bridge is needed, while for eight DW H200 GPU cards, two bridges are required.” Cisco’s spec sheet for the UCS C845A M8 (Rev. A.11, 11 May 2026) allows eight cards as two 4-way or four 2-way bridges, each fully populated. Eight cards therefore form two domains of four, connected to each other over PCIe; our guide to what an 8× H200 NVL server runs sizes that layout model by model. That is the layout this article sizes against. Each card also ships with a five-year NVIDIA AI Enterprise subscription, which covers NIM microservices and NVIDIA support; open-source runtimes such as TensorRT-LLM and vLLM run without it, as our licensing article explains.

H200 NVL and NVFP4: FP8 and INT4, no FP4 arithmetic

Hopper added FP8 to the Tensor Cores; FP4 came with Blackwell. NVIDIA’s TensorRT-LLM documentation is explicit that “NVFP4 is only supported on NVIDIA Blackwell”, and its quantisation matrix lists neither NVFP4 nor MXFP4 for Hopper. Hopper runs FP8 in several scaling schemes, and 4-bit integer weights through AWQ and GPTQ, either with 16-bit activations (W4A16) or with FP8 activations (W4A8).

This matters for sizing, because a 4-bit footprint on a model card is often NVFP4, which only Blackwell runs natively. The NVFP4 version of Qwen3-235B is 134 GB, which would suggest one H200 NVL, but TensorRT-LLM does not support NVFP4 on Hopper, and vLLM runs it there only weight-only (W4A16): the weights keep their 4-bit size in memory, but the arithmetic runs in 16-bit, which vLLM’s documentation says “may reduce throughput for compute-heavy workloads”. For serving on Hopper, size against the FP8 checkpoint of the -2507 release at 236 GB, or an INT4 AWQ or GPTQ version if the model has one and its accuracy is acceptable. NVIDIA’s own DeepSeek-R1 recipe for TensorRT-LLM uses the FP8 checkpoint on Hopper and reserves the FP4 one for Blackwell. Which FP4 models run on Hopper and Ada, and how, is in our guide to FP4 models on the H200 NVL and Ada GPUs.

How many cards each model needs for its weights

Model cards and repository sizes on Hugging Face, September and October 2026; the Llama 3.1 405B FP8 figure is our sum from the checkpoint’s parameter types. “Cards” is the smallest of NVIDIA’s one, two, four or eight-card configurations that holds the weights by our users-per-card rule: 0.9 × the 140.4 GiB the driver reports, less 3 GiB per card, which gives 123.4, 246.7, 493.4 or 986.9 GiB; what remains is for the KV cache. Kimi-K2.6 and DeepSeek-V3.2 use multi-head latent attention, so with tensor parallelism in vLLM every card keeps the full KV cache of each conversation, and that cache must fit in one card’s spare memory, unless decode context parallelism (-dcp) splits it across the cards.

gpt-oss-120b is the exception to the FP4 rule, because its MXFP4 weights run on Hopper too: OpenAI states that it runs on a single 80 GB GPU such as the H100, and NVIDIA’s TensorRT-LLM performance table runs it on one H200 in FP8. One card is enough either way, with room left for its cache.

KV cache and context on top of the weights

The weights are only part of the memory a model needs. Every user in a conversation keeps a KV cache on the cards, and for long contexts the cache outgrows the spare memory quickly. Two model publishers give figures for this. GLM’s hardware table for GLM-4.5 gives four H200 as the minimum at FP8 and eight for the full 128K-token context, under stated conditions: speculative decoding, batches of up to eight and more than 1 TB of server memory. Qwen states that Qwen3-235B-A22B-2507 needs about 1,000 GB of GPU memory in total to use a context of one million tokens.

Our 70B sizing example shows the method on a smaller model: weights, then cache per token times context times users, then a reserve for the runtime. Here we use our users-per-card rule: 90 per cent of the memory the driver reports, less 3 GiB per card, less the weights. For Llama 3.1 405B in FP8, the four cards of a domain report 561.6 GiB to the driver, so 493.4 GiB are usable. The weights take about 487 GB, or 454 GiB, which leaves about 40 GiB for the cache. At 252 KiB per token in FP8, that is about 166,000 tokens, or five conversations of 32,000 tokens. Serving a larger team usually needs the second domain.

We size H200 NVL servers the same way. Tell us the model and the number of concurrent users, and we reply within one business day with a configuration and a quote.

One model across two NVLink domains

Inside a domain of four, the cards can split every layer between them over NVLink (tensor parallelism). Between the two domains of an eight-card server, traffic runs over PCIe at a seventh of that bandwidth, so a common layout splits the model into stages instead (pipeline parallelism), with each stage on one NVLink domain. vLLM’s documentation describes the same split across servers, with tensor parallelism within each server and pipeline parallelism between them. This layout works, but it adds some latency per token compared with one large NVLink domain.

That is also why NVIDIA’s published numbers for these models do not transfer directly. NVIDIA’s TensorRT-LLM performance overview (version 1.3.0rc29, updated 26 September 2026) reports Qwen3 235B in FP8 on four H200 at 3,268 output tokens per second per GPU with 1,024-token prompts and answers, and DeepSeek R1 in FP8 on eight at 1,620 per GPU. Those are 700 W H200 SXM GPUs in a DGX H200, where all eight share one NVLink domain. The four-GPU figure maps onto one H200 NVL domain better than the eight-GPU figure maps onto a server with two. For the H200 NVL itself, the published multi-card results come from MLPerf® benchmarks. In MLPerf® Inference v6.0, published in April 2026, a Dell PowerEdge XE7740 with eight H200 NVL processed 32,004 tokens per second on Llama 2 70B, 99 per cent accuracy variant, in the offline scenario (datacenter category, closed division, entry 6.0-0021, retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association). That is about 4,000 per card by our division of the system result by its number of GPUs, not an MLPerf® metric.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

H200 NVL configurations: one, two, four or eight cards

Memory is the sum of 141 GB per card; two-way and four-way bridges from NVIDIA’s H200 NVL specifications; two four-way domains in an eight-card server from Dell’s PowerEdge XE7740 technical guide (Rev. A04) and Cisco’s UCS C845A M8 spec sheet (Rev. A.11), read on 24 October 2026.

The server around eight cards is a 3U to 5U machine with eight double-width PCIe slots rated for 600 W each, such as Dell’s 4U PowerEdge XE7740, Lenovo’s 3U ThinkSystem SR675 V3 (with 5th Gen AMD EPYC processors) or Supermicro’s 5U SYS-521GE-TNRT. Eight cards at 600 W draw 4.8 kW before the processors, the memory and the fans, which puts the whole server beyond a single-phase 16 A circuit of about 3.7 kW; our article on how many GPUs fit in one server covers the power and airflow side. Order the NVLink bridges together with the cards; HPE’s QuickSpecs for the ProLiant DL380a Gen12 note that the 2-way and 4-way bridges for the H200 NVL “can only be supported with H200 NVL GPUs” and that the “10DW configuration does not support GPU NVL bridges”, so eight is the bridged maximum there.

If you need the whole server, we build it to order around the H200 NVL and its bridges, and check the rack, power and airflow before we quote. Send us the rack position and its circuit rating.

When the H200 NVL is not the right card

For models published natively in FP4, such as the expert weights of DeepSeek-V4-Flash (its successor V4.1-Flash-1017 of 17 October 2026, about 510 GB, is sized in our DeepSeek hardware guide), Hopper has no native path, and Blackwell cards are the better fit: the RTX PRO 6000 with NVFP4 on PCIe, or a DGX B300 with 2.1 TB of HBM3e in one NVLink domain, which our B300 comparison covers. Beyond about 1 TB of weights, eight H200 NVL leave little room for the cache that serving needs. For a 70B model, the choice is between one H200 NVL and two RTX PRO 6000 cards, which our 70B sizing example works through. Where an NVFP4 checkpoint exists, a server with four RTX PRO 6000 Server Edition cards, 384 GB with FP4 Tensor Cores, is the alternative to a four-way H200 NVL set; Qwen3-235B in NVFP4 (134 GB) fits two of its cards, and our inference node is built that way.

What we supply

We supply the NVIDIA H200 NVL with its two-way and four-way NVLink bridges, as cards for a server you already run or in an AI server built to order, EU-wide with manufacturer warranty, on one EU contract and invoice. Each card comes with its five-year NVIDIA AI Enterprise subscription. Send us the model and the context you need, and we return the card count, the bridge layout and the power budget; for a server you already run, we also check its power and airflow before you order.

FAQ