H200 NVL for 235B to 671B models: how many cards, and how they connect

The unit of capacity is four cards

The H200 NVL is a dual-slot, air-cooled PCIe card with 141 GB of HBM3e at 4.8 TB/s and a configurable power limit of up to 600 W. Its interconnect is what makes large models possible on PCIe servers: two- or four-way NVLink bridges connect the cards at 900 GB/s per GPU, against 128 GB/s for PCIe Gen5, which NVIDIA summarises as GPU-to-GPU communication “7x faster than fifth-generation PCIe”. NVIDIA’s reference architecture states the size of that domain, “up to four GPUs connected by NVIDIA NVLink”, and its arithmetic: “the combined H200 GPU memory is 564 GB”.

NVIDIA’s partner servers take up to eight cards. NVIDIA does not spell out in one sentence how the bridges sit in such a server, but a four-card NVLink limit and eight cards per server mean two domains of four, joined to each other over PCIe. That is the layout this article sizes against. Each card also ships with a five-year NVIDIA AI Enterprise subscription, which covers NIM microservices and NVIDIA support; open-source runtimes such as TensorRT-LLM and vLLM run without it, as our licensing article explains.

FP8 yes, FP4 no

Hopper added FP8 to the Tensor Cores; FP4 came with Blackwell. NVIDIA’s TensorRT-LLM documentation is explicit that “NVFP4 is only supported on NVIDIA Blackwell”, and its quantisation matrix lists neither NVFP4 nor MXFP4 for Hopper. What Hopper does run is FP8 in several scaling schemes and 4-bit integer weights through AWQ and GPTQ, either with 16-bit activations (W4A16) or with FP8 activations (W4A8).

That changes the arithmetic. A 4-bit footprint on a model card is often NVFP4, which only Blackwell runs natively: the NVFP4 version of Qwen3-235B is 134 GB, which would suggest one H200 NVL, but TensorRT-LLM does not support NVFP4 on Hopper, and vLLM runs it there only weight-only, with 16-bit arithmetic. On Hopper the choices are the FP8 checkpoint at 236 GB, or an INT4 AWQ or GPTQ version if the model has one and its accuracy is acceptable. NVIDIA’s own DeepSeek-R1 recipe for TensorRT-LLM uses the FP8 checkpoint on Hopper and reserves the FP4 one for Blackwell.

Weights, model by model

Model cards and repository sizes on Hugging Face, September 2026; the Llama 3.1 405B FP8 figure is our sum from the checkpoint’s parameter types. “Cards” is the smallest of NVIDIA’s one, two, four or eight-card configurations whose 141 GB per card hold the weights, before any cache or runtime; by capacity alone, three cards would hold GLM-4.5, Qwen3.5-397B and Llama 4 Maverick in FP8, but the bridges join two or four.

gpt-oss-120b is the exception to the FP4 rule, because its MXFP4 weights run on Hopper too: OpenAI states that it runs on a single 80 GB GPU such as the H100, and NVIDIA’s TensorRT-LLM performance table runs it on one H200 in FP8. One card is enough either way, with room left for its cache.

What comes on top

Weights are the floor. Every user in a conversation keeps a KV cache on the cards, and for long contexts the cache outgrows the spare memory quickly. Two model publishers put numbers on it. GLM’s hardware table for GLM-4.5 gives four H200 as the minimum at FP8 and eight for the full 128K-token context, under stated conditions: speculative decoding, batches of up to eight and more than 1 TB of server memory. Qwen states that Qwen3-235B-A22B-2507 needs about 1,000 GB of GPU memory in total to use a context of one million tokens.

Our 70B sizing example shows the method on a smaller model: weights, then cache per token times context times users, then a reserve for the runtime, which we put at ten per cent of the memory here. Applied to Llama 3.1 405B in FP8: the four cards of a domain report 561.6 GiB to the driver, ten per cent of that stays in reserve, and the weights take about 487 GB, or 454 GiB, which leaves about 52 GiB for the cache. At 252 KiB per token in FP8, that is about 215,000 tokens, or six conversations of 32,000 tokens. Serving a larger team usually needs the second domain.

Crossing the domain

Inside a domain of four, the cards can split every layer between them, tensor parallelism, over NVLink. Between the two domains of an eight-card server, traffic runs over PCIe at a seventh of that bandwidth, so a common layout splits the model into stages instead, pipeline parallelism, with each stage on one NVLink domain, the same split vLLM’s documentation describes across servers: tensor parallelism inside, pipeline parallelism between. It works, and it costs some latency per token compared with one large NVLink domain.

That is also why NVIDIA’s published numbers for these models do not transfer directly. NVIDIA’s TensorRT-LLM performance table, updated in September 2026, reports Qwen3 235B in FP8 on four H200 at 3,268 output tokens per second per GPU with 1,024-token prompts and answers, and DeepSeek R1 in FP8 on eight at 1,620 per GPU. Those are 700 W H200 SXM GPUs in a DGX H200, where all eight share one NVLink domain. The four-GPU figure maps onto one H200 NVL domain better than the eight-GPU figure maps onto a server with two. For the H200 NVL itself, the published multi-card results come from MLPerf® benchmarks: in MLPerf Inference v6.0, published in April 2026, a Dell PowerEdge XE7740 with eight H200 NVL processed 32,004 tokens per second on Llama 2 70B, 99 per cent accuracy variant, in the offline scenario (datacenter category, closed division, entry 6.0-0021, retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association), about 4,000 per card by our division of the system result by its number of GPUs, not an MLPerf metric.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Configurations that make sense

Memory is the sum of 141 GB per card; two-way and four-way bridges from NVIDIA’s H200 NVL specifications; two four-way domains in an eight-card server is our reading of NVIDIA’s four-GPU NVLink limit.

The server around eight cards is a 3U to 5U machine with eight double-width PCIe slots rated for 600 W each, such as Dell’s 4U PowerEdge XE7740, Lenovo’s 3U ThinkSystem SR675 V3 or Supermicro’s 5U SYS-521GE-TNRT. Eight cards at 600 W draw 4.8 kW before the processors, the memory and the fans, which puts the whole server beyond a single-phase 16 A circuit of about 3.7 kW; our article on how many GPUs fit in one server covers the power and airflow side. The bridges are part of the order, not an accessory: HPE’s documentation notes that H200 NVL bridges work only with H200 NVL cards.

When the H200 NVL is the wrong tool

For models published natively in FP4, such as the expert weights of DeepSeek-V4-Flash, Hopper has no native path, and the Blackwell cards are the natural home: the RTX PRO 6000 with NVFP4 on PCIe, or a DGX B300 with 2.1 TB of HBM3e in one NVLink domain, which our B300 comparison covers. Beyond about 1 TB of weights, eight H200 NVL leave little room for the cache that serving needs. And for a 70B model, one H200 NVL or two RTX PRO 6000 cards is the question, which our 70B sizing example answers.

What we supply

Eurokommerz supplies the NVIDIA H200 NVL EU-wide with manufacturer warranty, together with the two-way and four-way NVLink bridges and the server built to order around them. Each card comes with its five-year NVIDIA AI Enterprise subscription. Send us the model and the context you need, and we will return the card count, the bridge layout and the power budget.

FAQ