4× NVIDIA H200 NVL in one server: 564 GB in one NVLink domain, models, users and power

Four H200 NVL in one server: 564 GB in one NVLink domain

A server with four NVIDIA H200 NVL cards and one 4-way NVLink bridge has 564 GB of HBM3e in a single NVLink domain, of which 493.4 GiB holds model weights and KV cache by our users-per-card rule. That runs Llama 3.3 70B in FP8 with up to 340 conversations of 8K in flight, gpt-oss-120b with 384 conversations of 32K, or one copy of a 400B-class model such as Llama 4 Maverick in FP8. The four cards draw 2.4 kW at 600 W, and the whole server about 3.4 kW by our estimate.

Four is the largest number of H200 NVL cards one NVLink bridge can join. NVIDIA's product brief PB-12128-001 (April 2025) gives each card 18 NVLink links and "a single wide NVLink bridge per card", either a 2-slot bridge that joins two cards or a 4-slot bridge that joins four. NVIDIA's H200 page lists "2- or 4-way NVIDIA NVLink bridge: 900GB/s per GPU" against 128 GB/s for PCIe Gen5, so inside the domain every card reaches the others at seven times the PCIe rate.

NVIDIA H200 page and product brief PB-12128-001_v01 (11 April 2025), NVIDIA MIG user guide (13 October 2026), read on 24 October 2026; 140.4 GiB is the memory the driver reports.

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for CUDA graphs and activations: 0.9 × 140.4 − 3 = 123.36 GiB per card. What is left after the weights is KV cache. The derivation is in our guide to how many users one RTX PRO 6000 can serve, and the same rule sizes every H200 NVL page on this site.

Which models fit 493 GiB, and what is left for the cache

Hopper has FP8 Tensor Cores and no FP4 arithmetic, so the H200 NVL is sized against FP8 or INT4 weights. An NVFP4 checkpoint built for Blackwell runs on it in vLLM only as a weight-only format, with the arithmetic in 16-bit, as our guide to FP4 models on the H200 NVL explains. gpt-oss-120b is the exception, because its MXFP4 weights also run on Hopper.

Our estimates by the users-per-card rule. Weights: Hugging Face repositories read on 24 October 2026 (Llama 3.3 70B FP8, gpt-oss-120b, Qwen3-235B FP8, Llama 4 Maverick FP8 in 84 shards of up to 4.99 GB); Llama 3.1 405B and DeepSeek-V3.2 from our guide to large models on the H200 NVL.

The domain is what makes the 400B class possible on four PCIe cards. Llama 4 Maverick takes about 388 GiB of the 493.4 GiB, and Llama 3.1 405B about 454 GiB, which leaves room for roughly five conversations of 32K at 252 KiB per token in an FP8 cache. Such a server holds a large model for a small team, and a 70B or 235B-class model for a large one. Our guide to how many H200 NVL cards large models need lists more models, and the eight-card H200 NVL server covers DeepSeek-V3.2, GLM-5.3 and Kimi K2.

Users per layout: four copies, two pairs or one TP=4 copy

Four cards can run a model in three layouts. Four copies, one per card, exchange nothing between the cards. Two pairs split each copy over two cards with tensor parallelism (TP=2). One copy spans all four (TP=4). Every copy stores its own weights, so fewer copies leave more memory for conversations, and inside an NVLink domain the split costs far less than it does over PCIe.

Our estimates by the users-per-card rule, concurrent conversations at full context length; 70B is Llama 3.3 70B in FP8 and Qwen3-235B the FP8 checkpoint, gpt-oss-120b with a 16-bit cache. Cache per conversation: Llama 3.3 70B 2.5 GiB (16-bit) or 1.25 GiB (FP8) at 8,192 tokens; gpt-oss-120b 1.125 GiB at 32K; Qwen3-235B 188 KiB per token in 16-bit, 5.9 GiB at 32K, from its config.json.

For the TP=4 column of Llama 3.3 70B, four cards hold 493.4 GiB; after 67.7 GiB of weights, 425.7 GiB remain, enough for 170 conversations at 2.5 GiB or 340 at 1.25 GiB. One card alone leaves 55.7 GiB, 22 or 44 conversations. These are requests in flight at the same moment, which a company's gateway shows at the busy hour; a staff of 1,000 with an assistant open in every browser produces far fewer at once.

Speed per user follows the bandwidth. By our estimate, one card reading the 70.6 GB of FP8 weights per token gives a single user at most 68 tokens per second at 4.8 TB/s; split over four cards, each card reads a quarter, which raises that ceiling towards 270 before the NVLink exchanges are counted. Four copies give the best isolation, since a failed card takes out a quarter of the service. One TP=4 copy holds the most conversations and gives the fastest single stream. We build this server with the bridge layout your models need; send us the models, context length and peak requests in flight, and our team replies with a configuration and the figures for each layout.

One 4-way bridge or two 2-way bridges

Four cards take either bridge set. Cisco's spec sheet for the UCS C845A M8 (revision A.11, 11 May 2026) lists two 2-way bridges or one 4-way bridge for four H200 NVL, and requires each bridge to be fully populated. HPE lists both the 2-way bridge S4A90C and the 4-way bridge S4A91C for the ProLiant Compute DL380a Gen12. Dell's technical guide for the PowerEdge XE7740 (revision A04, April 2026) describes one 4-way bridge for four cards and two for eight.

Cisco UCS C845A M8 spec sheet A.11; HPE DL380a Gen12 QuickSpecs; Dell XE7740 technical guide A04; NVIDIA product brief PB-12128-001_v01 section 4.5; vLLM custom all-reduce source and vLLM 0.30.0 release notes (22 September 2026), read on 24 October 2026.

NVIDIA's product brief states that bridged cards "should be within the same CPU domain" and recommends "the same (even) number of GPUs under each CPU socket". NVIDIA's reference architecture puts it the same way: pairing under the same CPU socket is best, across sockets "acceptable but not recommended". In a two-socket server with two cards under each processor, two 2-way pairs follow that advice exactly, and a 4-way bridge spans both sockets. Dell's slot table shows the trade-off: the first four cards go into slots 25, 27, 29 and 31, and the guide connects slots 21 to 28 to the first processor and 29 to 36 to the second, so by our reading a four-card XE7740 bridges two cards under each processor, which the same guide calls possible but not recommended.

In practice the 4-way bridge pays off when one copy needs more than 280 GiB, or when TP=4 should run on vLLM's custom all-reduce, which its source enables on more than two GPUs only when they are fully connected over NVLink, as the four cards on a 4-way bridge are by our reading. That is the default: vLLM 0.30.0 (release notes of 22 September 2026, #53576) added an opt-in FlashInfer all-reduce over PCIe IPC for systems without NVLink. Two pairs suit a server that runs two models, or two copies of a model that fits 280 GiB. Our article on the H200 NVL NVLink bridge, 2-way or 4-way shows how to check the result with nvidia-smi topo -m.

2U or 4U, and room to grow to eight cards

The H200 NVL is a passive, dual-slot, full-length card of 1,217 g, and it runs only in a server whose maker lists it with the bridge you want.

Supermicro product page; Lenovo Press LP2128 (20 October 2026); Dell XE7740 technical guide A04; HPE DL380a Gen12 QuickSpecs; Cisco UCS C845A M8 spec sheet A.11; read on 24 October 2026.

The 2U option is narrow. Supermicro lists four double-width cards including the H200 NVL in a single-socket 2U chassis with three 2,000 W supplies in 2+1, and its option list shows a GPU bay for two cards with NVLink, so by our reading four cards there form two pairs. NVIDIA's certified-systems guide asks for at least two sockets, which a single-socket server does not meet. Lenovo's 2U SR650a V4 lists two H200 NVL at 600 W.

A 4U chassis built for eight is the safer start for four. The cards, bridges and memory stay; growth adds four cards, a second 4-way bridge or two more pairs, and power supplies. HPE requires five 2,400 or 3,200 W supplies for two or four double-wide GPUs and eight for eight; Dell marks 512 GB and 1 TB of memory as not validated for eight H200 NVL and sets a 30 °C inlet limit for that configuration. Since the bridge spans adjacent slots, ask the maker how the bridge positions change from four cards to eight before you order, as our guide to planning a GPU server for growth recommends.

Processors, host RAM and network for four cards

The three published host-memory rules give three different answers for four cards.

NVIDIA reference architecture, compute node hardware; NVIDIA-Certified configuration guide (30 September 2026); HPE DL380a Gen12 QuickSpecs; Cisco spec sheet A.11; read on 24 October 2026.

We size H200 servers by the 2× rule, as for all our AI servers: two AMD EPYC 9005 processors have 24 memory channels, so 24 modules of 48 GB give 1,152 GB with one module per channel. The 3× level suits fine-tuning with CPU offload or a KV cache that spills into system memory; our guide to how much RAM a GPU server needs works through the cases.

NVIDIA's reference architecture asks for at least seven physical cores per GPU, two more per GPU for the operating system or virtualisation, and two per MIG instance. Four cards therefore need 36 cores, and 92 with all 28 MIG instances; two 32-core EPYC 9355 give 64. It also recommends one PCIe Gen5 x16 link per GPU and at least 1 TB of NVMe per processor socket for inference, 2 TB for training. For networking, NVIDIA's reference design pairs one 400G SuperNIC with every two GPUs, so a four-card node joining a cluster takes two; a node serving chat alone needs 25 GbE for users and a faster link to the model store.

Power: 2.4 kW of cards on EU feeds

NVIDIA's product brief gives the H200 NVL a board power of 600 W maximum by default and 200 W minimum, with one 16-pin connector; if the cable signals less than the default cap, "the card will not boot".

Our estimate: board power from NVIDIA's product brief, 280 W per processor for two AMD EPYC 9355, plus 15 per cent for memory, drives, network cards, fans and conversion losses.

At 600 W the server is above the 2.94 kW we plan per 230 V 16 A single-phase feed at 80 per cent, so its rack position needs 32 A single-phase (5.89 kW planned) or a 400 V three-phase feed. With redundant supplies on an A and a B feed, each side carries the whole server, and a measurement on the delivered system confirms or corrects our 15 per cent. About 3.4 kW of heat needs roughly 540 CFM of air at an 11 °C rise, by the 160 CFM per kW in our guide to GPU rack power and cooling. Our page on GPU server power on 16 A, 32 A and three-phase feeds covers the circuits.

A cap at 450 W keeps the server within a 16 A feed by our estimate, at a cost in throughput that nobody has published for this card; our page on the H200 NVL at 450 W collects what is known. We check rack, feed and airflow before we quote; tell us the feed type, the A/B requirement and the inlet temperature, and we say which power mode and chassis fit.

Published results for the H200 NVL

MLPerf® Inference v6.1 has no H200 NVL entry. The nearest published system result has eight cards: in MLPerf® Inference: Datacenter v6.0, closed division, Dell's PowerEdge XE7740 with eight H200 NVL reached 32,004.0 tokens per second on Llama 2 70B (llama2-70b-99) in the offline scenario, 29,084.9 in the server scenario and 16,343.8 in the interactive scenario, entry 6.0-0021, retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association. That is about 4,000 tokens per second per card offline, our division of the system result, not an MLPerf metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

StorageReview ranked enterprise GPUs on 14 October 2026 with vLLM 0.27.1 and tested one H200 NVL at 450 W, "the most a Dell PowerEdge R770 GPU slot delivers". It reached 3,435 output tokens per second on Ministral 3 14B in FP8 at 32 streams and 7,336 at 128, and ran 50 to 81 per cent ahead of the RTX PRO 6000 Workstation Edition on three FP8 and BF16 models at 32 streams, but behind it on an NVFP4 model, 467 against 616. Our H200 NVL benchmarks cover the per-user figures.

MIG, vGPU and the included NVIDIA AI Enterprise

NVIDIA's MIG guide lists up to seven instances of 1g.18gb per card, or one of 7g.141gb, so four cards offer up to 28 isolated GPUs, as our H200 NVL MIG profiles show. For virtual machines the H200 NVL has compute vGPU (C-series) only, through NVIDIA AI Enterprise, and no graphics vGPU; NVIDIA lists it in the AI Enterprise 8.2 support matrix, whose compute vGPU hypervisors include VMware vSphere 8.0 and later.

NVIDIA's licensing guide (updated 2 September 2026) states: "Each NVIDIA H200 NVL Tensor Core GPU includes a five-year NVIDIA AI Enterprise subscription." The term starts 90 days after the board ships to the server maker, "to account for integration and final delivery", cannot be changed, and is activated with the GPU serial number; the software runs on the certified system where the GPUs are installed. A four-card server therefore brings four subscriptions, as our article on activating NVIDIA AI Enterprise explains. Open-source engines such as vLLM run without it.

Four H200 NVL or four RTX PRO 6000 Server Edition

The RTX PRO 6000 Blackwell Server Edition is the other four-card option we build. The H200 NVL has more memory per card, three times the bandwidth, NVLink and 30 TFLOPS of FP64; the Server Edition has FP4 Tensor Cores and graphics vGPU.

NVIDIA product pages; conversations at 8K with an FP8 cache by our users-per-card rule; Server Edition figures from our guide to the four-card RTX PRO 6000 Server Edition server.

The H200 NVL suits large FP8 models, many users of a 70B model and fine-tuning; the Server Edition suits NVFP4 models and mixed graphics and AI work, as our four RTX PRO 6000 or four H200 NVL comparison and our guide to the four-card RTX PRO 6000 Server Edition server explain.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight H200 NVL and their NVLink bridges, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, drivers, CUDA and a container runtime on request. The configuration and the ordering steps are on our page to buy a 4× H200 NVL server; the cards alone are on our H200 NVL page. Models, RAG and the platform on top are our Private AI/ML service.

FAQ