Eight NVIDIA H200 NVL in one server: models, users and the NVLink layout

Eight H200 NVL in one server: 1,128 GB in two NVLink domains

A server with eight NVIDIA H200 NVL has 1,128 GB of HBM3e, 141 GB per card, of which 986.9 GiB holds model weights and KV cache by our users-per-card rule. The cards are not one pool: two 4-way NVLink bridges join them into two domains of four, and the two domains talk over PCIe 5.0. That runs Llama 3.3 70B for several hundred conversations at once, or one model of 690 to 760 GB such as DeepSeek-V3.2 or GLM-5.3 for a few dozen users at 32K.

The layout comes from the server makers. Dell's technical guide for the PowerEdge XE7740 (Rev. A04, April 2026) states: "For four GPUs, one 4-way bridge is needed, while for eight DW H200 GPU cards, two bridges are required." Cisco's spec sheet for the UCS C845A M8 (Rev. A.11, May 2026) allows eight cards as two 4-way or four 2-way bridges, and OVHcloud's guide for its eight-card H200 NVL instances, updated on 1 October 2026, describes two sets of four GPUs with NVLink only inside each set and PCIe between the two sets. Inside a domain, NVIDIA gives 900 GB/s per GPU over NVLink; across the PCIe Gen5 link, 128 GB/s. Our article on the H200 NVL NVLink bridge, 2-way or 4-way covers the bridge mechanics.

NVIDIA H200 product page and product brief PB-12128-001_v01 (11 April 2025), read on 24 October 2026; 140.4 GiB is the 143,771 MiB the driver reports, as in our H200 NVL benchmark article.

Our users-per-card rule takes 90 per cent of the memory the driver reports, the default gpu_memory_utilization of vLLM before release 0.20.0, and subtracts 3 GiB per card for CUDA graphs and activations: 0.9 × 140.4 − 3 = 123.4 GiB. What remains after the weights is KV cache. The derivation is in our guide to how many users one GPU serves.

Which LLMs fit, and how much memory is left

Four layouts are possible: eight copies, one per card; four TP=2 pairs; two TP=4 copies, one per NVLink domain; or one copy over all eight cards, as TP=4 × PP=2 or with expert parallelism for some MoE models. Each copy stores its own weights, so fewer copies leave more room for conversations, and a split that crosses PCIe costs time on every step.

Our estimates by the users-per-card rule, conversations at full length, 16-bit KV cache unless stated. Weights: Hugging Face repository sizes as in our model guides (September and October 2026). Qwen3-235B: 188 KiB of 16-bit cache per token from its config.json (94 layers, 4 KV heads, head dimension 128), read on 24 October 2026. DeepSeek-V3.2, GLM-5.3 and Kimi figures from our DeepSeek, GLM and Kimi K2 guides; Kimi-K2.6 with decode context parallelism as reported by vLLM's Kimi-K2-Thinking recipe.

Hopper has no FP4 arithmetic, so the table sizes against FP8 or INT4 checkpoints; an NVFP4 checkpoint built for Blackwell runs here only as a weight-only format, as our guide to H200 NVL card counts for large models explains. The three large rows differ in layout because the published serving recipes differ. vLLM's DeepSeek-V3.2 recipe advises against TP=8 with its sparse attention kernel and recommends data parallelism with expert parallelism; the GLM-5.3 recipe states that the FP8 checkpoint fits a single eight-GPU H200 node; Kimi K2 keeps its compressed cache in full on every card under tensor parallelism, so decode context parallelism multiplies the usable cache. The details are in our DeepSeek, GLM and Kimi K2 hardware guides.

Concurrent users per layout for a 70B model

Llama 3.3 70B is the model most companies name first. NVIDIA's FP8 checkpoint is 72.7 GB, 67.7 GiB, and each conversation of 8,192 tokens takes 2.5 GiB of 16-bit cache or 1.25 GiB in FP8.

Our estimate by the users-per-card rule, conversations of 8,192 tokens at full length; whole conversations per copy, rounded down, times the number of copies.

For the third row, one domain holds 4 × 123.36 = 493.4 GiB; after 67.7 GiB of weights, 425.7 GiB remain, and 425.7 ÷ 1.25 gives 340 conversations per copy, 680 for two. One card alone leaves 55.7 GiB, 44 conversations with an FP8 cache.

These are requests in flight at the same moment. A company of 2,000 staff with an assistant open all day produces a small fraction of that at its busiest hour, so size against the peak your gateway logs show. For a 70B model, the two-domain layout is the usual choice: it keeps every all-reduce on NVLink, and a failed card stops only the copy it belongs to. The fourth row adds little over the third for a model that fits one domain. We build this server and size the layout per model; send us your models, context length and peak requests in flight, and our team replies with the figures for each layout.

One model across both domains: TP=4 × PP=2

A model of more than about 490 GiB has to span both domains. Tensor parallelism over all eight cards would send every layer's all-reduce across PCIe, and OVHcloud's guide says to "avoid using --tensor-parallel-size 8". Its command for all eight cards splits the model into two pipeline stages, and the four cards of each NVLink domain run one stage with TP=4:

• export CUDA_DEVICE_ORDER=PCI_BUS_ID
• vllm serve with --pipeline-parallel-size 2 and --tensor-parallel-size 4

Only the activations between the two halves of the model cross PCIe, once per step. For two separate TP=4 copies, the same guide starts one server per domain with CUDA_VISIBLE_DEVICES=0,1,2,3 and 4,5,6,7 on two ports. The numbering has to match the bridges: nvidia-smi topo -m shows which cards share NVLink, and NVIDIA's reference architecture for the H200 NVL states that "pairing of GPU cards under the same CPU socket is best". Dell wires slots 21 to 28 to the first processor and 29 to 36 to the second, and calls a 4-way bridge across the two processors possible but not recommended. Our article on one model on several GPUs over PCIe or NVLink explains the trade-offs between tensor, pipeline and expert parallelism.

Published MLPerf® results for eight H200 NVL

The published multi-card results for this card come from MLPerf® Inference, which records every system with its hardware and software.

MLPerf® Inference: Datacenter v6.0, closed division, available category, Llama 2 70B (llama2-70b-99), offline, server and interactive scenarios, entry 6.0-0021, Dell PowerEdge XE7740 with eight H200 NVL, retrieved from mlcommons.org on 24 September 2026. Result verified by MLCommons Association. Per card is our division of the system result, not an MLPerf metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The interactive scenario limits the time to the first token to 450 ms and the time per output token to 40 ms at the 99th percentile, so it is the closest to a chat service with fast answers. A year earlier, in MLPerf® Inference: Datacenter v5.0, closed division, Dell's PowerEdge XE7745 with eight cards reached 31,149.9 tokens per second offline on the same benchmark (entry 5.0-0017, retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association), within 3 per cent of v6.0. The v6.1 round of 16 September 2026 contains no H200 NVL entry. Our H200 NVL benchmarks compare these results with eight RTX PRO 6000 Server Edition cards in the same chassis.

Fine-tuning on eight cards: LoRA, QLoRA and full fine-tuning

Training needs more memory than serving. Mixed-precision Adam holds 16 bytes per parameter before activations, by the ZeRO paper's count, so the model states of Llama 3.3 70B, with 70.55 billion parameters, come to about 1,129 GB, or 1,051 GiB. That exceeds the 986.9 GiB of our rule; spread over eight cards it is 141.1 GB per card, as much as the 141 GB the card is listed with, before any activation.

Model states from our guide to GPU memory for fine-tuning (ZeRO paper count, rank-16 adapters on all linear layers, GB = 10⁹ bytes), divided over the cards by our arithmetic and compared with the listed capacity of 141 GB per card, not the figure the driver reports; activations come on top.

A full fine-tune of a 70B model therefore needs offloading. ZeRO-Offload moves optimiser states and gradients to CPU memory and runs the update on the processors; the FP32 optimiser states alone are 12 bytes per parameter, about 847 GB for this model, which the 2.3 TB of host memory in our training node holds by our arithmetic. The step time rises, so LoRA is the practical method for most company projects. A job that fits four cards should stay inside one NVLink domain, because sharding traffic between the domains crosses PCIe. Our comparison of the H200 NVL and RTX PRO 6000 for fine-tuning shows how the link between cards affects these runs.

MIG for many teams: up to 56 instances

MIG splits each card into isolated instances with their own memory and compute. NVIDIA's MIG user guide, updated on 13 October 2026, lists for the H200 141 GB product 1g.18gb up to seven times, 1g.35gb up to four, 2g.35gb up to three, 3g.71gb up to two, and 4g.71gb or 7g.141gb once. Eight cards therefore give up to 56 instances of 18 GB, for example for development teams, notebooks or small models next to production serving.

MIG is set per card. One domain can serve a TP=4 copy while two cards of the other run in 1g.18gb slices, but a split model runs on whole cards, because NVIDIA states that NCCL does not work with MIG. NVIDIA's reference architecture asks for two processor cores per MIG instance on top of the cores per GPU, so a fully split server needs over 180 cores by that rule. MIG on bare metal needs no licence; MIG-backed vGPU for virtual machines is compute vGPU through NVIDIA AI Enterprise, which comes with each card for five years, as our H200 NVL MIG profiles explain.

Host, power and rack for the eight-card server

NVIDIA's reference architecture for this card is "2-8-5": two sockets, eight GPUs and five network adapters, with at least seven cores per GPU plus two per GPU for the operating system, at least 128 GB of system memory per GPU and at least 2 TB of NVMe per socket for training. NVIDIA's configuration guide for NVIDIA-Certified Systems, updated on 30 September 2026, asks for system memory of at least twice the GPU memory, and Dell marks 512 GB and 1 TB of memory as not validated for eight H200 NVL in the XE7740. Our training node follows the larger figure: two AMD EPYC processors, 2.3 TB of DDR5 ECC, eight NVMe drives and 400G networking.

Our estimate: board power from NVIDIA's product brief, two 64-core AMD EPYC 9555 at 360 W each as listed in Cisco's C845A M8 spec sheet, plus 15 per cent for memory, drives, network cards, fans and conversion losses, an assumption that a measurement on the delivered system confirms or corrects.

We plan a 230 V 32 A single-phase feed at 5.89 kW, so at 600 W per card the server needs a 400 V three-phase position (8.87 kW planned at 16 A) on each of the A and B feeds. About 6.3 kW of heat needs roughly 1,000 CFM of air at an 11 °C rise, by the 160 CFM per kW in our guide to GPU rack power and cooling, and Dell sets the inlet limit for eight H200 NVL in the XE7740 at 30 °C, against 35 °C for other cards. NVIDIA's blog of November 2024 puts about 70 per cent of enterprise racks at 20 kW or below with air cooling: such a rack takes two of these servers with room for switches. The power cap and what it costs are covered in H200 NVL at 450 W. We check rack, feed and airflow before we quote; tell us the feed type and inlet temperature at the rack position, and we say whether the server fits.

When another configuration fits better

Four H200 NVL on one bridge, 493.4 GiB by our rule, hold the same 70B service for half the users and every model up to Llama 4 Maverick in FP8, as our guide to the 4× H200 NVL server shows. Where the service must survive the loss of a whole server, two four-card servers are safer, as one 8-GPU server or two 4-GPU servers compares. Models published in FP4 run natively on Blackwell: eight RTX PRO 6000 Server Edition cards compute NVFP4 on PCIe, as our eight-card RTX PRO 6000 allocation shows. A model that needs TP=8 in one NVLink domain of eight, or more than 1 TB of weights with room for users, belongs on a DGX B300 with 2.1 TB of HBM3e, which we quote per project, as our H200 NVL vs H200 SXM comparison explains.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. Our training node has eight H200 NVL with two 4-way NVLink bridges, two AMD EPYC processors, 2.3 TB of DDR5 ECC, eight NVMe drives and 400G networking, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component; the configuration and ordering steps are on our page to buy an 8× H200 NVL server. Models, RAG and the serving platform on top are our Private AI/ML service.

FAQ