Two NVIDIA H200 NVL on a 2-way NVLink bridge: what 282 GB runs, for how many users

Two H200 NVL, one bridge: 282 GB in one NVLink domain

A server with two NVIDIA H200 NVL cards on a 2-way NVLink bridge has 282 GB of HBM3e, of which 246.7 GiB holds model weights and KV cache by our users-per-card rule. That runs Llama 3.3 70B at full BF16 precision for 46 conversations of 8K tokens at once with a 16-bit cache, the same model in FP8 with an FP8 cache for about 140, or gpt-oss-120b for about 165 conversations of 32K. The two cards draw 1.2 kW at their default 600 W, and the whole server about 2.0 kW by our estimate.

The bridge is what makes the pair one domain. NVIDIA's product brief PB-12128-001 (April 2025) specifies "a single wide NVLink bridge per card", in a 2-slot version for two cards and a 4-slot version for four, and NVIDIA's H200 page lists "2- or 4-way NVIDIA NVLink bridge: 900GB/s per GPU" against 128 GB/s over PCIe Gen5. A model split over the two cards exchanges its partial results over that link on every layer, about seven times faster than it could over PCIe.

NVIDIA H200 page and product brief PB-12128-001_v01 (11 April 2025), NVIDIA MIG user guide (updated 13 October 2026), read on 24 October 2026; 140.4 GiB is the memory the driver reports.

Our users-per-card rule takes 90 per cent of the memory the driver reports and keeps 3 GiB per card for CUDA graphs and activations: 0.9 × 140.4 − 3 = 123.36 GiB per card, 246.72 GiB for the pair. Whatever the weights leave is KV cache, the memory each conversation needs for its context. The derivation is in our guide to how many users one RTX PRO 6000 can serve, and the same rule sizes the 4× H200 NVL server and the 8× H200 NVL server.

Llama 3.3 70B in BF16 needs both cards

Many teams want a 70B model without quantisation, for example to evaluate a fine-tuned checkpoint exactly as it was trained. In BF16 every parameter takes two bytes, so Llama 3.3 70B with about 70.6 billion parameters needs about 141 GB, or 131.5 GiB, by our estimate. That is more than the 123.4 GiB one card offers by our rule, so the model has to be split. On a bridged pair it runs with tensor parallelism of two (TP=2), and 115.2 GiB remain for the cache.

In FP8 the same model shrinks to the 72.7 GB of NVIDIA's FP8 checkpoint, 67.7 GiB, and fits one card. The H200 NVL has FP8 Tensor Cores and no FP4 arithmetic, so FP8 is the usual reduced precision for this card; an NVFP4 checkpoint built for Blackwell runs on it only weight-only, as our guide to FP4 models on the H200 NVL explains.

Our estimates by the users-per-card rule. Weights from Hugging Face repositories as listed in our four-card H200 NVL guide (read on 24 October 2026); BF16 size of Llama 3.3 70B by our arithmetic at two bytes per parameter.

Speed per user follows the bytes each card reads per generated token. By our estimate, one card reading all 70.6 GB of the FP8 weights gives a single user at most about 68 tokens per second at 4.8 TB/s. Over the pair, each card reads half: about 35 GB in FP8, which lifts that ceiling towards 136 tokens per second, or 70.6 GB in BF16, which puts the BF16 model back at about 68. These ceilings come before the NVLink exchanges and fall as more users share each step. Qwen3-235B fits the pair but leaves little cache, and models of the 400B class need four cards, as our guide to how many H200 NVL cards large models need lists.

Users per layout: two copies or one copy over the bridge

A pair runs a model in two ways. Two copies, one per card, exchange nothing and need no bridge; one copy over both cards stores the weights once and leaves more memory for conversations.

Our estimates by the users-per-card rule, concurrent conversations at full context length. Cache per conversation for Llama 3.3 70B: 2.5 GiB (16-bit) or 1.25 GiB (FP8) at 8,192 tokens; gpt-oss-120b 1.125 GiB at 32K with a 16-bit cache; Qwen3-235B 2.94 GiB at 32K with an FP8 cache.

For the second row, the pair holds 246.7 GiB; after 67.7 GiB of weights, 179.0 GiB remain, enough for 143 conversations at 1.25 GiB each. Two copies leave 55.7 GiB per card, 44 conversations each, 88 together. The copies win on isolation, because a failed card stops only half the service, and the TP=2 copy wins on capacity and on speed per user.

These are requests in flight at the same moment, which the gateway shows at the busiest hour. A company of 500 staff with an assistant open in every browser produces a small fraction of that at once, so a pair often serves one or two departments with room left over. We build servers with two H200 NVL and the layout your model needs; send us the model, precision and peak requests in flight, and our team replies with a configuration and the conversations for each layout.

Long contexts: 32K and 128K conversations

NVIDIA's model card for its FP8 build of Llama 3.3 70B gives a context length of up to 128K tokens. Every token of context keeps its keys and values in the cache, 320 KiB per token in 16-bit or 160 KiB in FP8, from the 2.5 GiB and 1.25 GiB per 8,192 tokens our sizing rule uses for this model, so a 128K conversation of 131,072 tokens holds 40 GiB of 16-bit cache or 20 GiB in FP8.

Our estimates by the users-per-card rule, concurrent conversations at full length, FP8 cache unless stated; 1.25, 5 and 20 GiB per conversation with an FP8 cache, 2.5, 10 and 40 GiB with a 16-bit cache.

The table shows why long contexts favour one copy over the bridge. At 128K, a single card with FP8 weights has room for only two conversations, while the pair as one copy holds eight. For agents, document review or code assistants that keep large contexts, that is the difference between a pilot and a service for a small team. Prefill, the reading of a long prompt before the first token, is compute work that grows with the prompt; we found no published prefill time for a 128K prompt on this card, so measure it on your own documents. Our guide to a server for LLM agents with long contexts covers the layouts on four and eight cards.

Fine-tuning on two cards

Training needs more memory per parameter than serving. Mixed-precision Adam keeps 16 bytes per parameter before activations, by the ZeRO paper's count used in our guide to GPU memory for fine-tuning. Measured against the listed capacity of 141 GB per card, not the figure the driver reports, a pair holds the following.

Model states from our guide to GPU memory for fine-tuning (ZeRO count, rank-16 adapters on all linear layers, GB = 10⁹ bytes), divided over the cards by our arithmetic; activations come on top.

LoRA of a 70B model is the typical company project on this server. The 144.4 GB of model states split into 72.2 GB per card with FSDP, and the gradient traffic between the two cards runs over the bridge at 900 GB/s per GPU, where without it every exchange would cross PCIe. A full fine-tune of a 32B model exceeds the 282 GB of HBM3e: with ZeRO-Offload the FP32 optimiser states, 12 bytes per parameter or about 393 GB, move to host memory, which the 768 GB of DDR5 we fit holds by our arithmetic, at a longer step time. Larger full fine-tunes belong on eight cards, as our fine-tuning server with 2, 4 or 8 GPUs and the H200 NVL and RTX PRO 6000 fine-tuning comparison explain.

MIG on two cards: up to 14 instances

MIG divides a card into isolated GPU instances, each with its own memory and compute slices. NVIDIA's MIG user guide, updated 13 October 2026, lists for the H200 141 GB product up to seven 1g.18gb instances, four 1g.35gb, three 2g.35gb, two 3g.71gb, or one 4g.71gb or 7g.141gb. Two cards give up to 14 instances of 18 GB.

MIG is set per card, which makes mixed use of a pair practical. Card one serves Llama 3.3 70B in FP8 as a whole GPU, 44 conversations of 8K with an FP8 cache by our rule, while card two runs seven 1g.18gb slices for notebooks, development and small models. The TP=2 copy needs both cards whole, because NVIDIA states that NCCL, the library for the exchanges between GPUs, does not work with MIG. NVIDIA's reference architecture for the H200 NVL asks for seven cores per GPU, two more per GPU for the operating system and two per MIG instance: 18 cores for the pair without MIG and 46 with all 14 instances. For virtual machines the card offers compute vGPU (C-series) through NVIDIA AI Enterprise and no graphics vGPU, as our H200 NVL MIG profiles explain.

2U servers for two cards, and where 4U fits better

Two H200 NVL are the configuration that fits the most chassis. The makers' documents we read on 24 October 2026 list these servers.

Lenovo Press LP2128 (updated 20 October 2026); Dell PowerEdge R770 technical guide Rev. A09 (June 2026); Supermicro SYS-212GB-NR product page; HPE DL380a Gen12 QuickSpecs; Cisco UCS C845A M8 spec sheet Rev. A.11 (11 May 2026); read on 24 October 2026.

Dell's R770 guide states that the H200 NVL is "designed to support up to 600 W" but is "power capped to 450 W support on R770", in riser positions RC 6-2 and RC 11-2, with high-performance Platinum fans and slot-specific power cables; what that cap means is in our article on the H200 NVL at 450 W. Supermicro's 2U model documents a bridged pair but has one processor socket, one fewer than NVIDIA's certified-systems guide asks for. In the documents we read, a bridged pair at 600 W in a two-socket server therefore appears in the 4U servers from HPE and Cisco, which also leave room to add cards later. NVIDIA's brief recommends that bridged cards sit "within the same CPU domain", so both cards of the pair go under one processor. Our list of H200 NVL compatible servers has the details per maker.

Host memory, power and cooling

NVIDIA's configuration guide for NVIDIA-Certified Systems asks for system memory of at least twice the GPU memory, 564 GB for two cards. Two AMD EPYC 9005 processors have 24 memory channels, so we fit 24 modules of 32 GB, 768 GB, one per channel. NVIDIA's reference architecture sets a lower floor of 128 GB per GPU, and Cisco recommends 1,024 GB for two cards in its C845A M8, which suits CPU offload; our guide to how much RAM a GPU server needs compares the rules.

Our estimate: board power from NVIDIA's product brief, two AMD EPYC 9355 at 280 W each, plus 15 per cent for memory, drives, network cards, fans and conversion losses.

Both figures are within the 2.94 kW we plan per 230 V 16 A single-phase feed at 80 per cent, so redundant supplies on an A and a B feed of 16 A each carry the server. About 2.0 kW of heat needs roughly 320 CFM of air at an 11 °C rise, by the 160 CFM per kW in our guide to GPU rack power and cooling. The card is passive and needs the 16-pin cable the server maker lists for its slot. We check feed, airflow and chassis before we quote; tell us the feed type and the inlet temperature at the rack position, and we say which chassis fits.

When two cards are not the right size

Four cards on a 4-way bridge, 493.4 GiB by our rule, run models of the 400B class and more than twice the 70B users, as our guide to the 4× H200 NVL server shows. Where models are published in FP4 or graphics vGPU is needed, two RTX PRO 6000 Blackwell Server Edition cards compute NVFP4 and support graphics vGPU, with 166.1 GiB usable by our rule, as our RTX PRO 6000 vs H200 NVL comparison explains. Double-precision solvers fit the H200 pair well, at 30 TFLOPS of FP64 per card by NVIDIA's H200 page, as our guide to GPUs for CFD simulation shows.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight H200 NVL and their NVLink bridges, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, drivers, CUDA and a container runtime on request. Each card includes a five-year NVIDIA AI Enterprise subscription, according to NVIDIA's licensing guide. The configuration and ordering steps are on our page to buy a 2× H200 NVL server, and the cards and bridges alone for a server you already run are on our H200 NVL page.

FAQ