One model on several GPUs: tensor, pipeline and expert parallelism over PCIe and NVLink bridges

Three ways to split one model

A model that does not fit one GPU, or has to answer faster than one GPU can, gets split across several. The three ways of splitting it send different traffic between the cards, which is why the link matters more for some than for others.

Tensor parallelism cuts each layer’s weight matrices across the GPUs. NVIDIA’s Megatron-LM paper splits the first matrix of each MLP block by columns and the second by rows, and the attention by heads, so that a transformer layer needs “only two all-reduces in the forward path”. Every token waits for both, on every layer.

Pipeline parallelism gives each GPU a consecutive range of layers, and only the activations at a stage boundary move, point to point. The later Megatron-LM paper of 2021 sums up the trade: “Pipeline model parallelism features cheaper point-to-point communication. Tensor model parallelism, on the other hand, uses all-reduce communication.” The cost is idle time, the pipeline bubble: in training, the paper puts it at the number of stages minus one, divided by the number of micro-batches, as a fraction of the ideal compute time, so it stays small only with many more micro-batches than stages. In serving, a stage has work only when enough requests are in flight to fill the pipeline.

Expert parallelism applies to mixture-of-experts models. Whole experts sit on different GPUs, and each token travels to the GPUs that hold its experts and back, an all-to-all exchange that the DeepSeek-V3 technical report calls dispatching and combining. vLLM switches it on with --enable-expert-parallel and sets the expert-parallel size to the tensor-parallel size times the data-parallel size.

Megatron-LM papers (NVIDIA, 2019; NVIDIA and others, 2021), DeepSeek-V3 technical report, vLLM expert-parallel and data-parallel documentation. Training with data parallelism all-reduces gradients at every step, and vLLM’s data parallelism for MoE models synchronises the expert layers every forward pass.

The link: 128 GB/s or 900 GB/s

The RTX PRO 6000 has no NVLink on any edition, so its cards exchange data over PCIe 5.0 x16. NVIDIA’s Hopper architecture description credits a PCIe Gen 5 x16 interface with “128 GB/sec total bandwidth (64 GB/sec in each direction)”. The H200 NVL adds bridges that join two or four cards at 900 GB/s per GPU, and NVIDIA’s H200 page prints that beside PCIe Gen5’s 128 GB/s, about seven times more. NVIDIA’s H200 NVL product brief calls the 900 GB/s “bidirectional”, and the Hopper description counts it as 18 NVLink links at 25 GB/s each way, so 450 GB/s in each direction: the same basis as the PCIe figure. The brief’s “14×” compares it with one direction of PCIe. The bridge domain stops at four cards, so an eight-card H200 NVL server holds, by our reading, two NVLink domains joined over PCIe.

The path between two cards counts too. nvidia-smi topo -m prints it for every pair: through one PCIe switch (PIX), several switches (PXB), a host bridge (PHB), between host bridges of one NUMA node (NODE), or across the link between processors (SYS). Akamai’s documentation for its RTX PRO 6000 Server Edition instances warns that peer-to-peer copies between GPUs on different root complexes “may experience reduced bandwidth”.

What tensor parallelism sends: a 70B example

Take Llama 3.3 70B. Its config.json gives a hidden size of 8,192 and 80 layers. Our assumptions: activations in BF16; two all-reduces per layer, as in Megatron-LM; a ring all-reduce, in which each GPU sends the equivalent of the whole buffer with two GPUs, 1.5 times the buffer with four and 1.75 times with eight, the factors NCCL’s test documentation uses; and no allowance for latency, protocol overhead or overlap with computation.

One token’s activation vector is 8,192 × 2 bytes, 16 KiB. Two all-reduces on each of 80 layers make 160 per forward pass, which carry 2.5 MiB of data per token.

Our arithmetic from the model’s config.json, the Megatron-LM paper and NCCL’s test documentation. 64 GB/s is NVIDIA’s per-direction figure for PCIe 5.0 x16; real transfers run below it.

Two readings follow. For one user generating text, a few MiB per token is nothing for the link; what costs time is 160 synchronisation points per token, “with every one of those exchanges blocking the next computation”, as StorageReview put it. For long prompts and large batches the volume itself becomes the cost: 82 to 143 milliseconds of transfer for a 2,000-token prompt over PCIe; for two or four H200 NVL cards on their bridge, at best about a seventh of the 82 and 123 milliseconds, by NVIDIA’s figures: 450 GB/s per GPU each way against 64 GB/s.

Peer-to-peer over PCIe: IOMMU and ACS

Over PCIe, these collectives are fastest when GPUs write into each other’s memory directly. NVIDIA’s NCCL documentation says the library “heavily relies on GPU Direct”, meaning “direct point-to-point PCI messages”, and names two settings that break or slow it. On bare-metal Linux, “CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer”, so translated IOMMU mode should be avoided for the GPU path. And in its section on PCI Access Control Services it warns that IO virtualisation can interfere by “redirecting all PCI point-to-point traffic to the CPU root complex, causing a significant performance reduction or even a hang”; if PCI switches have ACS enabled, “it needs to be disabled”. The same page adds: “Virtual machines require ACS to function, hence disabling ACS is not an option.” A virtualised host trades that path for isolation.

Check before blaming the cards. SGLang’s documentation tells users who see “peer access is not supported between these two devices” to add --enable-p2p-check, and NVIDIA’s nccl-tests suite measures the all-reduce rate a machine actually reaches with all_reduce_perf. Google reported in October 2025 what the path is worth: “up to 168% throughput gains and 41% lower latency (inter-token latency)” for tensor-parallel serving on its RTX PRO 6000 Server Edition machines with an enhanced PCIe peer-to-peer path, “compared to standard non-P2P offerings”.

What vLLM and SGLang recommend

vLLM’s guidance is short: “if the model fits on a single GPU, distributed inference is probably unnecessary”; if it is too large for one GPU but fits one node, use tensor parallelism; across nodes, set the tensor-parallel size to the GPUs per node and the pipeline-parallel size to the number of nodes. In its section on uneven splits it adds a note that applies to every PCIe-only server: “if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead”. SGLang’s documentation states that “Data parallelism is better for throughput if there is enough memory” and that it can be used together with tensor parallelism.

What the published measurements show

StorageReview: HPE ProLiant DL380a Gen12 review, 6 November 2025, with vLLM; DGX Spark cluster review, 11 May 2026, OEM GB10 systems paired over one 200 Gb link, equal input and output lengths, engine not stated. SE: Server Edition. Figures as the reviews report them.

The PCIe server runs a 70B model split four ways without NVLink at about 33 tokens per second for one user. The two OEM DGX Spark systems, whose 200 Gb link carries 25 GB/s each way against 64 GB/s for one PCIe 5.0 x16 slot, show both sides of the trade-off: pipeline parallelism more than doubles throughput at batch 128, while tensor parallelism is 37 per cent faster for a single user, as it splits each token’s weight reads between the two units. At 65.3 GB in its original MXFP4 release, the model fits one 128 GB unit, so the test shows the trade-off rather than a reason to split it. Our two-unit DGX Spark article has the rest of that test. We have not found a published side-by-side of tensor and pipeline parallelism on RTX PRO 6000 servers.

When one copy per GPU beats splitting

Replicas, one full copy of the model per card behind a load balancer, send nothing between the cards, fail independently and add throughput card by card. Splitting pays in three cases: the model does not fit one card; one user needs more speed than one card delivers, since tensor parallelism shares out the weight reads of each token; or many long conversations need more cache than one copy per card leaves.

The third case is easy to miss. Take two RTX PRO 6000 and Llama 3.3 70B in FP8, 68 GiB of weights, with an FP8 cache of 1.25 GiB per 8,192-token conversation, and a planning rule of 90 per cent of the 95.6 GiB the driver reports. By our arithmetic, one card leaves 18 GiB of cache, about 14 conversations, so two replicas serve 28. Split with TP=2, each card holds 34 GiB of weights and 52 GiB of cache: about 83 conversations for one instance, paid for with 160 all-reduces per forward pass. Models such as Qwen3-235B-A22B, 134 GB in NVFP4 and 236 GB in FP8, leave no choice: two or four cards, as our four-card workstation article lists.

A layout per platform

RTX PRO 6000. Replicate whatever fits one card. Above that, tensor parallelism over PCIe works best where peer-to-peer works; test pipeline parallelism against it, as vLLM suggests for GPUs without NVLink, and keep the cards of one model on the shortest path nvidia-smi topo -m shows. For mixture-of-experts models, expert parallelism is a third option, with its all-to-all traffic on the same PCIe.

H200 NVL. Bridge the cards that share a model: tensor parallelism inside a two- or four-card NVLink domain, pipeline parallelism between the two domains of an eight-card server, which is vLLM’s rule for nodes applied to domains. Card counts per model are in our H200 NVL guide.

Beyond both. A model that needs tensor parallelism across eight GPUs at interactive latency, or training that exchanges gradients at every step, belongs on an NVLink system such as a DGX B300, as our B300 comparison sets out.

What we supply

Eurokommerz supplies the RTX PRO 6000 Workstation Edition and Max-Q, and the H200 NVL with its two-way and four-way NVLink bridges, EU-wide with manufacturer warranty, as cards or in workstations and AI servers built to order with the slot layout a split model needs. Send us the model and the number of users, and we will tell you whether it needs one card, replicas or a split.

FAQ