One model on several GPUs: tensor, pipeline and expert parallelism over PCIe and NVLink bridges

Three ways to split one model

A model that does not fit one GPU, or has to answer faster than one GPU can, gets split across several. The three ways of splitting it send different traffic between the cards, which is why the link matters more for some than for others.

Tensor parallelism cuts each layer’s weight matrices across the GPUs. NVIDIA’s Megatron-LM paper splits the first matrix of each MLP block by columns and the second by rows, and the attention by heads, so that a transformer layer needs “only two all-reduces in the forward path”. An all-reduce leaves every GPU with the sum of all GPUs’ partial results, and each token has to wait for both all-reduces on every layer.

Pipeline parallelism gives each GPU a consecutive range of layers, and only the activations at a stage boundary move, point to point. The later Megatron-LM paper of 2021 sums up the trade: “Pipeline model parallelism features cheaper point-to-point communication. Tensor model parallelism, on the other hand, uses all-reduce communication.” The cost is idle time, the pipeline bubble: in training, the paper puts it at the number of stages minus one, divided by the number of micro-batches, as a fraction of the ideal compute time, so it stays small only with many more micro-batches than stages. In serving, a stage has work only when enough requests are in flight to fill the pipeline.

Expert parallelism applies to mixture-of-experts models. Whole experts sit on different GPUs, and each token travels to the GPUs that hold its experts and back, an all-to-all exchange that the DeepSeek-V3 technical report calls dispatching and combining. vLLM switches it on with --enable-expert-parallel and sets the expert-parallel size to the tensor-parallel size times the data-parallel size.

Megatron-LM papers (NVIDIA, 2019; NVIDIA and others, 2021), DeepSeek-V3 technical report, vLLM expert-parallel and data-parallel documentation. Training with data parallelism all-reduces gradients at every step, and vLLM’s data parallelism for MoE models synchronises the expert layers every forward pass.

The link: 128 GB/s or 900 GB/s

The RTX PRO 6000 has no NVLink on any edition, so its cards exchange data over PCIe 5.0 x16. NVIDIA’s Hopper architecture description credits a PCIe Gen 5 x16 interface with “128 GB/sec total bandwidth (64 GB/sec in each direction)”. The H200 NVL adds bridges that join two or four cards at 900 GB/s per GPU, and NVIDIA’s H200 page prints that beside PCIe Gen5’s 128 GB/s, about seven times more. NVIDIA’s H200 NVL product brief calls the 900 GB/s “bidirectional”, and the Hopper description counts it as 18 NVLink links at 25 GB/s each way, so 450 GB/s in each direction: the same basis as the PCIe figure. The brief’s “14×” compares it with one direction of PCIe. The bridge domain stops at four cards, so an eight-card H200 NVL server holds, by our reading, two NVLink domains joined over PCIe.

The path between two cards counts too. nvidia-smi topo -m prints it for every pair: through one PCIe switch (PIX), several switches (PXB), a host bridge (PHB), between host bridges of one NUMA node (NODE), or across the link between processors (SYS). Akamai’s documentation for its RTX PRO 6000 Server Edition instances warns that peer-to-peer copies between GPUs on different root complexes “may experience reduced bandwidth”.

What tensor parallelism sends: a 70B example

Take Llama 3.3 70B. Its config.json gives a hidden size of 8,192 and 80 layers. We assume activations in BF16, two all-reduces per layer as in Megatron-LM, and a ring all-reduce, in which each GPU sends the equivalent of the whole buffer with two GPUs, 1.5 times the buffer with four and 1.75 times with eight, the factors NCCL’s test documentation uses. We make no allowance for latency, protocol overhead or overlap with computation.

One token’s activation vector is 8,192 × 2 bytes, 16 KiB. Two all-reduces on each of 80 layers make 160 per forward pass, which carry 2.5 MiB of data per token.

Our arithmetic from the model’s config.json, the Megatron-LM paper and NCCL’s test documentation. 64 GB/s is NVIDIA’s per-direction figure for PCIe 5.0 x16; real transfers run below it.

These figures have two consequences. For one user generating text, a few MiB per token is a small load for the link; the time goes into 160 synchronisation points per token, “with every one of those exchanges blocking the next computation”, as StorageReview put it. For long prompts and large batches the volume itself costs time, 82 to 143 milliseconds of transfer for a 2,000-token prompt over PCIe. On two or four H200 NVL cards joined by their bridge, the transfer takes at best about a seventh of the 82 and 123 milliseconds, by NVIDIA’s figures of 450 GB/s per GPU each way against 64 GB/s. We supply the RTX PRO 6000 and the H200 NVL with its NVLink bridges and build AI servers to order for both. Tell us your model and how many users it serves, and our team will help you choose the split.

Peer-to-peer over PCIe: IOMMU and ACS

Over PCIe, these collectives are fastest when GPUs write into each other’s memory directly. NVIDIA’s NCCL documentation says the library “heavily relies on GPU Direct”, meaning “direct point-to-point PCI messages”, and names two settings that break or slow it. On bare-metal Linux, “CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer”, so translated IOMMU mode should be avoided for the GPU path. And in its section on PCI Access Control Services it warns that IO virtualisation can interfere by “redirecting all PCI point-to-point traffic to the CPU root complex, causing a significant performance reduction or even a hang”; if PCI switches have ACS enabled, “it needs to be disabled”. The same page adds: “Virtual machines require ACS to function, hence disabling ACS is not an option.” On a virtualisation host, ACS therefore stays on, and peer traffic between GPUs passed to virtual machines takes the route through the CPU root complex.

Test the peer-to-peer path before drawing conclusions about the cards. SGLang’s documentation tells users who see “peer access is not supported between these two devices” to add --enable-p2p-check, and NVIDIA’s nccl-tests suite measures the all-reduce rate a machine actually reaches with all_reduce_perf.

What vLLM and SGLang recommend

vLLM’s guidance is short: “if the model fits on a single GPU, distributed inference is probably unnecessary”; if it is too large for one GPU but fits one node, use tensor parallelism; across nodes, set the tensor-parallel size to the GPUs per node and the pipeline-parallel size to the number of nodes. In its section on uneven splits it adds a note that applies to every PCIe-only server: “if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead”. SGLang’s documentation states that “Data parallelism is better for throughput if there is enough memory” and that it can be used together with tensor parallelism. The 2021 Megatron-LM paper gives the same rule for training: tensor model parallelism “should generally be used up to degree g when using g-GPU servers”, and pipeline parallelism across servers; each of its servers held eight A100 GPUs joined by NVLink and NVSwitch.

What the published measurements show

StorageReview: HPE ProLiant DL380a Gen12 review, 6 November 2025, with vLLM, no precision stated for Llama 2 70B; DGX Spark cluster review, 11 May 2026, OEM GB10 systems paired over one 200 Gb link, equal input and output lengths, engine not stated. SE: Server Edition. Figures as the reviews report them.

The PCIe server runs a 70B model split four ways without NVLink at about 33 tokens per second for one user. The two OEM DGX Spark systems, whose 200 Gb link carries 25 GB/s each way against 64 GB/s for one PCIe 5.0 x16 slot, show both sides of the trade-off: pipeline parallelism more than doubles throughput at batch 128, while tensor parallelism is 37 per cent faster for a single user, as it splits each token’s weight reads between the two systems. At 65.3 GB in its original MXFP4 release, the model fits one 128 GB system, so the test shows the trade-off rather than a reason to split it. Our article on two DGX Spark nodes has the rest of that test. For one user at TP=2, each of the 160 all-reduces per token moves about 16 KiB per GPU by our arithmetic, about 0.26 µs at the 64 GB/s of PCIe 5.0 x16, so for token generation the delay of each exchange matters more than the bandwidth, and working P2P shortens that delay by skipping the copy through host memory. We found no published side-by-side of tensor and pipeline parallelism on RTX PRO 6000 servers.

When one copy per GPU beats splitting

Replicas, one full copy of the model per card behind a load balancer, send nothing between the cards, fail independently and add throughput card by card. Splitting pays in three cases: the model does not fit one card; one user needs more speed than one card delivers, since tensor parallelism shares out the weight reads of each token; or many long conversations need more cache than one copy per card leaves.

The third case is easy to miss. Take two RTX PRO 6000 and Llama 3.3 70B in FP8, with 68 GiB of weights and an FP8 cache of 1.25 GiB per 8,192-token conversation. As a planning rule we use 90 per cent of the 95.6 GiB the driver reports, less 3 GiB per card for activations and CUDA graphs. By our arithmetic, one card leaves 15 GiB of cache, 12 conversations, so two replicas serve 24. Split with TP=2, each card holds 34 GiB of weights and 49 GiB of cache: about 78 conversations for one instance, paid for with 160 all-reduces per forward pass. Models such as Qwen3-235B-A22B, 134 GB in NVFP4 and 236 GB in FP8 (the 2507 release), need two or four cards in any case, as our four-card workstation article lists.

For a specific model, we work out both layouts, replicas or a split, and the number of cards each needs; we supply the cards for either. Write to us with the model and how many long conversations it must hold at once.

A layout per platform

RTX PRO 6000. Replicate whatever fits one card. Above that, tensor parallelism over PCIe works best where peer-to-peer works; test pipeline parallelism against it, as vLLM suggests for GPUs without NVLink, and keep the cards of one model on the shortest path nvidia-smi topo -m shows. Two cards and four behave differently in vLLM: its own all-reduce kernel runs on two PCIe cards with working peer-to-peer, but by default its source code switches it off for more than two cards without NVLink (“not supported on more than two PCIe-only GPUs”), so TP=4 leaves every all-reduce to NCCL unless the opt-in backend described below is switched on. On a server with four RTX PRO 6000 Server Edition cards, a model that fits two cards is therefore worth testing as two TP=2 instances, each pair on one processor or PCIe switch where possible, against one TP=4 copy; no published measurement compares the two, so a load test on your model decides. For mixture-of-experts models, expert parallelism is a third option, with its all-to-all traffic on the same PCIe. We build AI servers to order, including a configuration with four RTX PRO 6000 Server Edition cards. Send us the model and the split you plan, and we reply with a configuration and a quote within one business day.

vLLM 0.30.0, released on 22 September 2026, added an opt-in FlashInfer PCIe IPC all-reduce for servers without NVLink (pull request 53576), and it is still in 0.33.0 of 2 November 2026. It is off by default and is switched on with VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1. The code supports tensor-parallel sizes 2, 4 and 8 with BF16 or FP16 tensors on one node, needs CUDA Graph capture sizes and a FlashInfer build that provides PcieIpcAllReduceWorkspace, and otherwise falls back with a warning in the log. Little is published about the gain. FlashInfer’s pull request for the kernel reports a microbenchmark on eight L40S in BF16 in which it beat NCCL at all 72 points, by 1.22 to 5.92 times; a full-model run posted there, on two RTX PRO 6000 Max-Q at TP=2, cut the step time by 4.57 per cent at one concurrent request and 5.59 per cent at eight, with no prefill benefit. vLLM’s parallelism guide, updated on 15 October 2026, still suggests pipeline parallelism on GPUs without NVLink and does not mention the backend, so treat it as an A/B test against NCCL on your own model. Our comparison of TP=2, TP=4 and replicas without NVLink covers its limits in detail.

H200 NVL. Bridge the cards that share a model: tensor parallelism inside a two- or four-card NVLink domain, pipeline parallelism between the two domains of an eight-card server, which is vLLM’s rule for nodes applied to domains. Card counts per model are in our H200 NVL guide, and the choice between the two-way and four-way bridge in our NVLink bridge article.

Beyond both. A model that needs tensor parallelism across eight GPUs at interactive latency, or training that exchanges gradients at every step, belongs on an NVLink system such as a DGX B300, as our B300 comparison sets out.

What we supply

We supply the RTX PRO 6000 in all three editions (Workstation, Max-Q and Server Edition) and the H200 NVL with its two-way and four-way NVLink bridges, across the EU with manufacturer warranty, as cards or in workstations and AI servers built to order with the slot layout a split model needs.

FAQ