NVIDIA RTX PRO 6000 Blackwell without NVLink: TP=2, TP=4 or replicas on 2, 4 and 8 cards

Tensor parallelism on RTX PRO 6000 without NVLink

No edition of the NVIDIA RTX PRO 6000 Blackwell has NVLink, neither the Server Edition nor the Max-Q nor the Workstation Edition, so a model split across several cards runs its tensor-parallel traffic over PCIe 5.0, which works on 2, 4 and 8 cards. Run one copy per card whenever the model and its cache fit, use TP=2 pairs for models up to about 150 GiB of weights and TP=4 up to about 310 GiB, and on an eight-card server compare two TP=4 copies with one TP=8 instance.

The main software difference between TP=2 and TP=4 is the all-reduce: by default vLLM uses its own custom kernel for two PCIe-only GPUs and leaves four or eight to NCCL.

Weights limits by our sizing rule: 0.9 × 95.6 GiB the driver reports − 3 GiB = 83.0 GiB per card, 166.1, 332.2 and 664.3 GiB for 2, 4 and 8 cards, with at least 15 to 25 GiB per card left for KV cache. All-reduce paths from vLLM's custom_all_reduce source (latest docs) and the v0.32.0 source, read 24 October 2026.

Many long conversations need more cache than these margins, which can make a split pay even when the weights fit one card; our article on splitting one model over PCIe or NVLink works this through for Llama 3.3 70B.

What vLLM does at TP=2, TP=4 and TP=8 over PCIe

Tensor parallelism cuts every layer across the cards, and after each split layer the cards sum their partial results in an all-reduce. vLLM has two ways to do that on one node: its own custom all-reduce kernel, which writes directly into the other GPUs' memory over peer-to-peer (P2P), and NVIDIA's NCCL library.

The source of custom_all_reduce in vLLM's current documentation sets the rule. When the GPUs are on one node, the world size is above two and the GPUs are not fully connected, vLLM logs "Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs". The kernel also stays off when vLLM's P2P test fails and for world sizes outside 2, 4, 6, 8 and 16. The flag --disable-custom-all-reduce turns it off by hand; vLLM's help text says it will "fall back to NCCL".

vLLM custom_all_reduce source (latest documentation) and flashinfer_pcie_ipc_all_reduce.py at tag v0.32.0, read 24 October 2026, for one node with PCIe-only GPUs.

So TP=2 and TP=4 on the same server run different code. A TP=2 pair that hangs points at the custom kernel and the P2P path, a slow TP=4 instance at NCCL over PCIe. The pull request for vLLM's new backend describes the cost: on PCIe-only multi-GPU boxes, "the default NCCL RING_LL all-reduce dominates small-message decode collectives", the many small all-reduces of token generation.

The opt-in FlashInfer PCIe IPC all-reduce in vLLM 0.30.0

The release notes of vLLM 0.30.0, published on 22 September 2026, list "an opt-in FlashInfer PCIe IPC all-reduce for NVLink-less boxes" (pull request 53576), and the module is still in 0.32.0 of 18 October 2026. Its docstring says it "targets small tensor-parallel all-reduces on a single PCIe-only node". It is off by default: VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC is a boolean set to False, and the pull request switches it on with VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1.

The code sets the limits: world sizes 2, 4 and 8, tensors in BF16 or FP16, every rank on one node, CUDA Graph capture sizes required, and no DBO or multi-ubatch execution yet. If the installed FlashInfer build lacks PcieIpcAllReduceWorkspace, vLLM logs a warning and falls back to another backend, so check the log after switching it on.

Little is published about the gain.

• FlashInfer's pull request for the kernel reports a microbenchmark on eight L40S in BF16: "All 72 points beat NCCL, from 1.22× to 5.92×", over TP 2, 4 and 8 and batch 1 to 128.
• A full-model run posted there, on two RTX PRO 6000 Max-Q at 300 W on PCIe Gen4 with TP=2, cut the step time by 4.57 per cent at one concurrent request and 5.59 per cent at eight, with "no prefill benefit".
• vLLM's pull request names an end-to-end test on four RTX PRO 6000 at TP=4 with DeepSeek-V4-Flash-0731 and reports no speed figures.

We found no published end-to-end gain at TP=4 or TP=8 on RTX PRO 6000 cards. vLLM's parallelism guide, updated on 15 October 2026, still advises for GPUs without NVLink to "leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead", and does not mention the new backend. Treat the opt-in as a candidate for an A/B test with your model, with NCCL as the baseline.

We build servers with 2, 4 or 8 RTX PRO 6000 Server Edition cards and lay out the slots for the split you plan; send us the model, its precision and the context length and our team replies with a layout and a configuration.

Published P2P and tensor-parallel measurements

Level1Techs threads "RTX Pro 6000 - Everthing That's Gone Wrong" and a four-card Max-Q build log; StorageReview HPE ProLiant DL380a Gen12 review (6 November 2025); Helix engineering blog (14 August 2026); Google Cloud blog on G4 VMs (20 October 2025). Figures as published, read 24 October 2026; forum figures come from single systems.

Three things follow. P2P matters more for latency than for bandwidth: with P2P on, the latency between two cards fell from 12.68 to 0.53 µs, while one-way bandwidth rose only from about 42 to 46 GB/s, below the 64 GB/s that PCIe 5.0 x16 carries each way. Tensor parallelism over PCIe runs at usable speeds, with Llama 2 70B at about 33 tokens per second for one user on four Server Edition cards. And replicas add throughput where one split instance runs out: Helix doubled throughput at 32 concurrent requests with two TP=4 instances behind a sticky load balancer. We found no published side-by-side of TP=4 and TP=8 on the same RTX PRO 6000 server.

Workstation boards, dual-socket servers and PCIe switches

The slot wiring decides which pairs talk fast. nvidia-smi topo -m prints the path for every pair: PIX for one PCIe bridge, PXB for several, PHB through a host bridge, NODE between host bridges of one NUMA node, SYS across the link between the processors.

• Workstations. Each card sits on a root port of one processor; two Max-Q cards on a Threadripper PRO board showed NODE in the Level1Techs thread. NVIDIA rates the Max-Q for "up to four GPUs in a single system".
• Dual-socket servers without switches. The cards split across two NUMA nodes. In a vLLM forum report of April 2026, eight Server Edition cards sat four per node, and a TP=2 instance hung at start-up until --disable-custom-all-reduce was set. Keep each TP set on one socket.
• Servers with PCIe switches. Dell's XE7745 technical guide (September 2026) attaches each GPU slot to one of four Broadcom Atlas II switches. Supermicro's SYS-422GA-NRT datasheet lists a "PCIe 5.0 x16 Switch Dual-Root" for up to eight 600 W cards at 35 °C, with the Server Edition, the Max-Q and the H200 NVL on its GPU list; it states the eight for the Server Edition and gives no count for the Max-Q. NVIDIA's certified-systems guide (30 September 2026) recommends Gen5 switches and calls a switch "optional for cost-effective inference servers".

On bare metal, NVIDIA's NCCL documentation names two settings that break the P2P path: "CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer", and "If PCI switches have ACS enabled, it needs to be disabled", because IO virtualisation can redirect peer traffic to the CPU root complex, "causing a significant performance reduction or even a hang". It adds that "Virtual machines require ACS to function", so a model split inside VMs gives up part of this path.

Layouts per model size on 2, 4 and 8 cards

Weights from the Hugging Face repositories of openai and Qwen read 24 October 2026 (65.3 GB, 236 GB, 470 GB), deepseek-ai (167 GB repository for DeepSeek-V4-Flash-0731; Helix gives about 156 GB as served) and our sizing facts for Llama 3.3 70B (conversations of 8,192 tokens, FP8 cache). Cache room by our rule, assuming an even split of the weights.

A model that fits one card is cheapest to run as replicas: nothing crosses PCIe, a failed card takes down one copy, and throughput adds card by card. Replicas need session affinity, so that a returning conversation reaches the copy that holds its cache. Split when the weights do not fit, when one user needs the speed of several cards, or when the cache does not fit beside a copy: a TP=2 pair holds 78 conversations of Llama 3.3 70B against 24 for two copies, by our rule.

Qwen's model card serves its 235B FP8 model with TP=4. Its config.json gives four KV heads, so TP=4 puts one head on each card and the cache splits evenly. At TP=8, vLLM's context-parallel documentation says the cache "will be duplicated for tp_size / H times", here twice, and vLLM's Qwen3 usage guide advises TP=4, or TP=8 with --enable-expert-parallel, for FP8 MoE checkpoints whose expert slices would not divide by the 128-wide quantisation blocks. Our article on a server for AI agents with 100K context sizes this model for long conversations, and our 8 × RTX PRO 6000 allocation guide splits eight cards among several models.

TP=2 hangs: what vLLM issue 30682 and the forums report

vLLM issue 30682 was opened in December 2025 for two RTX PRO 6000 Max-Q at TP=2 on vLLM 0.12.0 and driver 580.95.05. The log repeats "No available shared memory broadcast block found in 60 seconds" and ends in an RPC timeout. The reporter closed it on 19 December 2025 after adding iommu=pt to the kernel options; users kept reporting the hang into June 2026, and no maintainer posted a fix. The workarounds in its 16 comments:

• --disable-custom-all-reduce, alone or with NCCL_P2P_DISABLE=1: worked for some users, not for others;
• iommu=pt in the GRUB options: fixed it for the reporter, not for another user on different GPUs;
• a newer NCCL wheel, suggested without a result, and more shared memory for Docker or Podman, which "helped a lot" for one user.

The Level1Techs thread on two Max-Q cards shows the same pattern: with P2P on, NCCL hung on the first all-reduce right after "Connected all rings". One user fixed it by disabling IOMMU in the BIOS, another with amd_iommu=on iommu=pt and ACS off; with P2P disabled, the two-card all-reduce reached about 31 GB/s of bus bandwidth. A four-card Max-Q build log fixed a TP=2 stall by disabling ACS and IOMMU in the BIOS.

A sequence that follows these sources: check the topology, run NVIDIA's simpleP2P sample until it ends with "Test passed", run all_reduce_perf from nccl-tests on each TP set, check the IOMMU mode in /proc/cmdline and ACS on the switches, then try --disable-custom-all-reduce in vLLM, and keep NCCL_P2P_DISABLE=1 for last. IOMMU and ACS change how devices are isolated, so agree them with whoever owns the host's security.

What to check before ordering a multi-card server

1. The edition. In a rack server, the RTX PRO 6000 Server Edition, which NVIDIA designs "for multi-GPU server deployments requiring passive cooling". The Max-Q belongs in a rack only where the maker lists it, as Lenovo's guide LP2364 does for two ThinkSystem servers and Supermicro does in the GPU list of the SYS-422GA-NRT. Supermicro gives no Max-Q count for that chassis, so we have the maker confirm the count for the exact configuration before we quote.
2. The topology for your slot population: which pairs share a switch, which cross a socket, and nvidia-smi topo -m from a populated unit.
3. The BIOS options for ACS and IOMMU, and whether the host will run virtual machines.
4. P2P and all-reduce results at acceptance, as in our guide to burn-in and acceptance testing.
5. The versions to pin: vLLM, NCCL, driver, and a FlashInfer build if you test the opt-in backend.
6. Growth from two cards to four, or four to eight, as in our guide to adding GPUs later.

We supply the RTX PRO 6000 in all three editions and build the server around the layout, assembled and tested before delivery; describe the model and the number of users, and we answer with the card count and the slot plan.

When NVLink is worth it

The H200 NVL joins two or four cards with NVLink bridges at 900 GB/s per GPU, against 128 GB/s for PCIe Gen5 on NVIDIA's H200 page, so TP=4 inside a four-way domain keeps its all-reduces off PCIe. An eight-card H200 NVL server holds two such domains, as our guide to the H200 NVL NVLink bridge explains, and our comparison of RTX PRO 6000 and H200 NVL for LLM inference weighs 96 against 141 GB per card. TP=8 at NVLink speed across eight GPUs needs a DGX B300, which we quote per project. For most models up to about 300 GiB of weights, PCIe and a sound layout on RTX PRO 6000 Server Edition cards carry the load.

FAQ