NVIDIA RTX PRO 6000 Blackwell multi-GPU P2P on Linux: topology, IOMMU, ACS, NCCL tests

What P2P gives on RTX PRO 6000 cards

Peer-to-peer (P2P) lets one GPU read from and write to another GPU's memory directly over PCIe, without a copy through host memory. None of the three NVIDIA RTX PRO 6000 Blackwell editions, the Server Edition, the Max-Q Workstation Edition and the Workstation Edition, has NVLink, so P2P over PCIe 5.0 is the fastest path between two cards. Check it with nvidia-smi topo -m, NVIDIA's simpleP2P sample and nccl-tests, and if it fails or hangs, look at the IOMMU mode and at ACS before anything else.

Tensor parallelism uses that path for every layer. vLLM's custom all-reduce writes directly into the other cards' memory and needs working P2P, and NCCL uses P2P as one of its transports. In the published test on two Workstation Edition cards below, P2P cut latency from 12.68 to 0.53 µs, while one-way bandwidth rose only from 42.18 to 45.85 GB/s.

Level1Techs forum, "RTX Pro 6000 - Everthing That's Gone Wrong" (17 July 2025), p2pBandwidthLatencyTest on two RTX PRO 6000 Workstation Edition; Level1Techs forum, "Dual RTX PRO 6000 Blackwell Max-Q: how to make P2P NCCL work?" (December 2025 to January 2026), driver 580.95.05, NCCL 2.28.9, 64 MiB messages. Single systems, figures as published, read 24 October 2026.

One-way bandwidth rises by less than 10 per cent with P2P, and both figures stay below the 64 GB/s that PCIe 5.0 x16 carries each way, half of the 128 GB/s that NVIDIA's H200 page gives for PCIe Gen5 in total. Latency falls by a factor of about 24. Token generation sends many small all-reduces, one after every split layer, so the latency is what a tensor-parallel pair feels. Our comparison of RTX PRO 6000 layouts without NVLink explains when a split pays and when replicas need no P2P at all.

Reading nvidia-smi topo -m

nvidia-smi topo -m prints a matrix with the path between every pair of GPUs and the CPU and NUMA affinity of each card. The legend it prints defines the path types.

Legend printed by nvidia-smi topo -m; expectations are our summary, with the published examples in the text below.

Published systems show the common cases. Two Max-Q cards on a Threadripper PRO workstation board showed NODE in the Level1Techs thread above, because each card sits on its own root port of one processor. In a vLLM forum report of April 2026, eight RTX PRO 6000 Blackwell Server Edition cards sat four on each NUMA node of a dual-socket server, so every pair across the sockets showed SYS. Servers with PCIe switches show PIX or PXB for cards under the same switch.

The matrix shows wiring, not whether P2P works. NVIDIA's NCCL troubleshooting guide adds nvidia-smi topo -p2p p, which prints a P2P status matrix for PCIe, with "OK" for each working pair on a healthy system. The same guide warns that "OK" alone does not rule out IOMMU or ACS problems, so the tests further down still have to run. Keep each tensor-parallel set on pairs that show PIX, PXB or NODE, and avoid SYS for a TP=2 pair.

BIOS settings: IOMMU, ACS, Above 4G and Resizable BAR

Four firmware settings decide whether P2P works on a multi-GPU host. Their names differ between board makers, so look for the function rather than the label.

• IOMMU (AMD IOMMU, Intel VT-d). In translated mode the IOMMU remaps device addresses. NCCL's guide states that on bare-metal Linux "CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer" and advises avoiding translated mode for GPU P2P; it also warns of silent corruption of device memory.
• ACS (Access Control Services). With ACS on, PCIe bridges and switches send peer traffic up to the processor's root complex. NCCL's guide says this can cause "a significant performance reduction or even a hang", and that "If PCI switches have ACS enabled, it needs to be disabled". On some systems turning off IO virtualisation or VT-d in the BIOS also disables ACS.
• Above 4G decoding and Resizable BAR. The README of a community P2P patch for NVIDIA's open kernel modules lists both as firmware prerequisites, because each GPU needs "a BAR1 aperture large enough to map its usable VRAM". Check both on workstation and server boards alike.

The Level1Techs threads show how this plays out. On the two-card Max-Q workstation, one user fixed the NCCL hang by disabling the IOMMU in the BIOS, with ACS already off, and another with amd_iommu=on iommu=pt on the kernel command line and ACS off. A four-card Max-Q build log, posted in July 2026, fixed a TP=2 stall by disabling ACS and the IOMMU in the BIOS.

We build multi-card systems with the RTX PRO 6000 Server Edition, Max-Q or Workstation Edition and assemble and test them before delivery; send us the card count and whether the host will run virtual machines, and our team replies with a slot plan and the firmware settings to agree.

Kernel parameters and their security cost

Start by reading the current state. NCCL's guide checks the IOMMU mode with cat /proc/cmdline and with dmesg filtered for iommu, dmar and default domain. Lines such as "iommu: Default domain type: Translated" or "DMAR: IOMMU enabled" are the warning signs it names.

Three kernel settings appear in the reports and documents.

Linux kernel documentation, admin-guide kernel-parameters (intel_iommu "on"/"off": "Enable/Disable intel iommu driver"; amd_iommu "off": "do not initialize any AMD IOMMU found in the system"); vLLM issue 30682 and Level1Techs threads for iommu=pt. Read 24 October 2026.

The kernel documentation lists no on value for amd_iommu; the amd_iommu=on in the forum fix has no documented effect, and iommu=pt is the part that changes the mode. ACS can also be switched off from Linux. NCCL's guide finds bridges with ACS through sudo lspci -vvv | grep ACSCtl, where "SrcValid+" suggests ACS is on, and clears it per bridge with setpci -s 03:00.0 ECAP_ACS+0x6.w=0000, the bus ID replaced by each bridge's own. On Broadcom PLX switches it notes that this has to be repeated after every reboot, so it belongs in a boot script.

Each of these settings removes a protection. The IOMMU limits which memory a device may reach by DMA, and ACS makes peer traffic pass a check at the root complex. The patch README states it plainly: "IOMMU passthrough (iommu=pt) removes DMA isolation. Do not use it with untrusted devices." NCCL's guide adds that "Virtual machines require ACS to function, hence disabling ACS is not an option", so a host that passes devices to virtual machines cannot use this tuning for those devices. Our guide to the RTX PRO 6000 Max-Q in virtual machines covers that side. Agree the settings with whoever owns the host's security, record them in the build documentation, and keep a dedicated inference host apart from hosts that run untrusted workloads.

Testing with simpleP2P, p2pBandwidthLatencyTest and nccl-tests

Run the tests from the smallest to the largest. Each step isolates one layer.

1. Topology. nvidia-smi topo -m and nvidia-smi topo -p2p p, saved for the build record.
2. simpleP2P. NVIDIA's cuda-samples include it, and NCCL's guide recommends it because it verifies the transferred data; a successful run ends with "Test passed". Build the samples with CMake as the cuda-samples README describes; its current version supports CUDA Toolkit 13.4, and -DCMAKE_CUDA_ARCHITECTURES=120 limits the build to compute capability 12.0, the RTX PRO 6000's.
3. p2pBandwidthLatencyTest. The cuda-samples test that produced the Level1Techs figures above prints bandwidth and latency matrices with P2P disabled and enabled. NCCL's guide points to NVIDIA's nvbandwidth tool for bandwidth measurement.
4. nccl-tests. Build with make and the CUDA_HOME and NCCL_HOME paths, as the nccl-tests README shows, then run all_reduce_perf with -b 8 -e 128M -f 2 -g 8 for eight GPUs on one node: sizes from 8 bytes to 128 MB, doubling each step. For a tensor-parallel pair, set -g 2 and choose the pair with CUDA_VISIBLE_DEVICES. The -c option checks results on each iteration.
5. Without P2P. Repeat step 4 with NCCL_P2P_DISABLE=1. NCCL's guide suggests comparing runs with NCCL_P2P_DISABLE and NCCL_P2P_LEVEL to see whether a problem lies in the P2P transport.

A run that hangs right after NCCL's "Connected all rings" message, with NCCL_DEBUG=INFO set, and completes with P2P disabled, points at the IOMMU or ACS. NCCL's environment documentation describes NCCL_DEBUG=INFO as printing debug information. Keep the outputs: the same tests after a BIOS, kernel or driver update show whether anything changed. Our guide to burn-in and acceptance testing puts them into a full acceptance run with DCGM.

vLLM flags when P2P fails

vLLM checks P2P itself before it uses its custom all-reduce. When the check fails, its source logs "Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed" and the all-reduce runs on NCCL. On more than two PCIe-only GPUs it uses NCCL by default anyway, so the custom kernel matters mainly for TP=2 pairs. vLLM 0.30.0 (release notes of 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for servers without NVLink (pull request 53576); our comparison of layouts without NVLink covers its limits and the little that is published about its gain.

vLLM engine arguments ("Disable the custom all-reduce kernel and fall back to NCCL"), vLLM custom_all_reduce source and parallelism guide (updated 15 October 2026); vLLM issue 30682 for the workarounds. Read 24 October 2026.

vLLM issue 30682, opened in December 2025 for two Max-Q cards at TP=2, collects these workarounds: --disable-custom-all-reduce alone or with NCCL_P2P_DISABLE=1 worked for some users and not for others, the reporter closed the issue after adding iommu=pt, and more shared memory for the container helped one user. In April 2026 a TP=2 instance on eight Server Edition cards hung at worker start-up until --disable-custom-all-reduce was set. vLLM's parallelism guide advises, for GPUs without NVLink, to "leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead".

Disable P2P in NCCL last. On the two-Max-Q system it let the all-reduce run, at about 31 GB/s of bus bandwidth, but it also hides the fault: the next driver or BIOS update may fix P2P without anyone noticing, so record the setting and test without it after each update.

Workstation boards, dual-socket servers and PCIe switches

The board decides how many pairs get a short path. On the Threadripper PRO workstation in the Level1Techs thread, every card hangs off a root port of one processor, which gives NODE between cards. On a dual-socket server without switches, half the cards sit on each socket, and a set that spans both shows SYS. In servers with PCIe switches, Dell's XE7745 technical guide of September 2026 attaches each GPU slot to one of four Broadcom Atlas II switches, and Supermicro's SYS-422GA-NRT datasheet lists a dual-root PCIe 5.0 switch design for up to eight 600 W cards; the switches add ACS to the list of settings to check.

NVIDIA's certified-systems guide, updated on 30 September 2026, recommends PCIe Gen5 switches and calls a switch optional for cost-effective inference servers. For tensor-parallel sets of two or four cards, a switch per set or a set per socket keeps the traffic short. In a rack server for continuous operation, plan RTX PRO 6000 Server Edition cards; the Max-Q belongs in a workstation or in a server whose maker lists it.

We supply the RTX PRO 6000 in all three editions and build the system around your planned layout, assembled and tested before delivery; describe the models and the split you plan, and we answer with the card count and the slot layout.

FAQ