llama.cpp on two to four GPUs: split-mode layer, row and tensor on NVIDIA RTX PRO cards

How llama.cpp puts one model on several GPUs

llama.cpp spreads a GGUF model over several NVIDIA GPUs with --split-mode. The default, layer, gives each card a slice of the layers and their KV cache and runs the cards one after another as a pipeline; row and the newer tensor mode split each weight matrix so that the cards compute in parallel. Without NVLink, as on every RTX PRO Blackwell card, layer mode sends the least over PCIe, and tensor mode is the one that can make a single conversation faster.

The flags below come from the llama-server README on GitHub, read on 24 October 2026, when the newest release was build b11829 of 22 October. llama.cpp publishes numbered builds rather than versions, several a day, so flags and defaults can change between builds; check llama-server --help on the build you run.

llama.cpp tools/server README and tools/llama-bench README on GitHub, read 24 October 2026; release list at build b11829 (22 October 2026).

--fit deserves a note. Since it is on by default, llama-server shrinks unset values such as the context size until the model fits the cards, down to --fit-ctx, 4,096 tokens by default, and --fit-target sets a margin in MiB per device. Set the context and the number of slots (-np) yourself on a shared machine, so that a restart after a driver update does not quietly give users a shorter context.

Layer, row and tensor: what moves over PCIe

In layer mode the first card computes its layers for a batch of tokens and hands the result to the second, which computes its layers, and so on. What crosses PCIe at each boundary is the activations of the tokens in flight, kilobytes per token, a tiny fraction of the weights, so the mode runs well over a plain PCIe link and needs no peer-to-peer access. The price is that for one conversation only one card works at a time. Two cards hold twice the model, but one stream generates no faster than it would on one card with enough memory. With several conversations, llama-server batches them and the pipeline keeps more than one card busy; the build guide recommends CUDA_SCALE_LAUNCH_QUEUES=4x "especially for multi-GPU pipeline parallelism" to improve prompt processing.

Row mode splits each weight matrix by rows, so every card computes part of every layer and the main GPU collects the results and holds the KV cache. Every layer then needs an exchange between the cards. In the three-way comparison of the tensor-mode pull request, on two RTX 4090 cards with an 8B model in Q4_0, row mode was the slowest of the three: 67.00 tokens per second in generation against 175.43 for layer, and 2,997 against 12,551 in prompt processing.

Tensor mode, added in pull request 19378 titled "backend-agnostic tensor parallelism (experimental)", splits weights and KV cache and synchronises the cards only where the computation requires it. Its description, read on 24 October 2026, says that "only the CUDA backend has received the necessary extensions and performance optimizations", that NCCL should be installed "for good performance", and that the feature "should be considered as an experimental feature that is not yet production ready".

llama.cpp pull request 19378, description read 24 October 2026; tokens per second with llama-bench, measured by the pull request's author on GeForce RTX 4090 cards, "All GPUs are connected via 16x PCIe 4.0". Generation is tg128, prompt processing pp512.

The author sums it up: tensor mode "works comparatively better for slow GPUs with fast interconnect running large, dense models" and worse for fast GPUs on slow links with small models. The 70B row shows the gain for a dense model, and the gpt-oss 20B row shows how little a small mixture-of-experts model gains. We found no published llama.cpp measurement of the split modes on RTX PRO Blackwell cards, which have PCIe 5.0 links, so treat these consumer-card figures as the direction of the effect, not as a forecast for your workstation.

Limits of tensor mode as of October 2026

The pull request lists the conditions under which tensor mode runs today:

• FlashAttention "must be enabled" and KV cache quantisation "must be disabled", so the cache stays in f16 or bf16
• --fit and backend sampling are not implemented, and memory use is "inefficient due to overallocations", so leave headroom on each card
• a VRAM leak from CUDA graphs that are "continually created but never destroyed"
• 23 architectures are listed as "definitely broken", among them deepseek2, mamba, mamba2, olmoe and t5
• any number of GPUs, up to 16; ROCm runs through HIP, Vulkan "technically works at short contexts", other backends "should be assumed to be broken"

For a production service that has to run for months, layer mode is the setting we would plan on today, with tensor mode tried on the same build in a test run against your own model. A model whose architecture is on the broken list stays in layer mode.

Peer-to-peer access between the cards

llama.cpp's CUDA backend copies data between GPUs through system memory unless peer-to-peer access is enabled. The build guide states that the environment variable GGML_CUDA_P2P "can be set to enable peer-to-peer access between multiple GPUs", that it needs driver support, usually on workstation or data-centre GPUs, and that it may cause crashes or corrupted output on some motherboards and BIOS settings, such as those with the IOMMU enabled. A published test showed P2P working between two RTX PRO 6000 Workstation Edition cards; whether it works in a given machine depends on the IOMMU and ACS settings, which our guide to RTX PRO 6000 P2P on Linux works through with nvidia-smi topo -m, NVIDIA's simpleP2P sample and nccl-tests. Layer mode barely needs P2P; tensor mode, which exchanges data at every layer, benefits from it and from NCCL.

We build workstations and servers with two to four RTX PRO cards and install the Linux release, driver and CUDA on request; send us your GGUF models and the number of people using them, and our team replies with a configuration and a starting split mode.

Sizing: which GGUF files fit two or four cards

llama.cpp does not reserve a fixed share of memory the way vLLM does, but the cards still need room for the CUDA runtime, compute buffers and the cache. We plan with our users-per-card rule: 90 per cent of the memory the driver reports, less 3 GiB per card, minus the weights. That gives 83.0 GiB on a 96 GB RTX PRO 6000 Max-Q and 61.5 GiB on a 72 GB RTX PRO 5000, scaled from the 95.6 GiB a 96 GB card reports. File sizes are from the Hugging Face file lists.

Our estimates by the users-per-card rule with a 16-bit KV cache: conversations of 8,192 tokens at 2.5 GiB each for Llama 3.3 70B, of 32,768 tokens at 1.125 GiB each for gpt-oss-120b. File sizes from bartowski/Llama-3.3-70B-Instruct-GGUF (Q8_0 in two parts of 39.9 and 35 GB) and ggml-org/gpt-oss-120b-GGUF, read 24 October 2026.

The arithmetic for one line: two Max-Q give 2 × 83.04 = 166.1 GiB; Q8_0 weights of 74.9 GB are 69.8 GiB, which leaves 96.3 GiB, or 38 conversations of 8K at 2.5 GiB. On a single 72 GB RTX PRO 5000, gpt-oss-120b leaves only 2.5 GiB, cache for two long conversations, so on that card it belongs on a pair. The cache figure leaves out the small cache of gpt-oss's sliding-window layers. llama-server's --cache-type-k accepts q8_0 and q4_0 among other types to shrink the cache, but not in tensor mode.

Where a model fits one card, do not split it. Two llama-server processes, each started with -sm none and its own GPU through CUDA_VISIBLE_DEVICES or -dev, give two independent copies with no traffic between the cards; a third process on another port can serve a second model. gpt-oss-120b on two Max-Q is the clear case: one copy per card holds 21 conversations each, and a failure on one card leaves the other answering. Our pages on two RTX PRO 6000 Max-Q in a workstation and four RTX PRO 5000 72 GB cover these builds for vLLM.

Speed for one conversation, by our estimate

Token generation reads the weights once per token, so memory bandwidth sets an upper bound. In layer mode the cards read their slices one after another, and one conversation sees the bandwidth of one card; in tensor mode they read in parallel, before the cost of synchronising.

Our estimate: 74.9 or 42.5 GB read per token divided by 1,792 GB/s per Max-Q or 1,344 GB/s per RTX PRO 5000 (NVIDIA product pages), times the number of cards for tensor mode. Upper bounds, not measurements; real results are lower, and tensor mode loses part of its gain to the exchange over PCIe.

The four-card figure in the pull request, 48.31 against 21.25 tokens per second for a 70B in Q4_K_M, a factor of 2.3 where the bound is 4, shows how much of the gain synchronisation over PCIe 4.0 consumed. For a dense 70B model used by one person at a time, tensor mode is the setting that changes how the model feels; for gpt-oss-120b, which reads only its active experts per token, a copy on one card is already fast, and splitting it gains less.

Models larger than GPU memory: experts in system memory

llama.cpp can keep part of a model in system memory, which vLLM handles differently. --cpu-moe keeps "all Mixture of Experts (MoE) weights in the CPU", and --n-cpu-moe N those of the first N layers; --override-tensor places tensors by name pattern. The attention layers and the shared weights stay on the GPUs, where they are read for every token, and the experts are read from DDR5 at the speed of the processor's memory channels. That makes a mixture-of-experts model larger than all GPU memory usable for one or two people, if slowly. Our guide to MoE expert offload with KTransformers works through such an offload for Kimi, and total vs active parameters explains why it works; the build guide also documents GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 on Linux, which lets the CUDA backend swap to system memory instead of failing when VRAM runs out.

Building llama.cpp for RTX PRO Blackwell

RTX PRO Blackwell cards have compute capability 12.0 in NVIDIA's CUDA GPU list. Build b11829 ships Ubuntu binaries for CUDA 12.8 and CUDA 13.4, and the build guide's CUDA build is cmake -B build -DGGML_CUDA=ON followed by cmake --build build --config Release, with CMAKE_CUDA_ARCHITECTURES set to your card's compute capability where nvcc cannot detect it. NCCL, which tensor mode wants, is a separate NVIDIA library installed beside CUDA.

Two settings in the build guide concern Blackwell's FP4 Tensor Cores. GGML_CUDA_MMQ_PREC sets the activation precision for NVFP4 and MXFP4 matrix multiplications: q4 keeps the native W4A4 path, which processes prompts faster at some cost in accuracy, and q8 uses W4A8 for every layer. gpt-oss-120b ships in MXFP4, so the setting applies to it. On driver branches, our guide to NVIDIA driver branches and CUDA versions explains which branch carries CUDA 13.

Comparing the modes on your own machine

llama-bench, which ships with llama.cpp, takes several values per parameter separated by commas, so one run compares the modes: -sm layer,tensor with -fa on measures both on the same model, -p 512 -n 128 sets the prompt and generation lengths, and -d adds a context depth, so the comparison can be repeated at 8K or 32K of context. Run it with the build, driver and power limits you will use in production. The GPUs a run uses can be limited with CUDA_VISIBLE_DEVICES, which also tells you how a pair compares with all four cards.

When vLLM is the better engine

llama-server batches the requests of its slots continuously, and with layer mode it serves a small team well. For an endpoint with dozens of requests in flight, a paged KV cache, prefix caching and tensor-parallel kernels matter more, and vLLM or SGLang is the engine we plan on, as our comparison of vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama explains. On cards without NVLink, vLLM's own guidance prefers pipeline over tensor parallelism; on RTX PRO 6000 cards, TP=2 pairs keep its custom all-reduce, which it disables by default on more than two PCIe-only GPUs, as RTX PRO 6000 without NVLink sets out. vLLM 0.30.0 (release notes, 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink (#53576).

Our recommendation from the llama.cpp README and pull request 19378 and vLLM's parallelism guide, as of October 2026.

Ollama, built on llama.cpp's GGML library, decides the split for you: it puts a model on one GPU when it fits and otherwise spreads it across all of them, without a choice of split mode.

What we supply

We supply the RTX PRO 6000 Max-Q and Workstation Edition, the RTX PRO 5000 48 GB and 72 GB and the RTX PRO 4500 and 4000, and build workstations with two to four cards to order, with the Linux release, driver, CUDA and container runtime installed on request, as our AI servers page states. For a shared service we build servers with RTX PRO 6000 Server Edition cards. The model, the serving engine and the platform on top are our Private AI/ML service; describe the models, the users and how they reach the machine, and we propose the build and the engine.

FAQ