4× NVIDIA RTX PRO 6000 Blackwell Server Edition in one server: LLMs, users and power

What four Server Edition cards add up to

A server with four NVIDIA RTX PRO 6000 Blackwell Server Edition cards has 384 GB of GPU memory, of which 332.2 GiB holds model weights and KV cache by our users-per-card rule. That runs gpt-oss-120b with about 240 conversations of 32K in flight, Llama 3.3 70B in FP8 with up to 211 conversations of 8K, or several smaller models side by side. The cards draw 2.4 kW at full power, and the whole server about 3.4 kW, more than one 16 A single-phase feed should carry.

The 384 GB are four pools of 96 GB. NVIDIA's product brief lists NVLink as "Not supported", so the cards exchange data over their PCIe Gen5 x16 links, and a model larger than one card is split with tensor or pipeline parallelism. The card is passive and takes its air from the server's fans.

NVIDIA RTX PRO 6000 Blackwell Server Edition product page and product brief SP-12355-001_v02 (June 2025), read on 24 October 2026; 95.6 GiB is the 97,887 MiB nvidia-smi reports.

Our users-per-card rule takes 90 per cent of the memory the driver reports, vLLM's default gpu_memory_utilization up to release 0.19, and subtracts 3 GiB per card for CUDA graphs and activations: 0.9 × 95.6 − 3 = 83.04 GiB. What is left after the weights is KV cache. The derivation is in our guide to how many users one RTX PRO 6000 can serve.

Which LLMs fit in 384 GB, and in which layout

Four cards can run a model in three layouts: four copies, one per card, with no traffic between the cards; two pairs, each copy split over two cards with tensor parallelism (TP=2); or one copy over all four (TP=4). Every copy stores its own weights, so fewer copies leave more memory for conversations.

Our estimates by arithmetic: concurrent conversations each holding a full 32,768-token context, 16-bit KV cache unless stated. Weights: safetensors files on Hugging Face, read on 24 October 2026. Cache per conversation: 1.125 GiB gpt-oss-120b, 2.14 GiB Qwen3.8-27B, 5 GiB Llama 3.3 70B in FP8, as in our model guides.

The cache comes from each config.json: gpt-oss-120b keeps the full context in 18 of 36 layers, 36 KiB per token in 16-bit, and Qwen3.8-27B in 16 of 64 layers, 64 KiB per token plus 144 MiB of linear-attention state. DeepSeek-V4-Flash-0731, a 167 GB checkpoint, also fits four cards as one TP=4 copy; it keeps its compressed cache in full on every card, as our DeepSeek hardware guide explains. Llama 4 Maverick (about 417 GB in FP8), DeepSeek-V3.2 (690 GB) and GLM-5.3 (756 GB) do not fit four cards, by the sizes in our LLM hardware requirements by model.

Users per model: copies, pairs or one copy on four cards

Llama 3.3 70B is the model most buyers ask about first; the table shows it at 8K, the context of a typical chat. NVIDIA's FP8 checkpoint is 72.7 GB, 67.7 GiB, and each 8K conversation takes 2.5 GiB of 16-bit cache or 1.25 GiB of FP8 cache.

Our estimate by the users-per-card rule, conversations of 8,192 tokens at full length; weights 67.7 GiB in FP8 and 39.8 GiB in NVFP4, NVIDIA's checkpoints of 72.7 and 42.7 GB.

For the third row, four cards hold 4 × 83.04 = 332.2 GiB; less 67.7 GiB of weights that leaves 264.5 GiB, and 264.5 ÷ 2.5 is 105 conversations, or 211 with an FP8 cache. One card alone leaves 15.3 GiB, six conversations.

These figures count requests in flight at the same moment. A company of 500 with an assistant open in every browser produces far fewer generations at once, so size against the peak at your gateway; conversations that never reach the full context use less cache.

The layouts trade capacity against independence. Four copies hold the fewest conversations, but nothing crosses PCIe and a failed card takes out only a quarter of the service. One copy over four cards holds the most, and every layer waits for an all-reduce across all four. We supply this server and size the layout per model; send us your models, context length and peak requests in flight, and our team replies with a configuration and the figures for each layout.

Splitting a model over PCIe without NVLink

vLLM's custom all-reduce is disabled on more than two GPUs without NVLink: its source logs "Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs" and uses NCCL instead. A TP=2 pair keeps the faster path and a TP=4 copy does not, by default. Release 0.30.0 of 22 September 2026 added an opt-in FlashInfer PCIe IPC all-reduce for such servers (pull request 53576), VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1, for TP sizes 2, 4 and 8 in BF16 and FP16; we found no published TP=4 gain for it on this card. vLLM's parallelism guide of 15 October 2026 still advises GPUs without NVLink to "leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead".

In a two-socket server, a pair under one processor avoids the link between the sockets, and nvidia-smi topo -m shows the path between every two cards. Our article on one model on several GPUs over PCIe or NVLink explains the options, and RTX PRO 6000 without NVLink: TP=2, TP=4 or replicas compares them on these cards.

Published results on four cards

MLPerf® Inference v6.0 holds a two-card and a four-card result from the same HPE server model. By our division of each result by its card count, the four-card system kept 97 per cent of the per-card rate of the two-card one, with its cards capped at 450 W.

MLPerf® Inference: Datacenter v6.0, closed division, available category, Llama 2 70B (llama2-70b-99), offline scenario, entries 6.0-0052, 6.0-0054 and 6.0-0047, retrieved from mlcommons.org on 24 September 2026. Results verified by MLCommons Association. Per card is our division of the system result, not an MLPerf metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

From two to four cards the result rose 1.94 times. The two runs also differ in driver build, so the 3 per cent per card is only an indication of what the cap costs. Our RTX PRO 6000 benchmarks add the server and interactive scenarios.

StorageReview measured four Server Edition cards with vLLM in an HPE ProLiant Compute DL380a Gen12 with two Xeon 6527P processors and 1 TB of memory.

StorageReview, HPE ProLiant DL380a Gen12 review, 6 November 2025, modified 14 October 2026; TP=4 for the batch-64 row from its enterprise GPU roundup of 14 October 2026. vLLM version and the precision of the Llama runs not stated.

At batch 32 each Llama 2 70B user still received 8.00 tokens per second, and gpt-oss-120b, which reads only its active experts per token, 47.54. Helix reported on 14 August 2026 that one TP=4 instance of DeepSeek-V4-Flash-0731 on four Server Edition cards reached 603 tokens per second at eight concurrent requests with vLLM and DSpark speculative decoding, with a community kernel filling a sparse-decode kernel missing for these cards.

600 W or the 450 W cap

NVIDIA's product brief defines two power modes, set by how the 16-pin cable is strapped: 600 W or 450 W maximum by default, each with a 300 W minimum. If the cable signals less than the default cap, "the card will not boot". A cap set with nvidia-smi must be set again after every driver load; one set out of band through SMBPBI persists. Our guide to the RTX PRO 6000 power limit shows the commands.

Server makers use the 450 W mode to fit four cards into 2U. Lenovo's version "Slot Capped at 450W" allows four cards in the ThinkSystem SR650a V4, against two at 600 W, and HPE offers iLO GPU power capping at 450 or 550 W for the ProLiant DL385 Gen11. StorageReview found that at 450 W the card "gave up 2 to 5 percent at short context" and "4 to 9 percent at long context", with 4.8 output tokens per second per rated watt on a 14B model against 3.8 at 600 W, measured on a Workstation Edition, which has the same GPU and memory. For four cards the cap saves 600 W, which often decides the power feed.

2U, 3U or 4U: servers that take four cards

The card runs only in a server whose maker lists it for that model, slot, cable and fan set. These four-card configurations are in the makers' documents.

Manufacturer documents read on 24 October 2026. HPE DL380a Gen12: five 2,400 or 3,200 W supplies for two or four double-wide GPUs, C19 or C19-C20 cords only. Cisco C845A M8: up to four 3.2 kW supplies with N+1 redundancy.

A 2U server packs four cards into the least space, at 450 W or with a maximum-performance fan kit; a 3U or 4U chassis has more room for air, power supplies and network cards. Inlet limits differ per configuration, and our list of RTX PRO 6000 Server Edition compatible servers covers cables and warranty.

Power at the wall and on EU feeds

With two AMD EPYC 9355, 32 cores each at a default TDP of 280 W, cards and processors are rated at 2.96 kW with the cards at 600 W. Memory, drives, network cards, fans and conversion losses come on top; we add 15 per cent for them, an assumption that a measurement on the delivered system confirms or replaces.

Our estimate: board power from NVIDIA's brief, 280 W per processor from AMD's EPYC 9355 page, plus 15 per cent for the rest of the server.

Our arithmetic: 230 V × amps, and √3 × 400 V × amps for three-phase; 80 per cent is the planning margin of our guide to GPU rack power and cooling.

At 600 W, a four-card server is above the 2.94 kW we plan per 16 A single-phase feed, so its rack position needs 32 A or three-phase; at 450 W it stays below by our estimate. With redundant power supplies on A and B feeds, each side has to carry the whole server. About 3.4 kW of heat needs roughly 540 CFM of air at an 11 °C rise, by the 160 CFM per kW in our guide to GPU rack power and cooling, and NVIDIA's brief asks for 43 to 125 CFM through each card at a 25 to 45 °C inlet. We check rack, feed and airflow before we quote; tell us your feed type, A/B requirement and inlet temperature, and we say which power mode and chassis fit.

Processors, memory and NVMe for four cards

NVIDIA's configuration guide for NVIDIA-Certified Systems, updated on 30 September 2026, asks for at least two sockets, a 2.0 GHz base clock, six physical cores per GPU, system memory of at least twice the total GPU memory and one NVMe drive per socket. For four cards that is 24 cores and 768 GB. Two EPYC 9355 give 64 cores and 24 memory channels, so 24 modules of 32 GB make 768 GB with one module per channel. Our inference node on AI servers follows this: four Server Edition cards, two AMD EPYC processors, 768 GB of DDR5 ECC and four NVMe drives. RAG with a large index or a KV cache offloaded to system memory needs about 1 TB, as our guide to CPU, RAM and storage for a 4-GPU server works out.

MIG and vGPU: 16 instances per server

MIG splits each card into up to four instances of 24 GB, two of 48 GB or one of 96 GB, so four cards offer up to 16 isolated GPUs of 24 GB. MIG is set per card: two cards can run a TP=2 pair while the other two serve small models in quarters. NVIDIA's MIG guide states that "NCCL is currently not supported with MIG", so a split copy runs on whole cards. Our RTX PRO 6000 MIG profiles show what fits in each size.

For virtual machines, the Server Edition is the only RTX PRO 6000 edition on NVIDIA's vGPU list: its brief states "Supports vGPU 19.0 or later", in the default Display Off mode. On vSphere, NVIDIA lists ESXi 8.0 Update 3g or later. MIG on bare metal needs no licence; compute vGPU profiles require NVIDIA AI Enterprise, licensed per GPU, as our vGPU profiles guide explains.

When two cards, eight cards or H200 NVL fit better

Two cards, 166.1 GiB by our rule, suit a pilot or one department. Eight cards, 664.3 GiB, hold Llama 4 Maverick in FP8; our eight-card RTX PRO 6000 allocation plans one for 1,000 staff. Where the service must survive the loss of a whole server, two four-card servers are the safer layout, as one 8-GPU server or two 4-GPU servers compares. For a 70B model serving many users interactively, the H200 NVL with HBM3e and NVLink bridges is the stronger card, as our RTX PRO 6000 vs H200 NVL comparison explains.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two to eight RTX PRO 6000 Server Edition cards, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, drivers, CUDA and a container runtime on request. The configuration, its variants and the ordering steps are on our page to buy a 4× RTX PRO 6000 Server Edition server. Models, RAG and the platform on top are our Private AI/ML service.

FAQ