Eight NVIDIA RTX PRO 5000 Blackwell 72 GB in one server: what 576 GB runs

576 GB at 2.4 kW: what eight cards add up to

A server with eight NVIDIA RTX PRO 5000 Blackwell 72 GB cards holds 576 GB of GPU memory at 300 W per card, of which 492.2 GiB is available for weights and KV cache by our sizing rule. That runs one model per card up to about 60 GiB of weights, Llama 3.3 70B in FP8 as four tensor-parallel pairs for 176 conversations of 8K, or Qwen3-235B-A22B in FP8 as two copies over four cards each. It does not hold models of 500 GB and more such as Kimi-K2.6, and the eight-card chassis that list this card also list the RTX PRO 6000 Max-Q: eight of those give 768 GB at the same 2.4 kW.

The NVIDIA RTX PRO 5000 Blackwell is a workstation card: 48 or 72 GB of GDDR7 with ECC, 1,344 GB/s, 300 W and an active blower in two slots, without NVLink. Eight of them belong only in a chassis whose maker lists the card. Which chassis do, and how ordering works, is on our page to buy an RTX PRO 5000 rack server; this page covers what the eight cards run.

NVIDIA RTX PRO 5000 product page (read 24 October 2026) and datasheets; NVIDIA MIG user guide (13 October 2026). Usable memory and sums by our estimate, ECC off. "Server" means Server Edition.

Usable memory follows our users-per-card rule: 90 per cent of the memory the driver reports, less 3 GiB per card for the CUDA context, graphs and activations. A 96 GB card reports 95.6 GiB; for the 72 GB card we scale that to 71.7 GiB, which leaves 61.5 GiB per card and 492.2 GiB for eight. The rule and its derivation are in our guide to how many users one RTX PRO 6000 can serve. If ECC reserved the 6.25 per cent that NVIDIA's CUDA Best Practices Guide gives for GDDR memory, a card would hold about 57.5 GiB; NVIDIA publishes no figure for GDDR7.

Eight separate 72 GB pools: one model per card or split

Each card addresses only its own memory. A model that fits one card runs as one copy per card with no traffic between cards, and a failed card takes out one copy. A larger model is split over two, four or eight cards over PCIe 5.0, and every generated token then waits for the cards to exchange partial results. The 72 GB size draws the line in an awkward place for one popular model: Llama 3.3 70B in NVIDIA's FP8 checkpoint has 67.7 GiB of weights, 6 GiB more than one card holds by our rule.

Our estimate by the users-per-card rule with 61.5 GiB per card. Weights from the published checkpoints as in our model guides: gpt-oss-120b 65.3 GB, Qwen3-235B-A22B-Instruct-2507-FP8 236 GB, GLM-4.7 FP8 362 GB, Kimi-K2.6 repository 595 GB, converted to GiB.

The 4-bit 70B model is the natural fit: 39.8 GiB of weights leave 21.7 GiB per card, enough for 17 conversations of 8K each, and eight copies give eight independent services. gpt-oss-120b loads on one card only with 0.7 GiB to spare, which holds no conversation and disappears if ECC takes memory, so we plan it on TP=2 pairs. A server that must hold Kimi-K2.6 or DeepSeek-V3.2, whose repository is 690 GB, needs more than 576 GB; eight RTX PRO 6000 Server Edition cards hold Kimi-K2.6 with about 13.8 GiB per card to spare, as our Kimi K2 hardware guide explains.

Conversations per layout by our sizing rule

The table counts conversations held at the same moment with an FP8 KV cache: 1.25 GiB per conversation of 8,192 tokens for Llama 3.3 70B, 1.0 GiB for Qwen3-32B and 0.625 GiB for Qwen3-14B. gpt-oss-120b is counted at 32K tokens with a 16-bit cache of 1.125 GiB. Each cell names the layout.

Our estimates, memory-bound, rounded down: (usable memory of the copy − weights) ÷ cache per conversation × copies. Not measurements. The RTX PRO 6000 Server Edition column matches our guide to what an 8× RTX PRO 6000 Server Edition server runs.

Fewer copies hold more conversations, because each copy stores its weights only once. Llama 3.3 70B in FP8 reaches 284 conversations as two TP=4 copies and 339 as one TP=8 copy on the 72 GB cards, at the price of more traffic over PCIe per token. As a planning aid, the example values in our guide to a private ChatGPT server by company size put about 80 requests in flight at peak for 2,000 employees, so four pairs of 70B in FP8 cover that case twice by memory, or two pairs with 88 conversations cover it while four cards run other models.

For a mixed platform the arithmetic is per card. A typical split: two pairs for Llama 3.3 70B in FP8 with 88 conversations, two cards with Qwen3-32B for coding, one card each for an embedding model, a reranker and speech-to-text, and one card for experiments. Our guide to several models on one GPU server describes how to route requests across such a layout. Send us your model list and the peak requests your gateway shows, and our team returns a card-by-card layout for eight RTX PRO 5000 and for the alternatives.

Larger models on 576 GB

Qwen3-235B-A22B in its FP8 release, 219.8 GiB, fits four cards with 26.3 GiB left for cache. With an FP8 cache of about 2.94 GiB per conversation of 32K, each copy holds 8 conversations, so two TP=4 copies hold 16; at 8K the two copies hold about 70. Two copies are the safer layout: vLLM's Qwen3 usage guide on GitHub (issue 17327) reports an error with FP8 block-128 MoE weights at some tensor-parallel sizes and suggests --tensor-parallel-size 4, or 8 with expert parallelism. On eight RTX PRO 6000 Server Edition the same two copies hold 76.

GLM-4.7 in FP8, 337.1 GiB, fits only over all eight cards and leaves 155.1 GiB for cache, about 13 conversations of 32K with a 16-bit cache, against about 28 on eight 96 GB cards. That works for a small team; for a department the 96 GB class is the better fit. Models of 500 GB and more are not sensible on 576 GB: DeepSeek-V3.2 does not load, and Kimi-K2.6 at 554.1 GiB exceeds the 492.2 GiB we count as usable. Our guide to LLM hardware requirements by model lists the sizes.

The RTX PRO 5000 has the same compute capability 12.0 as the RTX PRO 6000, so the kernel support and gaps that our eight-card RTX PRO 6000 guides describe apply to it as well. What differs is memory per card and bandwidth.

Speed: 1,344 GB/s per card and PCIe between them

Generating a token reads every active weight once, so bandwidth divided by the bytes read per token caps the speed of one conversation. By our arithmetic one RTX PRO 5000 is held to about 33 tokens per second on Llama 3.3 70B in NVFP4, which reads about 40.6 GB per token, against about 39 on an RTX PRO 6000 Server Edition and 44 on a Max-Q. A TP=2 pair running the FP8 version, 70.6 GB per token, has a ceiling of about 38 tokens per second before the exchange over PCIe. For gpt-oss-120b, a mixture-of-experts model that reads about 5 GB per token, the ceiling per card is about 270 for one conversation; in a batch the tokens reach different experts and a step reads more. These are upper bounds by our estimate from the specifications, not forecasts.

vLLM keeps its fast custom all-reduce for two PCIe-only GPUs and, by default, disables it on more than two, with the warning "Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs"; TP=4 and TP=8 then use NCCL. vLLM 0.30.0 of 22 September 2026 added an opt-in FlashInfer all-reduce over PCIe for two, four or eight GPUs; we found no published gain for it on eight cards. vLLM's parallelism guide recommends pipeline parallelism over tensor parallelism on GPUs without NVLink. Our article on one model on several GPUs over PCIe or NVLink explains the trade-off, and pairs belong on the same processor socket.

We found no published measurement of four or eight of these cards in one server. The published single-card figures are in other workloads: Puget Systems measured the 72 GB card 7 per cent behind the RTX PRO 6000 Max-Q in Topaz Video 1.6.1 in August 2026, as our RTX PRO 5000 benchmarks article reports.

MIG and vGPU on eight cards

NVIDIA's product page gives the RTX PRO 5000 up to two MIG instances. The MIG user guide lists the 48 GB card with two 24 GB instances, so eight cards give 16; the 72 GB card is not yet in the guide, and its datasheet gives two instances of 36 GB, which would make 16 isolated GPUs of 36 GB. By our rule a 36 GB instance holds Qwen3-14B in FP8 with cache for about 22 conversations of 8K, or a 32B model in 4-bit for about nine, but not a 32B model in FP8. MIG needs Linux, driver 575.51.03 or later and the cards switched to compute display mode with DisplayModeSelector, which turns their display outputs off. Read the profile names on the delivered cards with nvidia-smi mig -lgip.

Only the 72 GB card is on NVIDIA's vGPU list, from vGPU 20.2 of August 2026, supported until March 2027, and of the hypervisor pages we read only NVIDIA's validated platforms for Red Hat Enterprise Linux with KVM list it, with the graphics editions vWS and vPC. An eight-card server for virtual desktops on VMware needs the RTX PRO 6000 Server Edition or the RTX PRO 4500 Server Edition; our guide to MIG and vGPU: VMs per GPU counts seats per card.

Power, chassis and inlet temperature

The eight cards draw 2.4 kW. With two server processors and 15 per cent for memory, drives, fans and power-supply losses, we estimate about 3.6 kW at the wall with two 360 W AMD EPYC 9555 and about 3.9 kW with two 500 W Xeon 6900, the ceiling Supermicro gives for its SYS-322GA-NR. A 230 V 16 A feed planned at 80 per cent carries 2.94 kW, so the server needs a 32 A feed per side, which carries 5.89 kW. Eight RTX PRO 6000 Server Edition cards at 600 W come to about 6.3 kW by the same method. All of it becomes heat: 3.6 kW is about 12,300 BTU/h.

2CRSi Godì 1.8SP5-D product page and Supermicro datasheets, read on 24 October 2026. Each names the RTX PRO 5000 Blackwell in its GPU list; none gives a separate count for it.

2CRSi lists the RTX PRO 5000 with 48 or 72 GB at 300 W for the Godì, with slots rated for GPUs of up to 350 W, four PCIe 5.0 x16 slots per processor and 4+1 supplies of 2,000 W; its 25 °C ceiling needs a server room that holds the rack front below that. Supermicro's datasheets list a chassis maximum of double-width GPUs and no count per model, so we confirm eight cards with the maker. Supermicro's eight-card SYS-422GA-NRT and SYS-422GL-NR do not list the card. Feed types are in our guide to GPU server power on 16 A, 32 A and three-phase.

When to choose another configuration

• One 70B model in FP8 per card, or models above 500 GB: eight RTX PRO 6000 Server Edition, 96 GB per card, passive and listed by most GPU server makers; see our guide to what an 8× RTX PRO 6000 Server Edition server runs.
• The same 2.4 kW with more memory, on bare metal: eight RTX PRO 6000 Max-Q in a chassis that lists both cards; every chassis in the table above lists the Max-Q too, and our comparison of eight GPUs at 300 W sets out the Max-Q against a Server Edition held at 300 W.
• Many small models, MIG and vGPU on VMware within about 2.2 kW: eight RTX PRO 4500 Server Edition, as in our page on what an 8× RTX PRO 4500 Server Edition server runs.
• Half the size: four RTX PRO 5000 72 GB with 246.1 GiB usable, which fits one 16 A feed.
• Training and fine-tuning of large models: our training node with eight H200 NVL and NVLink bridges.

Eight RTX PRO 5000 remain a fair choice where the team already runs the card on its workstations, where every model fits 61.5 GiB per copy, or where graphics vGPU on Red Hat KVM is part of the job. If none of that applies, the choice between them and eight Max-Q comes down to the price of the two configurations, which we quote side by side.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the RTX PRO 5000 with 48 or 72 GB and build eight-card servers to order on chassis whose maker lists the card, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Quantities and lead time are confirmed in writing with the quote. Tell us the rack position, its feed and its inlet temperature, and we answer with the chassis, the card layout and the alternatives.

FAQ