4× NVIDIA RTX PRO 4000 Blackwell in one workstation: 96 GB, models, users and power

Four separate 24 GB GPUs in four slots

Four NVIDIA RTX PRO 4000 Blackwell cards put 96 GB of GPU memory into one tower, as four GPUs of 24 GB each. By our users-per-card rule that is room for four copies of a 14B model, one 32B model split over all four cards, or a 70B model in 4-bit over all four; it is not one 96 GB GPU. The build is at its best when four people, four models or four jobs each get their own card.

The RTX PRO 4000 is the single-slot card of the RTX PRO Blackwell range: in NVIDIA's desktop GPU overview it is the only Blackwell workstation card listed as single slot. Four of them fit in four adjacent slots, where four dual-slot cards need eight slot widths.

NVIDIA RTX PRO 4000 Blackwell datasheet (5323450, June 2026) and NVIDIA's professional desktop GPU overview, read on 24 October 2026. NVIDIA's product page gives 1,290 AI TOPS and 40 TFLOPS FP32 at the boost clock; the datasheet gives 1,178 and 37, and this page uses the datasheet. Four-card totals are our arithmetic.

The card has fifth-generation Tensor Cores with FP4, an active cooler and no NVLink, so the four cards exchange data only over PCIe. NVIDIA's MIG user guide does not list it, and NVIDIA's list of GPUs supported by vGPU software, updated on 2 October 2026, does not include it.

How much of the 96 GB holds models and cache

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for the CUDA context, CUDA graphs and activations; the rest holds weights and KV cache. A 96 GB RTX PRO 6000 reports 95.6 GiB. We scale that figure by capacity for the 24 GB card, an assumption, which gives 0.9 × 23.9 − 3 = 18.5 GiB per card and 74.0 GiB for four. Check it with nvidia-smi -q on your own card. Our guide to how many users one RTX PRO 6000 can serve derives the rule.

Four cards pay the 3 GiB overhead four times. One RTX PRO 6000 Max-Q with the same 96 GB holds 83.0 GiB by the same rule, 9 GiB more than the four RTX PRO 4000 together.

With an FP8 KV cache, one conversation of 8,192 tokens takes 0.5625 GiB on Qwen3-8B, 0.625 GiB on Qwen3-14B, 1.0 GiB on Qwen3-32B and 1.25 GiB on Llama 3.3 70B. A 16-bit cache doubles these.

Models and layouts on four 24 GB cards

Four cards allow three layouts: one copy per card, two TP=2 pairs, or one TP=4 copy over all four. The more cards a copy spans, the more room it has for conversations, and the more traffic it sends over PCIe on every layer.

Our estimates by the users-per-card rule, memory-bound and rounded down, not measurements: concurrent conversations of 8,192 tokens with an FP8 KV cache; gpt-oss-120b at 32,768 tokens with a 16-bit cache of 1.125 GiB. Weights as in our sizing guides.

Two rows show the arithmetic. One copy of Qwen3-14B on one card leaves 18.5 − 15.2 = 3.3 GiB, which is five conversations. One TP=4 copy of Llama 3.3 70B in NVFP4 has 74.0 GiB; less 39.8 GiB of weights that leaves 34.2 GiB, and 34.2 ÷ 1.25 gives 27 conversations.

The line sits at about 14B parameters in FP8 per card. Up to there, every card runs its own model. A 32B model in FP8 needs at least two cards, and a TP=2 pair has 37.0 GiB, which does not hold Llama 3.3 70B even in NVFP4. Llama 3.3 70B in FP8 loads on all four cards but leaves room for only five conversations; for that model our sizing points to a 96 GB card.

Users per model

Conversations are requests in flight at the same moment, not named users. Our guide to a private ChatGPT server by company size uses about 20 requests in flight at the busiest moment for 500 staff and about 40 for 1,000.

By those values, four copies of Qwen3-14B with 20 conversations cover a company of about 500, with the advantage that one card can restart or update while three keep serving. Qwen3-32B as one TP=4 copy with 42 conversations covers about 1,000. Llama 3.3 70B in NVFP4 with 27 suits close to 700 staff, and uses every card.

For a team, the more common use is mixed. One card runs a coding model for developers, one an embedding model and a reranker for document search, one a speech-to-text model, and the fourth stays free for experiments. Each engine then has its own memory and its own update cycle, as our guide to serving several models on one GPU server describes. We build this workstation to order and size it per model; send us the models you plan to run and the people who will use them, and our team replies with a layout per card.

Splitting a model over four cards without NVLink

A TP=4 copy cuts every layer into four parts, and after each layer the cards sum their partial results in an all-reduce over PCIe. vLLM uses its own custom all-reduce kernel only for up to two GPUs without NVLink; its source disables it "on more than two PCIe-only GPUs", so a TP=4 copy runs on NCCL. vLLM 0.30.0 of 22 September 2026 added an opt-in FlashInfer PCIe IPC all-reduce for two, four or eight cards without NVLink (pull request #53576), and we found no published result for it on the RTX PRO 4000. Our article on TP=2, TP=4 or replicas without NVLink explains both paths.

Bandwidth sets the ceiling for one conversation, because each token reads the weights once. By our arithmetic:

Our arithmetic, bandwidth ÷ bytes read per token, weights only. The TP=4 ceilings leave out the all-reduce time over PCIe, which nobody has published for this card; real decoding reaches a fraction of every ceiling.

The TP=4 ceilings look higher than the Max-Q's on paper, and the all-reduce takes part of that lead back on every layer. No publisher has measured four RTX PRO 4000 under TP=4, so plan on the memory figures and treat the speed as unknown until a test run on your model.

The slots matter as well. In Lenovo's ThinkStation PX the four PCIe 5.0 x16 slots sit two on each processor, and slots 1 to 5 need the second processor, so a TP=4 copy crosses the link between the sockets while each TP=2 pair stays on one. Dell's Precision 7960 wires two of its four GPU slots at PCIe 4.0 x16, which halves those cards' link bandwidth. Peer-to-peer traffic also depends on IOMMU and ACS settings, which our guide to RTX PRO 6000 P2P on Linux lists for any RTX PRO card.

Four cards, four jobs

Where the build stands out is separation. Each card is a whole GPU with its own memory, its own four DisplayPort 2.1b outputs and its own two encoders and two decoders. NVIDIA's datasheet gives each card up to four displays at 3840 × 2160 and 165 Hz, or two at 7680 × 4320 and 100 Hz.

• Four jobs at once. A container or process sees one card, and a crash or restart on one card leaves the other three running.
• Fine-tuning experiments in parallel. By the per-parameter rules in our GPU memory guide for fine-tuning, LoRA on an 8B model holds about 16.7 GB of model states, which leaves about 7 GB on a 24 GB card; four runs with different settings fit side by side. QLoRA on a 70B model, at 42.8 GB, fits on no single card.
• Video. Eight NVENC and eight NVDEC engines in one tower serve transcoding, video analytics or several remote sessions.
• Visualisation and AI beside each other. One or two cards drive the monitors for CAD or rendering, the others run models.

The RTX PRO 4000 has no MIG, so a card is never split; the unit of sharing is the whole card. Without vGPU, a virtual machine gets a card only by passthrough of the whole GPU, which the platform and hypervisor have to support; plan a test before relying on it.

Workstations that list four RTX PRO 4000

Dell Precision 7960 configurator (Canada), Lenovo PSREF ThinkStation PX specification of 9 October 2026 and HP Z8 Fury G6i specifications, read on 24 October 2026.

Dell's configurator offers the Precision 7960 with one, two, three or four RTX PRO 4000 Blackwell cards, listed at 145 W. Lenovo lists the card for the ThinkStation PX without a maximum count and with 140 W; NVIDIA's datasheet gives 145 W. For any tower, the maker's configurator decides the count, the slots and the power supply. We supply the cards for these towers and build four-card workstations to order on Threadripper PRO or Xeon W platforms; tell us the tower model your company uses, and we check slots, power cables and cooling before we quote.

By NVIDIA's configuration guide for NVIDIA-Certified Systems, written for servers and applied here by the same arithmetic, four 24 GB cards call for at least 192 GB of system memory and 24 processor cores. The cards take 64 PCIe 5.0 lanes; NVMe drives and a network card come on top.

Power and heat on 230 V

Our estimate at full load from NVIDIA's board power and a 350 W workstation processor, plus 15 per cent for memory, drives, fans and power-supply losses.

At about 1.1 kW the four-card tower draws under 5 A on a 230 V circuit, well inside a 16 A circuit planned at 80 per cent, 2.94 kW. The same 1.1 kW ends up as heat in the room under continuous load, so a closed office gets warm; a technical room or an air-conditioned office suits a machine that runs models all day. Four actively cooled cards in adjacent slots draw their air through narrow gaps, so the case fans and the tower's own GPU list matter more than in a two-card build.

One RTX PRO 6000 Max-Q, or four RTX PRO 4000

NVIDIA datasheets; conversations by our users-per-card rule with an FP8 KV cache, as on our page on two RTX PRO 4000 or one RTX PRO 6000 Max-Q.

For one large model the single Max-Q is the stronger option, at about half the board power, and with MIG it also gives four isolated 24 GB GPUs on Linux, at the cost of its display outputs in compute mode. Four RTX PRO 4000 remain the better choice for four whole GPUs with their own monitors, four separate jobs and video work, and for teams that grow card by card.

Two other builds sit between. Four RTX PRO 4500 Blackwell give 128 GB at 896 GB/s per card and 800 W in eight slot widths, as on our page to buy an RTX PRO 4500 workstation. Four RTX PRO 6000 Max-Q give 384 GB at 1,200 W of board power, as in four RTX PRO 6000 Max-Q in one workstation. The parts and the ordering steps for this build are on our page to buy an RTX PRO 4000 workstation, and our comparison of single-slot GPUs for AI covers the rack server alternative.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the RTX PRO 4000 Blackwell as cards for towers you already run and build workstations with up to four of them to order, assembled and tested before delivery, with the operating system, NVIDIA driver, CUDA and a container runtime installed on request. Every component carries the manufacturer warranty, handled through us, on one EU contract and invoice; quantities and lead time are confirmed in writing with the quote.

FAQ