Eight NVIDIA RTX PRO 6000 Blackwell Server Edition cards: which large LLMs fit 768 GB

768 GB in eight pools: what a model can use

A server with eight NVIDIA RTX PRO 6000 Blackwell Server Edition cards holds open models up to about 640 GiB of weights, which covers Llama 4 Maverick, Kimi-K2.6 in INT4, DeepSeek-V4.1-Flash and, with almost no room for users, DeepSeek-V3.2 in FP8. By our users-per-card rule 664.3 GiB of the 768 GB is usable for weights and KV cache. GLM-5.3 in FP8 is larger than that, and so are the BF16 versions of all these models.

The 768 GB are eight separate pools of 96 GB. NVIDIA's product brief SP-12355-001_v02 lists NVLink as "Not supported", so a model larger than one card is split over PCIe Gen5 x16, and how it is split decides how many conversations fit and how fast they run.

NVIDIA RTX PRO 6000 Blackwell Server Edition product page and product brief SP-12355-001_v02 (June 2025), read on 24 October 2026; 95.6 GiB is the 97,887 MiB nvidia-smi reports.

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for CUDA graphs and activations: 0.9 × 95.6 − 3 = 83.04 GiB per card. Whatever the weights leave is KV cache. The rule and its derivation are in our guide to how many users one RTX PRO 6000 can serve. It matters more at eight cards than at four, because a large mixture-of-experts model can leave a few GiB per card, and a few GiB is the difference between one conversation and fifty.

Large open MoE models on eight cards

The models that need eight cards are mixture-of-experts models: hundreds of billions of parameters stored, a few tens of billions read per token. Memory sets whether they fit; the bandwidth of each card, and the traffic between cards, set how fast they answer.

Our estimates by arithmetic, not measurements: concurrent conversations each holding 32,768 tokens, 16-bit KV cache unless stated. Checkpoint sizes from Hugging Face (Kimi-K2.6 595 GB and DeepSeek-V3.2 690 GB read on 24 October 2026; the others as in our model guides, read 9 to 24 October 2026). DCP: decode context parallelism.

The figures follow each model's cache layout. Qwen3-235B-A22B stores 188 KiB per token in 16-bit, 5.9 GiB per 32K conversation, and has four KV heads. GLM-4.7 has eight KV heads, one per card under TP=8, so its 11.5 GiB per conversation is spread over the server. Llama 4 Maverick takes about 3 GiB of FP8 cache per conversation. DeepSeek and Kimi K2 use compressed latent attention, which vLLM keeps in full on every card of a tensor-parallel copy; that is why the 13.8 GiB per card left by Kimi-K2.6 holds only six conversations, and why decode context parallelism, which splits each cache by token, raises the figure to about 51. Our guides to Kimi K2, DeepSeek and GLM work through each model, and LLM hardware requirements by model places them side by side.

Two newer models fit by memory as well: Meta's Muse Spark Open, whose NVFP4 checkpoint is about 533 GB and whose card asks for at least 700 GB of GPU memory, and Reflection AI's Beam-1.0 at about 503 GB in FP8. Our guide to Muse Spark Open hardware requirements sizes its cache and concurrent conversations on this server; we have not sized the cache of Beam-1.0 here. For Llama 4, note that Meta's Acceptable Use Policy withholds the Section 1(a) rights to the multimodal models from companies with their principal place of business in the EU, with an exception for end users of a product that includes them; our model hub quotes the clause, and your legal department decides whether it applies.

Layouts: TP=8, two TP=4 copies or expert parallel

Eight cards allow five useful layouts, and the choice is made per model.

• One copy per card: for models up to about 70 GB, such as gpt-oss-120b. No traffic between cards, and a failed card costs an eighth of the capacity.
• Four TP=2 copies: for models of 100 to 150 GB.
• Two TP=4 copies: for models of 150 to 300 GB. In a two-socket server each copy can sit under one processor, so no tensor-parallel traffic crosses the link between the sockets.
• One TP=8 copy: for models above 300 GB. Every layer then waits for an all-reduce across all eight cards and both sockets.
• TP=8 with expert parallelism: vLLM's --enable-expert-parallel places whole experts on cards "instead of tensor parallelism for MoE layers", while attention stays tensor-parallel.

For Qwen3-235B-A22B, two TP=4 copies hold 38 conversations of 32K by our estimate and one TP=8 copy about 37. vLLM's documentation explains the near tie: once the tensor-parallel size exceeds the number of KV heads, the cache "will be duplicated for tp_size / H times", here twice. vLLM's Qwen3 usage guide adds that the FP8 checkpoint fails at TP=8 on its 128-wide quantisation blocks and suggests "--tensor-parallel-size 4 or --tensor-parallel-size 8 --enable-expert-parallel". Two copies also keep half the service running when one copy has to restart.

All-reduce over PCIe is the cost of every split. By default vLLM disables its custom all-reduce on more than two GPUs without NVLink and uses NCCL; release 0.30.0 of 22 September 2026 added an opt-in FlashInfer PCIe IPC all-reduce for servers without NVLink (pull request 53576), VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1, for TP sizes 2, 4 and 8, with no published TP=8 gain on this card that we found. Pipeline parallelism sends only activations between stages, and for latent-attention models two stages halve the cache each card holds. Our comparison of TP=2, TP=4 and replicas without NVLink and the guide to P2P, IOMMU and ACS for NCCL cover the settings.

We build eight-card Server Edition servers to order and work out the layout per model before we quote; send us the model, its quantisation and the context length, and our team replies with the fit, the layout and the conversations it holds.

Published runs on eight cards

Few large-model results on these cards are published, and their conditions differ. The table lists the ones with a date, a layout and a stated engine.

Helix engineering blog (8 × Server Edition in a 4U server, two Xeon 6505P, 256 GB, driver 595.84; DSpark is a speculative-decoding draft); StorageReview review of a liquid-cooled 4U system with eight RTX PRO 6000 Blackwell cards, edition not stated, one EPYC 9474F, one card at PCIe x8, 256 input and 256 output tokens. Both read on 24 October 2026.

Helix ran the two TP=4 instances behind a load balancer that keeps a conversation on one instance, and found that setup ahead of an eight-card pipeline layout of a quantised build, which reached 122 tokens per second. The StorageReview rows are total throughput at a large batch in a liquid-cooled build whose card edition is not stated, so they are a ceiling for batch work and not a Server Edition figure for interactive chat.

MLPerf® Inference v6.1, published on 16 September 2026, holds eight-card Server Edition results from several server makers. They run Llama 2 70B and gpt-oss-120b, models small enough for one card each, so they show the server's total throughput on mid-sized models, not the speed of a large model split over all eight cards.

MLPerf® Inference: Datacenter v6.1, closed division, Llama 2 70B (llama2-70b-99.9) and gpt-oss-120b, offline scenario, tokens per second, entries 6.1-0070, 6.1-0013, 6.1-0036, 6.1-0050 and 6.1-0082 (Nebius RTX PRO 6000 n1), each with eight RTX PRO 6000 Blackwell Server Edition, retrieved from mlcommons.org on 24 October 2026. Results verified by MLCommons Association. Per card is our division of the Llama 2 70B result (gpt-oss-120b for 6.1-0082) by eight, not an MLPerf metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The four Llama 2 70B results lie within 2.6 per cent of each other, so the server maker matters less than the layout and the software. Our MLPerf Inference v6.1 for PCIe GPU servers adds the server scenario.

Kernel support on Blackwell RTX (SM120), as of October 2026

The RTX PRO 6000 has compute capability 12.0, the SM120 family, while most kernels for large MoE models are written first for the data-centre Blackwell chips. These are the gaps in published sources as of 24 October 2026; check each against the engine version you will run.

1. Sparse attention kernels. Helix found the FlashMLA sparse-decode kernel for DeepSeek-V4-Flash missing on SM120 and filled it with a community CUDA extension; its NVFP4 KV cache hit an assertion, solved with the fp8_ds_mla cache type.
2. Speculative decoding. Helix's DSpark launch failed on eight Blackwell cards until it switched to a community recipe and a checkpoint with the draft head bundled. Our DeepSeek guide records that vLLM's recipe of 29 September 2026 verified the -0731 checkpoint on eight RTX PRO 6000 without speculative decoding.
3. FP8 scaling. TensorRT-LLM's support matrix (1.3.0rc29) lists only per-tensor FP8 for this chip family, while the FP8 checkpoints of DeepSeek, GLM-5.3 and Qwen3.8 use 128 × 128 block scales.
4. NVFP4 selection. vLLM's Llama 4 Scout recipe selects the NVFP4 checkpoint only on compute capability 10.0, not 12.0.
5. Driver. NVIDIA's brief names R575 and CUDA 12.9 as the minimum; Helix ran 595.84.

Our page on Blackwell RTX SM120 software support tracks these items. For a model where a kernel is missing, the fallback path runs, but slower, and your acceptance test on the delivered server shows by how much.

Models that do not fit, and the alternatives

GLM-5.3 in FP8 needs 704 GiB for its weights, more than the 664.3 GiB of eight cards. Kimi-K2-Instruct-0905 in block FP8 is 1,029 GB. BF16 versions of the large models are about twice their FP8 size. DeepSeek-V3.2 and Mistral Large 3 load, but with one conversation of 32K each they suit only a test system; a service for a team needs more memory.

For these, eight H200 NVL give 987 GiB by the same rule, with NVLink bridges joining four cards at a time; our guide to an eight-card H200 NVL server sizes it. A DGX B300 has about 270 GB of HBM3e per GPU and NVLink across all eight; in MLPerf® Inference v6.1 (closed division, offline, entry 6.1-0071, retrieved from mlcommons.org on 24 October 2026, result verified by MLCommons Association) it reached 109,277 tokens per second on Llama 2 70B and 106,637 on gpt-oss-120b. We supply the DGX B300 per project; DGX B300 vs 8× RTX PRO 6000 compares the two.

Host memory, NVMe and power for weights this size

NVIDIA's configuration guide for NVIDIA-Certified Systems, updated on 30 September 2026, asks for at least two sockets, six physical cores per GPU and system memory of at least twice the GPU memory: 48 cores and 1,536 GB for eight cards. NVIDIA's RTX PRO AI Factory reference architecture of 18 May 2026 asks for at least 128 GB per GPU, 1 TB, and seven cores per GPU. Helix ran its server with 256 GB, below both, which works for one model held entirely in GPU memory but leaves nothing for a second checkpoint in the page cache or for a KV cache offloaded to the host. A 595 GB checkpoint plus the next version you want to test needs more than the one 1 TB NVMe drive per socket in NVIDIA's guide; plan several terabytes of local NVMe. Our guide to how much RAM a GPU server needs works through the cases.

Eight cards draw 4.8 kW at 600 W and 3.6 kW at the 450 W cap. With two AMD EPYC 9555, 64 cores at a default TDP of 360 W each, and 15 per cent for memory, drives, fans and conversion losses, we estimate about 6.3 kW at the wall; Dell's datasheet for its eight-card PowerEdge XE7740 and XE7745 gives "~8 kW" at full load. Either figure is above the 5.89 kW we plan per 230 V 32 A single-phase feed, so the rack position needs a 400 V three-phase feed, and with A/B feeds each side carries the whole server. Our guide to RTX PRO 6000 Server Edition power at 600, 450 and 300 W has the table per card count. We check rack, feed and airflow before we quote; tell us your feed and inlet temperature together with the model, and we say whether an eight-card server fits your room.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two to eight RTX PRO 6000 Server Edition cards, assembled and tested before delivery, on one EU contract and invoice, with manufacturer warranty on every component and the assembly covered by us. Quantities and lead time are confirmed in writing with the quote. Our page to buy an 8× RTX PRO 6000 Server Edition server lists the variants, and four cards are enough for models up to about 300 GB. Models, RAG and the platform on top are our Private AI/ML service.

FAQ