NVIDIA RTX PRO 4000 Blackwell for local LLMs: what 24 GB holds on one, two and four cards

A local LLM on the RTX PRO 4000 Blackwell and what 24 GB holds

One NVIDIA RTX PRO 4000 Blackwell has room for a local language model whose file is up to about 18 GiB in size: by our sizing rule its 24 GB give a budget of 18.5 GiB for the weights and the context. That covers gpt-oss-20b in MXFP4 and Ministral 3 14B Instruct in FP8 with room to spare, and Qwen3.5-9B in BF16 with almost none. Two or four cards add budget, each card with its own 24 GB. A measured figure in tokens per second has not been published for this card in the sources we accept.

The task behind the question is usually one of three. A developer wants a coding or chat model at the desk. A team wants answers from its own documents without sending them outside the company. Or an IT department tries models on one seat before it sizes a server. In all three the first thing to settle is whether the model file fits the memory, and that can be worked out before anything is ordered.

The 18.5 GiB budget and how we count it

A language model occupies GPU memory twice: once for its weights, which is the size of the file, and once for the KV cache, the working memory that grows with the text of the conversation. Our sizing rule reserves room for both. It takes 0.9 times the memory the driver reports and subtracts 3 GiB of overhead for the runtime; what remains is the budget for weights plus KV cache.

For the RTX PRO 4000 Blackwell that budget is 18.5 GiB. NVIDIA publishes no driver-visible memory figure for the card, so we scale the reported memory from the 95.6 GiB the driver shows on the 96 GB card. The scaling is our assumption; the figure that counts is the one nvidia-smi shows on the delivered card. The GDDR7 memory of the Blackwell cards has ECC that is always on and takes nothing from the usable memory.

The rule was written for a serving engine such as vLLM with several users. For one user who loads a GGUF file in llama.cpp or Ollama it is on the careful side. We state it with every figure because a number without its rule cannot be checked. Our guide to how many users one RTX PRO 6000 can serve derives it.

Which LLM files fit one RTX PRO 4000 Blackwell

The sizes are those of the publishers' own files on Hugging Face, read 10 November 2026. "Left" is what remains of the budget for context.

our arithmetic by the sizing rule, not a measurement; file sizes and licence tags from the publishers' Hugging Face repositories, read 10 November 2026.

Among the files of the same reading that do not fit one card are Gemma 4 12B-it in BF16 (22.28 GiB), Devstral Small 2 24B Instruct in FP8 (24.02 GiB), Qwen3.8-27B in FP8 (28.75 GiB) and Qwen3.6-35B-A3B in FP8 (34.88 GiB). Between 17.98 and 22.28 GiB our list holds no file, so on this card there is no borderline case.

Size a model by its file and not by its name. Gemma 4 E4B-it is a checkpoint of about 8.0 billion parameters, and its BF16 file has 14.89 GiB. gpt-oss-20b is published only in MXFP4. And one model comes in several files: Ministral 3 14B has 7.67 GiB as a Q4_K_M GGUF file with its mmproj file, 13.37 GiB as Q8_0, 14.65 GiB in FP8 and 25.97 GiB in BF16, of which the first three fit this card.

The room that is left decides how the model can be used. With 5.7 GiB left, gpt-oss-20b has room for context to spare; with 0.5 GiB left, Qwen3.5-9B in BF16 fits by the rule and has room for very little text. How many tokens a GiB of cache holds differs from model to model, and our guide to how much VRAM an LLM needs shows the arithmetic. The mixture-of-experts models of the 30B class on this card are the subject of 30B MoE models on 24 GB GPUs.

We supply the RTX PRO 4000 Blackwell across the EU; send us the model you plan to run, its format and the context length and we reply with the fit by our rule and a written quote within one business day.

FP4, FP8 and GGUF files on a Blackwell card

NVIDIA lists the RTX PRO 4000 Blackwell at compute capability 12.0. Its fifth-generation Tensor Cores add FP4 to FP8 and BF16, the formats in which several of the files above are published, and NVFP4 files compute natively only on Blackwell cards. GGUF files are the format of llama.cpp and Ollama. The formats and the cards that run them are set out in our article on FP8, NVFP4, MXFP4, INT4 and GGUF.

One figure from the datasheet should not be read as a speed. NVIDIA's datasheet (June 2026) gives 1,178 AI TOPS and 37 TFLOPS, its product page 1,290 and 40. AI TOPS is a theoretical FP4 figure with sparsity, no measured rate and no count of tokens.

RTX PRO 4000 Blackwell LLM benchmark, AI performance and tokens per second

For the RTX PRO 4000 Blackwell no measurement of language-model speed exists in the sources we accept, which are published tests with a stated system and method. We therefore print no tokens per second and no estimate of them.

The nearest thing is Puget Systems' professional GPU roundup of 18 December 2025, run on a Ryzen 9 9950X3D. It measured the card against the RTX 4000 Ada in content-creation software: 83 per cent faster in V-Ray GPU, 40 per cent in Blender Cycles, 39 per cent in Topaz Video AI. In the same article Puget ran MLPerf Client v1.0 and reports that the Blackwell cards averaged 50 per cent over the Ada cards of the same number in the rate of the second and later tokens, with the RTX PRO 2000 at 25 per cent. The text names no separate value for the RTX PRO 4000 and no absolute rate, and these are Puget's own runs, not results published by MLCommons.

What the speed of a local model depends on can be named without a number: the size and precision of the file, the engine, the length of the context, the number of requests at once and the card's memory bandwidth of 672 GB/s. A test run with your model on the delivered card settles it.

Two and four RTX PRO 4000 Blackwell cards and the limits without NVLink

The RTX PRO 4000 Blackwell is a single-slot card, so a tower takes two or four of them where it has the slots, the power and the maker's approval.

our arithmetic from NVIDIA datasheet 5323450 (June 2026) and the sizing rule above.

Several cards can be used in three ways. Each card runs its own model, for example a chat model, a coding model and an embedding model side by side. Each card runs a copy of the same model, which multiplies the requests that can be in flight. Or the serving engine splits one larger model over the cards. By weights alone, Gemma 4 12B-it in BF16 and Qwen3.8-27B in FP8 lie inside the 37.0 GiB of a pair, and Qwen3.6-35B-A3B in FP8 leaves 2.1 GiB of it for context. Whether a given checkpoint can be split depends on the engine and on the model.

The limits follow from the hardware. NVIDIA's datasheet lists no NVLink for the card, and each card works with its own 24 GB, so a model that is split exchanges data between the cards over PCIe. NVIDIA's MIG guide and vGPU list do not include the card; several users share a card through the serving engine. Our pages on four RTX PRO 4000 in one workstation, on two RTX PRO 4000 or one RTX PRO 6000 Max-Q and on one model on several GPUs over PCIe work through the layouts.

For one larger model a larger card is often the better order than a second small one.

NVIDIA datasheets 5323450 (June 2026), 5108623 (April 2026) and 5349550 (June 2026); budgets by our sizing rule; "largest file fitting" is the largest file of our list that fits by the rule; file sizes from Hugging Face, read 10 November 2026.

One NVIDIA RTX PRO 5000 Blackwell with 48 GB takes the two slot widths of a pair of RTX PRO 4000 at 300 W against 290 W and holds 40.0 GiB on one GPU instead of 37.0 GiB on two. By our reading the pair is the better order for two people or two models, and the single card for one model above 18.5 GiB. The NVIDIA RTX PRO 4500 Blackwell Workstation Edition sits between them; RTX PRO 4000 vs RTX PRO 4500 Blackwell compares the two neighbours. The card pages are buy the RTX PRO 4500 Blackwell with its 32 GB spec page and buy the RTX PRO 5000 Blackwell 48 GB with RTX PRO 5000 48 GB vs 72 GB. For the class below see local LLMs on 16 GB cards, and for the Ada generation RTX 5000 Ada and RTX 4500 Ada for AI.

Stable Diffusion and ComfyUI on the same 24 GB

Image models are sized differently, because the diffusion model, its text encoders and the VAE are separate files. By our arithmetic against nine tenths of the card's memory, with nothing measured, the following holds for 24 GB. SDXL base 1.0, Stable Diffusion 3.5 Medium and FLUX.2 klein 4B fit with all their parts. Stable Diffusion 3.5 Large fits with all parts when the T5 encoder is the FP8 file. Of FLUX.2-dev in NVFP4 only the model file fits, and the BF16 files of FLUX.1-dev and Qwen-Image exceed that limit and run only with partial loading from system memory.

The licences differ. FLUX.2 klein 4B is under Apache 2.0. SDXL is under OpenRAIL++, which permits commercial use within its use restrictions. Stable Diffusion 3.5 is under the Stability AI Community License, whose conditions need reading, and FLUX.1-dev and FLUX.2-dev are under non-commercial licences. An image model and a language model that are loaded at the same time share the 24 GB, and the budget above then no longer applies. Our guide to ComfyUI on RTX PRO 2000, 4000 and 4500 has the table for all three cards.

The workstation around the card

NVIDIA's quick start guide asks for a PCI Express x16 slot and for at least as much system memory as GPU memory, and it recommends twice as much: 24 GB and 48 GB for one card, 96 GB and 192 GB for four. Each card has a total board power of 145 W and one PCIe CEM5 16-pin connector, and an adapter from one 8-pin PCIe cable is in the box.

On Linux the current drivers 620.70.06 (R620), 595.112.07 (R595) and 580.196.07 (R580) list the card. NVIDIA pairs CUDA 13.5 with R620, 13.2 with R595 and 13.0 with R580, and CUDA 13.x applications need a driver of branch 580 or later. Among the Linux distributions in NVIDIA's release notes are Ubuntu 26.04, 24.04 and 22.04 LTS, RHEL and Rocky 10.x, 9.y and 8.y, and Debian 13 and 12.

You can order the card or the workstation alone, the setup of the model and its serving software alone, or both together: the workstation delivered with everything installed and configured; write to us and we suggest the best solution for your tasks and budget. The card itself is described on our page to buy the RTX PRO 4000 Blackwell and in the RTX PRO 4000 Blackwell specs; how it stands against GeForce cards of 16 to 32 GB is in RTX PRO 4000 Blackwell vs RTX 5070 Ti, 5080 and 5090.

FAQ