Best GPU for an LLM server in 2026: five NVIDIA server cards by model size and users

Five server cards for an LLM server, as of October 2026

For most LLM servers in 2026 the NVIDIA RTX PRO 6000 Blackwell Server Edition is the card to start from: 96 GB holds a 70B model in FP8 or gpt-oss-120b on one card, and its FP4 Tensor Cores run NVFP4 models natively. Choose the NVIDIA H200 NVL when many users chat with a 70B-class model at once, when one copy of a 400B-class model must fit four cards, or for fine-tuning. Choose the RTX PRO 4500 Blackwell Server Edition, L40S or L4 when the models are 32B or smaller and the server should run many of them at low power.

We compare only the server cards we supply, and the comparison reflects NVIDIA's documents and published measurements as of October 2026. All five are passive PCIe cards for rack servers. They differ in memory, which sets the largest model, in bandwidth, which sets the speed per user, and in power, which sets how many fit a rack position. Workstation cards are covered in our best workstation GPU for AI guide.

NVIDIA product pages and datasheets for the H200, RTX PRO 6000 Blackwell Server Edition, L40S, RTX PRO 4500 Blackwell Server Edition and L4, read on 24 October 2026. FP8 on Ada (L40S, L4) and Hopper (H200 NVL); FP4 Tensor Cores on Blackwell only.

Model size decides the card

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for the CUDA context and activations. The driver reports 95.6 GiB on a 96 GB card and 140.4 GiB on the H200 NVL; for the GDDR cards with less memory we scale the 96 GB figure in proportion, which gives the usable memory in the table. The derivation is in our guide to how many users one RTX PRO 6000 can serve.

Our estimate by the users-per-card rule; usable memory 18.5 GiB on the L4, 25.7 GiB on the RTX PRO 4500 Server Edition, 40.0 GiB on the L40S and 83.0 GiB on the RTX PRO 6000 Server Edition. Weights from the publishers' and NVIDIA's checkpoints on Hugging Face as listed in our model guides. The H200 NVL, 123.4 GiB usable, holds every model in the table with more room for cache.

Over the RTX PRO 6000 Server Edition, the H200 NVL adds 40 GiB per card and the NVLink bridge. Two bridged cards hold Qwen3-235B-A22B in FP8, 219.8 GiB, and four hold Llama 4 Maverick in FP8, about 388 GiB, as one copy in one NVLink domain. On the RTX PRO 6000 Server Edition, Qwen3-235B needs four cards split over PCIe and Maverick needs eight. On the H200 NVL, NVFP4 checkpoints run only weight-only with 16-bit arithmetic, so models are sized there in FP8 or INT4, as our guide to FP4 models on the H200 NVL explains.

Users per card by our sizing rule

A server is sized by the requests in flight at the busiest moment, each holding a KV cache next to the weights. The table counts conversations of 8,192 tokens with an FP8 KV cache: 0.5625 GiB each for Qwen3-8B, 0.625 GiB for Qwen3-14B, 1.0 GiB for Qwen3-32B and 1.25 GiB for Llama 3.3 70B.

Our estimate, memory-bound, not measurements: (usable memory − weights) ÷ cache per conversation, rounded down. The H200 NVL holds 203, 173, 91 and 44 conversations of Qwen3-8B, Qwen3-14B, Qwen3-32B in FP8 and Llama 3.3 70B in FP8.

With the example values of our private ChatGPT server sizing by company size, 500 employees produce about 20 requests in flight at peak, 1,000 about 40 and 2,000 about 80. For a 70B model at 2,000 staff that is seven copies on RTX PRO 6000 Server Edition cards or two on H200 NVL cards by the table; layouts that split a model over several cards hold more, since fewer copies store fewer weights. Our sibling articles count the layouts for four RTX PRO 6000 or four H200 NVL and for twelve RTX PRO builds. We supply all five cards and the servers around them; send us the models, the context length and the number of users, and our team proposes the card and the count with the figures per card.

Context length: where the cache runs out

Long contexts move the limit from the weights to the cache. At 32K tokens, Llama 3.3 70B needs 5 GiB of FP8 cache per conversation, so one RTX PRO 6000 Server Edition holds three such conversations beside its weights and one H200 NVL eleven. gpt-oss-120b keeps its full context in only half its layers: at 32K it needs 1.125 GiB of 16-bit cache, so one RTX PRO 6000 Server Edition holds 19 conversations and one H200 NVL 55.

The smaller cards show the same limit in measurements. In StorageReview's roundup, the RTX PRO 4500 Server Edition with 8,192-token inputs and 1,024-token outputs flattened from eight streams and reached 190 output tokens per second on Ministral 3 14B in FP8 at 32 streams, behind the L40S at 230; the L4 was flat from four streams on every model, at 34 to 130 tokens per second. The RTX PRO 6000 Workstation Edition reached 608 and the H200 NVL 1,038. Agents, RAG over long documents and coding assistants belong on 96 or 141 GB cards for this reason, as our guide to long-context LLM hardware works out.

Speed per user and published measurements

Generating a token for one user reads every weight of a dense model once, so the single-stream ceiling is bandwidth divided by the weight bytes. By our arithmetic, Qwen3-14B in FP8, about 16.3 GB, tops out near 18 tokens per second on the L4, 49 on the RTX PRO 4500 Server Edition, 53 on the L40S, 98 on the RTX PRO 6000 Server Edition and 294 on the H200 NVL. Llama 3.3 70B in FP8, with 70.6 GB read per token, reaches about 23 on the RTX PRO 6000 Server Edition and 68 on the H200 NVL. Real figures land below these ceilings.

StorageReview ran all five cards with the same software, the NGC vLLM 26.08 container with vLLM 0.27.1 and driver 595.71.05, and published the results on 14 October 2026. The H200 NVL ran at 450 W, the most its Dell PowerEdge R770 slot delivers, and the RTX PRO 6000 entry was a Workstation Edition at 600 W, the same GPU with 1,792 GB/s.

StorageReview, Best Enterprise GPUs, published 14 October 2026, modified 20 October 2026: output tokens per second at 32 concurrent streams; Ministral 3 14B Instruct in FP8 at 512/512 (chat) and 8,192/1,024 (8K); Qwen3.8-27B in NVFP4. Per rated watt is our division of the chat result by the tested power, not a measured efficiency.

The order follows bandwidth on the FP8 model and FP4 support on the NVFP4 model, where the Hopper card falls behind the RTX PRO 6000 Workstation Edition. StorageReview notes that the H200 NVL "is rated to 600W, and we have not run it there", so its figures are a floor. Among the low-power cards the RTX PRO 4500 Server Edition gives the most tokens per rated watt at short context.

MLPerf® Inference: Datacenter v5.0 and v6.0, closed division, available category, Llama 2 70B (llama2-70b-99), offline scenario, entries 5.0-0018, 5.0-0017, 6.0-0022 and 6.0-0021, retrieved from mlcommons.org on 24 September 2026. Results verified by MLCommons Association. Per card is our division of the system result, not an MLPerf metric; compare entries within one round. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Within each round the same Dell chassis was used for both cards: in v5.0 the H200 NVL delivered about nine times the L40S per card on Llama 2 70B, in v6.0 about 18 per cent more than the RTX PRO 6000 Server Edition. MLPerf® Inference v6.1 has no H200 NVL, L4 or L40S entry; we found no comparable language-model result for the L4 in recent rounds. Our RTX PRO 6000 benchmarks list the v6.1 results of the Server Edition.

Power per card and per server

Our estimate: NVIDIA board power, plus two processors at AMD's default TDP, 280 W each (EPYC 9355) with the L4, RTX PRO 4500 Server Edition and L40S and 360 W each (64-core EPYC 9555) with the 600 W cards, plus 15 per cent for memory, drives, fans and conversion losses. One feed: 230 V × 16 A = 3.68 kW, planned at 80 per cent, 2.94 kW.

Our estimate is a floor for the largest servers. Dell's AI Factory datasheet gives about 8 kW at full load for its eight-card RTX PRO 6000 Server Edition nodes, more than our estimate, so the feed is planned on the maker's figure. Eight 600 W cards need three-phase power or several feeds at the rack position, with redundant supplies on A and B feeds, since 6.3 kW is more than the 5.89 kW we plan on a 32 A feed; a four-card server at 600 W per card, about 3.4 kW, is already above one 16 A feed. Both the RTX PRO 6000 Server Edition and the H200 NVL can be capped, and StorageReview found 2 to 5 per cent lost at short context at 450 W on the RTX PRO 6000 Workstation Edition, which it used in place of the Server Edition. Our guide to GPU server power on 16 A, 32 A and three-phase covers feeds and supplies.

MIG, vGPU and licences

For several teams on one server, MIG splits a card into isolated instances: up to seven of 1g.18gb on the H200 NVL, four of 24 GB on the RTX PRO 6000 Server Edition and two of 16 GB on the RTX PRO 4500 Server Edition, by NVIDIA's MIG user guide. The L40S and L4 have no MIG and share by time slices. NVIDIA's vGPU list includes the L4 from release 15.2, the L40S from 16.1, the RTX PRO 6000 Server Edition from 19.0 and the RTX PRO 4500 Server Edition from 20.0; the H200 NVL has compute vGPU only, through NVIDIA AI Enterprise. Each H200 NVL includes a five-year NVIDIA AI Enterprise subscription; for the other four cards it is licensed per GPU where your stack needs it, while vLLM and other open-source engines run without it. Our guide to MIG and vGPU: how many VMs per GPU counts instances per card.

Which card for which LLM server

• Company assistant on a 70B or 120B-class model, up to a few hundred requests in flight: RTX PRO 6000 Server Edition, two or four per server. It runs NVFP4 natively, splits into four MIG instances and has graphics vGPU for mixed hosts.
• Interactive chat for thousands of staff, long contexts, 235B to 400B models: H200 NVL, four per NVLink domain, with three times the bandwidth per card and the AI Enterprise subscription included. It is also the card for fine-tuning across cards and for FP64.
• Many small models, embeddings and rerankers, dense racks on one 16 A feed: RTX PRO 4500 Server Edition, up to eight in a 3U or 4U server whose maker lists it, for models up to 14B in FP8 or 32B in NVFP4.
• A 32B model in FP8, or an existing L40S platform: L40S, with 48 GB and PCIe 4.0. For new servers with FP4 models, the Blackwell cards are the newer choice.
• Branch offices, edge servers and models up to about 8B: L4, at 72 W in a low-profile slot, with a few concurrent conversations per card.

The first AI project guide covers the smallest starting point, and the private LLM vs cloud API comparison covers whether to run the server at all. Where two cards fit equally, ask us for both configurations and we quote them side by side.

When one server is not enough

Eight H200 NVL in two NVLink domains, our training node on AI servers, hold DeepSeek-V3.2 in FP8 and give 1,128 GB of HBM3e for fine-tuning runs, as our eight-card H200 NVL server shows. Eight RTX PRO 6000 Server Edition give 768 GB over PCIe for Llama 4 Maverick and many models side by side. Models that need one NVLink domain larger than four cards go to the DGX B300, which we supply per project; we check price and availability for your project, as our DGX B300 page describes.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply all five cards for servers you already run and build AI servers to order with two to eight of them, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Quantities and lead time are confirmed in writing with the quote. Models, RAG and the platform on top are our Private AI/ML service, and the cards are on our GPU page.

FAQ