NVIDIA H200 NVL alternatives for LLM inference: which jobs run on 32 GB and 48 GB cards
Eurokommerz, Vienna, since 2006: Private AI/ML · IT Managed Services · Enterprise Training · AI Hardware & Software
- An NVIDIA H200 NVL has 141 GB of HBM3e at 4.8 TB/s and up to 600 W, and NVLink bridges join two or four cards at 900 GB/s per GPU; by our sizing rule one card holds 123.4 GiB of weights and KV cache
- Inference on models that stay within 25.7 or 40.0 GiB per card does not need that: servers with four or eight NVIDIA RTX PRO 4500 Blackwell Server Edition cards (32 GB, passive, 165 W) and builds with RTX PRO 5000 Blackwell 48 GB cards do this work
- Eight Server Edition cards give 256 GB, of which 205.6 GiB is usable by our rule, at 1,320 W of board power; eight copies of Qwen3-14B in FP8 then hold 128 concurrent conversations of 8,192 tokens with an FP8 KV cache, by our estimate
- Neither alternative has NVLink, so a model larger than one card is split by the serving software over PCIe: Llama 3.3 70B in FP8 (67.7 GiB) runs on four 32 GB cards only as one tensor-parallel copy, while one H200 NVL holds it with room for 44 such conversations
- You can order the server alone, the setup of vLLM and your models alone, or both together: the server delivered with everything installed and configured
Supplied by Eurokommerz: NVIDIA RTX PRO 4500 Blackwell Server Edition Check price and availability →
H200 NVL alternatives for inference servers with 32 GB and 48 GB cards
For LLM inference, the alternatives to the NVIDIA H200 NVL in our range are servers with four or eight NVIDIA RTX PRO 4500 Blackwell Server Edition cards, 32 GB each, passive and 165 W, and builds with two to four NVIDIA RTX PRO 5000 Blackwell cards of 48 GB. By our sizing rule they hold models whose weights and cache stay within 25.7 or 40.0 GiB per card, several copies of such a model, and the embedding, speech and vision models around it. The H200 NVL keeps the work that needs its 141 GB on one card, NVLink between cards, training and fine-tuning of large models, or FP64.
The comparison is by job and by memory. It contains one published measurement of token rates, for a single RTX PRO 4500 Server Edition, and otherwise memory arithmetic by a rule that is stated where it is used.
| ITEM | H200 NVL | RTX PRO 4500 SERVER | RTX PRO 5000 48 GB |
|---|---|---|---|
| Memory | 141 GB HBM3e | 32 GB GDDR7 with ECC | 48 GB GDDR7 with ECC |
| Memory bandwidth | 4.8 TB/s | up to 800 GB/s | 1,344 GB/s |
| Budget by our rule | 123.4 GiB | 25.7 GiB | 40.0 GiB |
| Board power | up to 600 W, minimum 200 W | 165 W, minimum 100 W | 300 W |
| Format, cooling | dual slot, passive | single slot, passive | dual slot, active |
| Link between cards | NVLink bridge for 2 or 4 cards, 900 GB/s per GPU | PCIe 5.0 x16; NVLink "Not supported" | PCIe 5.0 x16; datasheet lists no NVLink |
| MIG | up to 7 instances | up to 2 × 16 GB | 2 × 24 GB |
| vGPU list, 2 October 2026 | not listed | from release 20.0 | not listed |
| Tensor Core precisions | FP8, no FP4 | FP4 and FP8 | FP4 and FP8 |
| Compute capability | 9.0 | 12.0 | 12.0 |
H200 NVL: NVIDIA's published figures as collected in our H200 NVL specs article; RTX PRO 4500 Server Edition from NVIDIA's datasheet 5192801 (April 2026) and product brief PB-12790-001_02 (20 April 2026); RTX PRO 5000 from datasheet 5349550 (June 2026); MIG from NVIDIA's MIG user guide (4 November 2026) and the Server Edition datasheet; compute capability from NVIDIA's CUDA page (16 October 2026). Budgets are our estimate.
The budget is what our sizing rule leaves for weights plus KV cache: 0.9 times the memory the driver reports, less 3 GiB per card. The driver reports 140.4 GiB on the H200 NVL, which gives 123.4 GiB. For the 32 GB and 48 GB cards the reported memory is scaled from the 95.6 GiB the driver shows on a 96 GB card; that scaling is an assumption, and the driver on the delivered card decides. MIG on the RTX PRO 5000 needs Linux, driver 575.57.08 or later, vBIOS 98.02.73.00.00 and the display mode set to compute. The full sheets are in our H200 NVL specs, the RTX PRO 4500 Server Edition specs and RTX PRO 5000 with 48 GB or 72 GB.
Inference jobs that do not need 141 GB per card
A model belongs on the smaller cards when its file and the cache for its conversations stay inside the budget of one card. The table sorts model files by that test.
| JOB | EXAMPLE FILE | ONE 32 GB CARD | ONE 48 GB CARD |
|---|---|---|---|
| Assistant, 14B class | Qwen3-14B FP8, 15.2 GiB | fits, 16 conversations | fits, 39 conversations |
| 27B class in 4-bit | Qwen3.8-27B NVFP4, 20.4 GiB | fits, about 5.3 GiB left | fits |
| 27B class in FP8 | Qwen3.8-27B FP8, 28.75 GiB | above our budget | fits |
| 24B class | Devstral Small 2 24B Instruct FP8, 24.02 GiB | fits | fits |
| 32B class | Qwen3-32B FP8, 32.0 GiB | does not fit | fits, 8 conversations |
| 35B class | Qwen3.6-35B-A3B FP8, 34.88 GiB | does not fit | fits |
| Embeddings, reranking | Nemotron 3.5 Embed 8B or Rerank 8B BF16, 15.61 GiB each | fits | fits |
| Speech to text | Whisper large-v3, about 10 GB by OpenAI | within card memory | within card memory |
| 70B class | Llama 3.3 70B FP8, 67.7 GiB | does not fit | does not fit |
File sizes of Qwen3.8-27B, Devstral, Qwen3.6-35B and Nemotron from Hugging Face, read 10 November 2026; weights of Qwen3-14B, Qwen3-32B and Llama 3.3 70B as in our model guides; Whisper figure as OpenAI states it in its README; fit classes and conversation counts (8,192 tokens each, FP8 KV cache) by our sizing rule: our estimate, not a measurement.
"Above our budget" means larger than the budget and still within 0.9 times the card's memory, and we do not plan on such a file; Qwen's FP8 file of Qwen3.8-27B is the example on 32 GB, where NVIDIA's NVFP4 checkpoint of the same model fits. On 32 GB cards the embedding model and the reranker of a retrieval pipeline take one card each in 16-bit. For vision models our sources hold no primary memory figure, so the table names none. Each model runs as its own serving instance, on a whole card or on a MIG instance, as our guide to serving several models on one GPU server describes.
Where the list is one mid-size model in several copies plus a few small models, the smaller cards do the work in the simplest layout: one copy per card, no data between cards, and a failed card takes out only its own copy.
You can order the server alone, the setup of vLLM and your models alone, or both together: the server delivered with everything installed and configured; write to us and we suggest the best solution for your tasks and budget.
Servers with four or eight RTX PRO 4500 Server Edition cards
The RTX PRO 4500 Server Edition is a passive, single-slot card of full height and full length for server airflow. Its datasheet gives 32 GB of GDDR7 with ECC, up to 800 GB/s, 165 W by default with a minimum of 100 W, one 16-pin power connector and three NVENC and three NVDEC engines. It needs a driver of the R595 branch from 595.58.03 or of the R620 branch.
| BUILD | GPU MEMORY | USABLE | BOARD POWER | AT THE WALL |
|---|---|---|---|---|
| 4 × RTX PRO 4500 Server Edition | 128 GB | 102.8 GiB | 660 W | about 1.4 kW |
| 8 × RTX PRO 4500 Server Edition | 256 GB | 205.6 GiB | 1,320 W | about 2.2 kW |
| 4 × H200 NVL | 564 GB | 493.4 GiB | 2,400 W | about 3.4 kW |
| 8 × H200 NVL | 1,128 GB | 986.9 GiB | 4,800 W | about 6.3 kW |
Memory and board power are sums of NVIDIA's figures; usable memory by our sizing rule; wall power by our estimate as on our pages for these servers: board power plus two server processors, plus 15 per cent for the rest of the server, to be confirmed by a measurement on the delivered system.
For the Server Edition builds the estimate uses two 280 W processors. A 230 V circuit with 16 A, planned at 80 per cent, carries 2.94 kW, so either build stays on one such feed; with the cards set to their 100 W minimum the estimates fall to about 1.1 and 1.6 kW. Eight H200 NVL cards at about 6.3 kW need a 400 V three-phase feed.
| BUILD | QWEN3-14B FP8 | QWEN3-32B FP8 | LLAMA 3.3 70B FP8 |
|---|---|---|---|
| 4 × RTX PRO 4500 Server Edition | 64, four copies | 38, two TP=2 copies | 28, one TP=4 copy |
| 8 × RTX PRO 4500 Server Edition | 128, eight copies | 76, four TP=2 copies | 56, two TP=4 copies |
| 2 × RTX PRO 5000 48 GB | 78, two copies | 48, one TP=2 copy | 9, one TP=2 copy |
| 4 × RTX PRO 5000 48 GB | 156, four copies | 96, two TP=2 copies | 73, one TP=4 copy |
Our estimate, memory-bound, not a measurement: concurrent conversations of 8,192 tokens with an FP8 KV cache (0.625, 1.0 and 1.25 GiB per conversation); weights 15.2, 32.0 and 67.7 GiB. TP=2 and TP=4 are tensor-parallel copies over two and four cards.
Conversations are requests held in the cache at the same moment, far fewer than the staff who use the service. With the example values of our guide to a private ChatGPT server by company size, 40 per cent of staff active in the busy hour, 6 requests an hour, 30 seconds per request and a peak factor of 2, about 4 requests are in flight per 100 employees. By those values the 64 conversations of four copies cover about 1,500 staff, and the 128 of eight copies cover 2,000 staff with headroom. They are example values that a pilot replaces with the figures of its own gateway.
One card has been measured. StorageReview, on 14 October 2026, ran a single RTX PRO 4500 Server Edition in a Dell PowerEdge R770 with vLLM 0.27.1 at 32 concurrent requests: Ministral 3 14B in FP8 produced 1,107 output tokens per second with 512 input and 512 output tokens, and about 180 with 8,192 input and 1,024 output tokens. Multi-card throughput has not been published in a form we can cite.
MIG splits each card into two 16 GB instances, eight per four-card server and sixteen per eight-card server. A half has a budget of 11.3 GiB by the same rule, enough for Nemotron 3.5 Embed 1B in 16-bit (2.90 GiB) and too little for the 8B version. A split copy runs on whole cards.
Supermicro's page of systems for the card, read 24 October 2026, lists eight cards only in 3U, 4U and 5U models (SYS-322GA-NR, SYS-422GL-NR, SYS-422GL-FNR2, SYS-522GA-NRT) and four in the 2U SYS-222H-TN and SYS-222C-TN; its largest 2U listing is six. Our builds start at 24 processor cores and 256 GB of DDR5 ECC memory for four cards and 48 cores and 512 GB for eight. We supply Supermicro, Gigabyte, ASUS, Dell and HPE servers per project and check the rack, power and airflow before we quote; send us your models, their precision and the peak of requests in flight, and we reply with a layout per model. The builds are described in four RTX PRO 4500 Server Edition in 2U and eight RTX PRO 4500 Server Edition in one server, the orders in buy a 4× RTX PRO 4500 Server Edition server and buy an 8× RTX PRO 4500 Server Edition server, and the card alone in buy the RTX PRO 4500 Server Edition.
RTX PRO 5000 48 GB cards in workstations and listed chassis
The RTX PRO 5000 with 48 GB brings 40.0 GiB per card by our rule, which puts Qwen3-32B and Qwen3.6-35B-A3B in FP8 on one card. It is an actively cooled workstation card of 300 W in two slots. Two to four of them go into a workstation: four give 192 GB, of which 160.1 GiB is usable, at 1,200 W of board power and about 1.8 to 2 kW at the wall by our estimate with one 350 W processor.
In a rack the card belongs only in a chassis whose maker lists it, and for continuous operation in a rack we recommend a Server Edition card, as our page to buy a rack server with RTX PRO 5000 cards explains. The eight-card server on our pages, eight RTX PRO 5000 in one server, is built with the 72 GB version of the card, which we also supply. For the 48 GB card no LLM token rate with stated conditions has been published, so the table above gives its conversation counts and no speed. The card and its builds are on our pages to buy the RTX PRO 5000 Blackwell 48 GB and to buy a workstation with two to four of them.
Models larger than one card are split over PCIe
Eight 32 GB cards are eight memory pools of 32 GB, and four 48 GB cards are four of 48 GB. The product brief of the Server Edition states NVLink as "Not supported", and the datasheet of the RTX PRO 5000 lists none. A model larger than one card is split by the serving software, with tensor or pipeline parallelism, and every split puts traffic between the cards onto PCIe. NVIDIA's H200 page puts PCIe Gen5 at 128 GB/s against 900 GB/s per GPU for the NVLink bridges of the H200 NVL.
vLLM keeps its custom all-reduce on two GPUs without NVLink and disables it on more than two, where it falls back to NCCL; release 0.30.0 of 22 September 2026 added an opt-in all-reduce over PCIe. We found no consistent published measurement of tensor-parallel scaling on four or eight RTX PRO 4500 Server Edition or RTX PRO 5000 cards. The counts for split copies in the table therefore say what fits in memory and nothing about the speed of each conversation. Our article on one model on several GPUs over PCIe or NVLink covers the link.
The order of planning follows from this: choose a precision in which the model fits one card, then run copies, and split only what fits no single card.
What stays with the H200 NVL
- One large model on one card. One H200 NVL holds Llama 3.3 70B in FP8 with room for 44 conversations of 8,192 tokens with an FP8 KV cache; four cards hold 176 as four copies and eight hold 352, with no data between cards.
- Models of several hundred GB. By the sizes in our LLM hardware requirements by model, Llama 4 Maverick in FP8, about 417 GB, takes four H200 NVL cards, and DeepSeek-V3.2 with 690 GB or GLM-5.3 with 756 GB take eight. Models of that size do not fit a server with eight 32 GB cards.
- Split models on NVLink. Bridges join two or four cards; an eight-card server has two domains of four, joined by PCIe 5.0, as eight H200 NVL in one server shows.
- Training, fine-tuning and FP64. Training and fine-tuning of large models belong on H200 NVL cards with NVLink bridges; our AI servers page lists eight of them as its training node. NVIDIA gives the card 30 TFLOPS of FP64.
- The licence. Each H200 NVL includes a five-year NVIDIA AI Enterprise subscription.
The buying decisions for the card are in buying H200 NVL in the EU.
Choosing between the three
Start from the model list and the precision of each model. Models up to about 20 GiB of weights with their cache fit one RTX PRO 4500 Server Edition, and several of them make a four-card or eight-card server. Models up to the 35B class in FP8 fit one RTX PRO 5000 48 GB. A 70B model in FP8 for many users, anything of several hundred GB, and training are H200 NVL work.
Related reading for the server itself: Supermicro, Gigabyte and ASUS 8-GPU servers compared, Dell PowerEdge XE servers for H200 NVL and HGX B300 and HPE ProLiant servers for AI. The same comparison one class lower is RTX PRO 6000 alternatives with 48 GB and 32 GB, and what to put into a request is listed in private AI sizing request: what we need to know.
A first call about the workload is free of charge. Tell us which models have to run and how many people use them, or write to request@
FAQ
What is an alternative to the H200 NVL for LLM inference?
Can an inference server work without H200 cards?
How many users does a server with eight RTX PRO 4500 Server Edition cards serve?
Can a 70B model run on 32 GB or 48 GB cards?
Do the RTX PRO 4500 Server Edition and the RTX PRO 5000 have NVLink?
How much power does an RTX PRO 4500 Server Edition server draw against an H200 NVL server?
When is the H200 NVL the right card for inference?
We supply servers with RTX PRO 4500 Server Edition or H200 NVL cards alone, set up vLLM and your models alone, or deliver both together. Describe your task in a few sentences and our team will help. We reply within one business day.
Check price and availabilityWe reply within one business day