Sizing a GPU server for a 70B model: a worked example, from weights to wall socket

The example: an assistant for 30 users on Llama 3.3 70B

A manufacturing company wants an internal assistant on Llama 3.3 70B: document questions, drafting, summaries. Thirty people may be using it at once, conversations run to about 8,000 tokens of context, and replies have to stream at reading speed. All data stays on-premise. Here is the sizing worksheet for it, step by step.

Step 1: the weights

Llama 3.3 70B has 70.6 billion parameters. Bytes per parameter set the footprint: two in FP16, one in FP8, and 0.5625 in NVFP4, because every sixteen values share an FP8 scale.

Sources: Llama 3 paper and model cards, NVIDIA checkpoint cards on Hugging Face, NVIDIA NVFP4 technical blog.

In FP16 the weights alone take 131 GiB. They fill an H200 NVL with about 9 GiB left, which is no room for thirty users, and do not fit a 96 GB card at all, so FP16 needs two or four cards or a different model. FP8 and NVFP4 are the realistic options, and the choice between them is for whoever owns the evaluation set: a two-point drop on MMLU may not matter for a summariser but can rule a model out for a compliance tool.

Step 2: the KV cache

Every token in a live conversation keeps its keys and values in GPU memory for the whole session. The size per token is 2 × layers × KV heads × head dimension × bytes. For Llama 3 70B that is 2 × 80 × 8 × 128 = 163,840 values: 320 KiB per token in FP16, 160 KiB in FP8. This plan assumes an FP8 cache, which vLLM uses only when asked (--kv-cache-dtype fp8); by default it keeps the cache in the model’s 16-bit type.

Thirty users at 8,192 tokens each: 30 × 8,192 × 160 KiB = 37.5 GiB. Add about 8 GiB for activations, the CUDA context and fragmentation on one card, or 10 GiB in total across two cards (each card has its own CUDA context, while the activations are split), and the memory plan looks like this.

TP = tensor-parallel across two cards. Totals include about 8 GiB of overhead (10 GiB across two cards) for activations, the CUDA context and fragmentation. Capacities are what the driver reports: nvidia-smi shows 97,887 MiB (95.6 GiB) for the RTX PRO 6000 and 143,771 MiB (140.4 GiB) for the H200 NVL. This worksheet adds up weights, cache and a fixed overhead and compares the total with that full capacity; our VRAM guide starts instead from 90 per cent of card memory less about 3 GiB per card, so its session counts for the same card are lower.

This is the table that answers most sizing calls. The 96 GB card is a one-card solution for this workload only in NVFP4, and at 90 % utilisation there is little room for a longer context or five more users, which makes it a pilot configuration rather than a production endpoint. The H200 NVL takes the FP8 model with about 26 GiB free. Two Blackwell cards take it in FP8 with 75 GiB of headroom, at the cost of tensor-parallel traffic over PCIe, since these cards have no NVLink.

We build each of these options to order; the two-card plan is close to our Private AI starter server. Tell us where your case differs from this one, and we reply with a configuration and a quote within one business day.

Step 3: speed for one user and throughput under load

Speed has to be checked for one user and for all users together. For one user, generation is bound by memory bandwidth, and the ceiling is bandwidth divided by the weight bytes read per token. For the FP8 model that is about 68 tokens per second on the H200 NVL (4,800 GB/s) and about 23 on an RTX PRO 6000 (1,597 GB/s); for NVFP4 on the Blackwell card about 39. Reading speed is under 10 tokens per second, so all three pass for one user.

For all users together, what counts is the aggregate throughput within a latency limit. The interactive scenario of MLPerf® Inference v6.0 for Llama 2 70B caps time-to-first-token at 450 ms and time-per-token at 40 ms. Eight H200 NVL in a Dell XE7740 reached 16,343.77 tokens per second in FP8 (entry 6.0-0021), and an eight-card RTX PRO 6000 Server Edition system reached 6,238.12 in FP4 (entry 6.0-0004), both verified by MLCommons Association. Our division of each system result by its number of GPUs, which is not an MLPerf® metric, gives about 2,040 and 780 tokens per second per card. Thirty users reading at 10 tokens per second need 300 tokens per second in aggregate, with bursts above it. With the short prompts of the MLPerf® benchmark, one card of either kind carries that; the H200 NVL does it with the FP8 model and about 2.6 times the throughput. With fresh 8,192-token prompts, NVIDIA’s TensorRT-LLM performance overview for the RTX PRO 6000 Server Edition (version 1.3.0rc29, updated 26 September 2026), a maximum-throughput test without a latency limit, gives 296 output tokens per second per card in FP4, against 1,724 with 1,000-token prompts; the plan therefore relies on prefix caching, which lets each turn prefill only its new tokens.

MLPerf® Inference v6.0 Datacenter, Closed division, Available category, Llama 2 70B (llama2-70b-99), Interactive scenario: entries 6.0-0021 (Dell PowerEdge XE7740, eight H200 NVL, FP8) and 6.0-0004 (eight RTX PRO 6000 Blackwell Server Edition, FP4), retrieved from mlcommons.org on 24 September 2026. Results verified by MLCommons Association. The two results share version, division, category, scenario and GPU count and differ in precision. MLPerf® Inference v6.1 of 16 September 2026 has no interactive Llama 2 70B result for either card, so v6.0 remains the source. Per-card figures are our division of each system result by its number of GPUs, not an MLPerf® metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

If the same server also runs overnight batch work, the Blackwell card’s FP4 throughput counts: the same TensorRT-LLM overview shows 1,724 output tokens per second per RTX PRO 6000 on this model in FP4 with 1,000-token prompts, and 296 with 8,192-token prompts. Long prompts reduce throughput on any hardware.

Step 4: the server around the card

Vendor limits are from current product guides; the system-memory and storage rules are our sizing practice, not a vendor specification.

Two rows need a comment. The Lenovo SR650a V4 accepts four L40S at 350 W but only two H200 NVL at 600 W, so the same 2U chassis with four double-width slots holds only two of the cards this design needs. And the H200 NVL brings a five-year NVIDIA AI Enterprise subscription, which covers NIM microservices and NVIDIA support if the deployment uses them; the RTX PRO 6000 does not, and where the licence is needed it is a separate line on the quote.

Where the RTX PRO 6000 option needs an NVIDIA AI Enterprise licence, we put it on the same quote as the hardware. Ask us for a written quote on both options, and you can compare them in full.

What we would order

For thirty users, 8k context and an FP8 model as specified, we would order one H200 NVL in a 2U server with a second slot free, on a 16 A feed, in a hall at 25 °C or below. The model fits with about 26 GiB spare (about 16 GiB inside vLLM’s default share), the card carries the interactive load with margin, and the NVIDIA AI Enterprise licence comes with it.

If the evaluation set accepts NVFP4, one RTX PRO 6000 Server Edition runs the same job at the same power with almost no memory left. That suits a pilot but not a service that has to grow; for a service, plan two of them from the start, in FP8, with 75 GiB free. If the context grows to 32k for everyone, both plans change: the cache alone becomes 150 GiB, and the choice moves to four cards or to a smaller model. At 8k context and a hundred users or more, our inference node with four RTX PRO 6000 Server Edition cards is the next step. By our users-per-card rule (0.9 × 95.6 GiB, less 3 GiB per card, less the weights), two tensor-parallel FP8 pairs each hold about 78 conversations of 8,192 tokens each with an FP8 cache, about 156 in the server, and four single-card copies in NVFP4 about 136. These are memory figures, not measured throughput, so we would test two TP=2 instances against one TP=4 copy on your model. In a 2U chassis, check whether the maker allows four cards at 600 W or only at 450 W.

The context length and the number of concurrent users move the result most, and they are the two figures most requests leave out.

What we supply

We supply the H200 NVL and the RTX PRO 6000 Server Edition as cards or in a server built to order with the CPU, memory, storage, network and power from Step 4, on one EU contract and invoice with manufacturer warranty. Quantities and lead time are confirmed in writing with the quote.

FAQ