Sizing a GPU server for a 70B model: a worked example, from weights to wall socket

The request

A manufacturing company wants an internal assistant on Llama 3.3 70B: document questions, drafting, summaries. Thirty people may be using it at once, conversations run to about 8,000 tokens of context, and replies have to stream at reading speed. Nothing leaves the building. This is the request we see most often, so here is the sizing worksheet in the order our engineering partner Vixen.UNO works through it.

Step 1: the weights

Llama 3.3 70B has 70.6 billion parameters. Bytes per parameter set the footprint: two in FP16, one in FP8, and 0.5625 in NVFP4, because every sixteen values share an FP8 scale.

Sources: Llama 3 paper and model cards, NVIDIA checkpoint cards on Hugging Face, NVIDIA NVFP4 technical blog.

FP16 is 132 GiB: the weights alone fill an H200 NVL with about 9 GiB left, which is no room for thirty users, and they do not fit a 96 GB card at all; it is a two- or four-card job or a different model. FP8 and NVFP4 are the real candidates, and the choice between them belongs to whoever owns the evaluation set: two points on MMLU is nothing for a summariser and everything for a compliance tool.

Step 2: the KV cache

Every token in a live conversation keeps its keys and values in GPU memory for the whole session. The size per token is 2 × layers × KV heads × head dimension × bytes. For Llama 3 70B that is 2 × 80 × 8 × 128 = 163,840 values: 320 KiB per token in FP16, 160 KiB in FP8. Modern servers keep the cache in FP8.

Thirty users at 8,192 tokens each: 30 × 8,192 × 160 KiB = 37.5 GiB. Add about 10 % for activations, the CUDA context and fragmentation, and the memory plan looks like this.

TP = tensor-parallel across two cards. Totals include about 8 GiB of overhead (10 GiB across two cards) for activations, the CUDA context and fragmentation. Capacities are what the driver reports: nvidia-smi shows 97,887 MiB (95.6 GiB) for the RTX PRO 6000 and 143,771 MiB (140.4 GiB) for the H200 NVL.

This is the table that answers most sizing calls. The 96 GB card is a one-card solution for this workload only in NVFP4, and at 90 % utilisation there is little room for a longer context or five more users; it is a pilot configuration, not a production endpoint. The H200 NVL takes the FP8 model with about 26 GiB free. Two Blackwell cards take it in FP8 with 75 GiB of headroom, at the cost of tensor-parallel traffic over PCIe, since these cards have no NVLink.

Step 3: is it fast enough

Two speeds matter. The first is what one person sees: single-stream generation is bound by memory bandwidth, and the ceiling is bandwidth divided by the weight bytes read per token. For the FP8 model that is about 68 tokens per second on the H200 NVL (4,800 GB/s) and about 23 on an RTX PRO 6000 (1,597 GB/s); for NVFP4 on the Blackwell card about 40. Reading speed is under 10 tokens per second, so all three pass for one user.

The second is the aggregate under load with a latency promise. MLPerf Inference v6.0’s interactive scenario for Llama 2 70B caps time-to-first-token at 450 ms and time-per-token at 40 ms. Eight H200 NVL in a Dell XE7740 reached 16,344 tokens per second, about 2,040 per card; an eight-card RTX PRO 6000 system reached 6,238, about 780 per card (both are eight-GPU system results divided by eight). Thirty users reading at 10 tokens per second need 300 tokens per second in aggregate with bursts above it. One card of either kind carries that; the H200 NVL carries it with the FP8 model and two and a half times the margin.

If the same server also runs overnight batch work, the Blackwell card’s FP4 throughput counts: NVIDIA’s TensorRT-LLM table shows 1,724 output tokens per second per RTX PRO 6000 on this model in FP4 with 1,000-token prompts, and 296 with 8,192-token prompts. Long prompts are expensive everywhere.

Step 4: the server around the card

Vendor limits are from current product guides; the system-memory and storage rules are our engineering practice, not a vendor specification.

Two lines deserve a comment. The Lenovo SR650a V4 accepts four L40S at 350 W but only two H200 NVL at 600 W, so the same 2U that promises four slots holds two of the cards this design wants. And the H200 NVL brings a five-year NVIDIA AI Enterprise subscription, which this deployment will use for the inference stack; the RTX PRO 6000 does not, and that licence is a separate line on the quote.

What we would order

For thirty users, 8k context and an FP8 model as specified: one H200 NVL in a 2U with a second slot free, on a 16 A feed, in a hall at 25 °C or below. It fits with about 26 GiB spare, it carries the interactive load with margin, and the licence comes with it.

If the evaluation set accepts NVFP4 and the budget is one Blackwell card: one RTX PRO 6000 Server Edition runs the same job at the same power with almost no memory left, which is fine for a pilot and wrong for a service that has to grow; the honest plan is two of them from the start, in FP8, with 75 GiB free. If the context grows to 32k for everyone, both plans change, because the cache alone becomes 150 GiB and the conversation moves to four cards or to a smaller model.

That last sentence is the point of the worksheet. The card is never the first decision; the context length and the number of people are, and they are the two numbers most requests arrive without.

FAQ