How many people one RTX PRO 6000 Blackwell can actually serve, and how to work it out yourself

How many users is not a headcount

The question usually arrives as a staff number: four hundred employees, one card, will it hold. A GPU has no concept of employees. It holds sequences, meaning requests resident in the KV cache with tokens being generated for them at this instant. A hundred people with an assistant open in a browser tab are a hundred idle sessions and some much smaller number of simultaneous generations. We found no published ratio between the two, so we will not invent one: log requests in flight at your gateway and take the busy-hour peak. That is the only user number that enters the arithmetic.

Now the uncomfortable part, said early. NVIDIA’s absolute LLM figures for the card are maximum-throughput numbers for the Server Edition in its TensorRT-LLM performance overview, measured with its own engine: Qwen3-30B-A3B in FP4 on one GPU, for example, at 9,938 output tokens per second per GPU with 1,000-token input and output. We found no per-user figure from NVIDIA. Its inference performance page does carry a section headed RTX PRO 6000 Blackwell Server Edition, but every model in it is vision, image or video generation, measured in images or samples per second or videos per minute. NVIDIA’s other LLM claims are relative, such as up to five times an L40S, with no precision, batch size or sequence length attached. MLPerf Inference v6.0 of April 2026 holds MLCommons-verified results for this GPU, but they are throughputs of whole multi-GPU servers under the benchmark’s latency limits: on Llama 2 70B in the interactive scenario, which allows 450 ms to the first token and 40 ms per token after it, one eight-card Server Edition system reached 6,238.12 tokens per second with FP4 weights (entry 6.0-0004).

MLPerf Inference v6.0 Datacenter, Closed division, Available category, Llama 2 70B (llama2-70b-99), Interactive scenario, entry 6.0-0004, retrieved from mlcommons.org on 24 September 2026. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

So a per-user tokens-per-second figure for this card has no NVIDIA source, and a maximum-throughput number measured with NVIDIA’s own engine is not one. The absence is the finding, and the replacement is better: vLLM computes the concurrency it can sustain and prints it at startup.

INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x

The first value is the tokens that fit in the KV cache at once; the second, how many requests can be served concurrently if each requires the stated number of tokens, and for a model with full attention in every layer it is simply the first divided by the tokens-per-request figure: 643,232 / 40,960 = 15.70. Recent releases, such as vLLM 0.30 of September 2026, print both on one line. Read it on your card and you need nobody’s benchmark.

The memory budget: 96 GB minus the weights

Two things consume the 96 GB: the weights, fixed once the model and quantisation are chosen, and the KV cache, which grows with every user admitted, plus a few GB for activations, CUDA graphs and framework overhead. Because the weights do not move, the KV cache is what caps concurrency.

Weights are parameters multiplied by bytes per parameter: two for BF16, one for FP8, and for NVFP4 half a byte plus one FP8 scale per 16 weights, with the embeddings and output layer usually kept in BF16 in both FP8 and NVFP4 checkpoints. NVIDIA’s NVFP4 checkpoint of Llama 3.3 70B, the same architecture as 3.1 70B, is 42.7 GB, about 0.61 bytes per parameter. Llama 3.1 8B is 16.1, 9.1 (NVIDIA’s FP8 checkpoint) and, by our arithmetic, about 6.0 GB; Qwen3-32B 65.6, 34.3 (Qwen’s FP8 checkpoint) and 20.7 GB (NVIDIA’s NVFP4 checkpoint); Llama 3.1 70B 141.2 GB in BF16, which does not fit, about 72.7 at FP8 and about 42.7 at NVFP4 (NVIDIA’s Llama 3.3 70B checkpoints). NVIDIA’s NIM support matrix confirms the boundary: 70B runs at TP1 in FP8 and NVFP4, BF16 only at TP4 and TP8, and “BF16 TP2 with or without LoRA is not supported on NVIDIA RTX PRO 6000 Blackwell Server Edition”.

The KV cache is where sizing is won or lost:

KV bytes = 2 × layers × kv_heads × head_dim × bytes_per_element × seq_len × batch

The leading 2 is the key and the value; kv_heads is the number of key/value heads, not query heads; head_dim the per-head dimension; bytes_per_element 2 for an FP16 or BF16 cache and 1 for an FP8 one; seq_len the tokens held for that request; batch the concurrent sequences resident.

Here is the trap. NVIDIA’s inference optimisation blog writes the formula as “2 * (num_layers) * (num_heads * dim_head) * precision_in_bytes”. Read num_heads as the attention head count and you are wrong by the whole grouped-query attention ratio. Llama 3.1 70B has 64 query heads and 8 key/value heads: substituting 64 gives 2,560 KiB per token against the correct 320 KiB, an eightfold overestimate of the largest variable in the budget. vLLM names it num_kv_heads; in config.json, num_key_value_heads.

Head dimension is 128 in all three. Llama figures from Meta’s Llama 3 paper, Table 3; Qwen3-32B from its own config.json. KV per token is for an FP16 or BF16 cache; an FP8 cache halves every figure.

Grouped-query attention is why single-card serving works at all: without it a 70B model costs 20 GiB per 8K session instead of 2.5. The budget is the 95.6 GiB the driver reports times a gpu_memory_utilization of 0.9 (vLLM’s default up to release 0.19; 0.92 since 0.20.0 of April 2026), the figure we plan with, so 86.0 GiB, less about 3 GiB of overhead, less the weights.

Budget: 95.6 GiB × 0.9, less about 3 GiB of overhead, less the weights of the published checkpoints, which keep the embeddings and output layer in BF16. Sessions are concurrent requests each holding a full 8K context with an FP16 KV cache; the second figure is the same calculation with an FP8 KV cache.

Follow one row. Llama 3.1 70B at FP8 is 67.7 GiB of weights in NVIDIA’s checkpoint; 86.0 minus 67.7 minus 3 leaves 15.3 GiB; each 8K session at 320 KiB per token costs 2.5 GiB; 15.3 divided by 2.5 is 6. Six concurrent full-length conversations from one card, twelve with an FP8 cache, seventeen with NVFP4 weights. Weight quantisation is the biggest single lever on concurrency.

Context length is the second lever and it is brutal. At a declared 32K the same arithmetic leaves 17 sessions for the 8B model in BF16 and 1 for the 70B in FP8; one 70B session at 128K costs 40 GiB of FP16 KV, more than that whole budget.

An FP8 KV cache, vLLM’s kv_cache_dtype="fp8_e4m3", halves every KV figure. And read the counts as a floor, not a ceiling: PagedAttention allocates blocks on demand, so sessions that never fill their declared context cost less.

Bandwidth sets the ceiling, and which bandwidth matters

Capacity decides how many users fit; bandwidth decides how fast each is served. Decoding re-reads the weights from memory at every token, so at low batch sizes the per-user rate is bounded by bandwidth divided by bytes of weights, not by any FLOPS figure on the datasheet. All three editions share the GB202 die, 96 GB of GDDR7 with ECC and 24,064 CUDA cores. They do not share a memory clock.

NVIDIA datasheets and product pages. Lenovo’s product guide quotes “Up to 1597 GB/s” and documents an optional 450 W power cap on the Server Edition; NVIDIA’s reference architecture says up to 1.6 TB/s per GPU.

The Server Edition is the one that goes in a rack, and it is the slower-memory one. Most specification tables under this product name quote 1,792 GB/s, the Workstation and Max-Q figure, about 11 per cent higher; carried into a server sizing calculation it overstates decode throughput by roughly 12 per cent.

The ceiling that follows is easy to compute. Llama 3.1 70B at FP8 reads about 70.6 GB of weights per token, and 1,597 divided by 70.6 is about 23 decode steps per second for a single unbatched user, before any efficiency loss. At NVFP4 each token reads about 40.6 GB, the 4-bit layers plus the output layer in BF16 (the embedding table is only looked up), and the ceiling rises to about 39. That is arithmetic from the published bandwidth and the weight size, not a measurement, and no real system reaches it. Run the same division for your model.

How fast does it need to be? A 2019 meta-analysis of 190 reading studies in the Journal of Memory and Language puts mean silent reading of non-fiction at 238 words per minute; at the usual 0.75 words per token, about 5.3 tokens per second, 6.7 for a fast reader. Interactive services are built well above that, because users skim and abandon, and reasoning models emit thinking tokens nobody reads: the interactive scenario of MLPerf Inference v6.0 for Llama 2 70B, for example, requires the first token within 450 ms and at most 40 ms per token after it, 25 tokens per second per user.

Batching: total throughput against per-user latency

Between the two ceilings sits the scheduler, and the lever is batch size. The configurations where no metric improves without spoiling another are the Pareto front.

Before reading any throughput figure, separate two quantities that are routinely printed as one. Throughput per system is total output tokens per second across all requests running simultaneously; throughput per user is output length divided by end-to-end latency. NVIDIA describes one DGX B200 server with eight Blackwell GPUs both as a machine delivering over 1,000 tokens per second to each user and as a 72,000 tokens per second machine on Llama 4 Maverick, in two configurations of the same silicon. A figure that does not state aggregate-or-per-user, the concurrency, the sequence lengths and the precision says nothing.

Nor does batching trade monotonically. NVIDIA’s own TensorRT-LLM tuning guide publishes the sweep.

NVIDIA’s TensorRT-LLM performance tuning guide, Llama 3.3 70B on four H100. Different hardware to the card discussed here; what transfers is the shape of the curve, not the absolute values.

Raising the ceiling from 64 to 512 bought 27 per cent more throughput at essentially no latency cost. Raising it from 512 to 2,048 made throughput worse, by about 17 per cent, and returned nothing in latency. Past the knee you lose on both axes, and the default max_batch_size in that stack is 2,048: an argument for sweeping it on your own model, not an argument against the default.

Slicing one card: MIG, vGPU and the licence

MIG works on all three editions: NVIDIA’s MIG user guide covers Workstation, Max-Q and Server Editions in one profile table, offering 1g.24gb with four instances, 2g.48gb with two and 4g.96gb with one. Note what is missing: there is no 3g profile. The granularity is 24, 48 or 96 GB and nothing between, so a 32 GB model cannot take a 24 GB slice and has to occupy a 48 GB one.

vGPU is stricter. The only RTX PRO 6000 entries on NVIDIA’s supported-GPU list are the Blackwell Server Edition, from vGPU 19.0, and its liquid-cooled variant, from 20.0; the Workstation and Max-Q editions appear nowhere in it, so if the plan involves virtual machines the Server Edition is the only option. For compute the relevant profiles are the C-series: one DC-96C per card, two DC-48C, three DC-32C, four DC-24C, six DC-16C, eight DC-12C, or twelve DC-8C at 8 GB each. The MIG-backed variants, which isolate rather than time-slice, top out at four.

One number needs killing before it reaches a design document. NVIDIA’s vGPU 19.0 blog says these GPUs support as many as 48 concurrent virtual machines on one GPU, then explains the arithmetic itself: four MIG instances multiplied by twelve VMs each. That is virtual-desktop density from small graphics profiles, not LLM serving, where the ceiling is twelve. Our guide on MIG and vGPU counts per card works the same distinction across the range. Partitioning never creates capacity: four slices hold four small models, not more users of one large one.

Licensing is counted two ways and they get mixed up constantly. NVIDIA AI Enterprise, which the C-series compute profiles require, is licensed per GPU: one for every GPU in the server that hosts the software. vApps, vPC and RTX vWS are sold per concurrent user, one CCU licence per concurrent virtual machine. So a card serving a model to 200 people over an API needs one AI Enterprise licence if it runs NVIDIA AI Enterprise software such as NIM, because it is one GPU, and none if it runs an open-source engine such as vLLM on bare metal; concurrent-user counting applies to virtual desktops, not to HTTP clients of an inference endpoint.

When one card stops being enough

Four things force a second card; only three are about capacity.

1. The model does not fit. Llama 3.1 70B in BF16 is 141.2 GB, and NVIDIA’s NIM matrix lists it only at TP4 and TP8 while explicitly refusing BF16 TP2 on this GPU.
2. The context is long. One 70B session at 128K costs 40 GiB of FP16 KV; two of those plus FP8 weights are over the card.
3. Concurrency exceeds the KV budget. That is the arithmetic above, and quantisation is the first move against it.
4. Availability. One card is a single point of failure: a driver update, an ECC fault or a hung process takes the service down, and MIG and vGPU share its failure domain entirely. A second card, ideally in a second host, is the only answer, and it stands even when the arithmetic says one card is plenty.

What makes the second card different here is that there is no NVLink on any edition. Lenovo’s product guide states it as a flat No; NVIDIA’s specification table and architecture whitepaper omit it, and the reference architecture describes GPU placement purely in PCIe terms.

NVIDIA’s own H200 page prints both figures side by side: NVLink at 900 GB/s per GPU against PCIe Gen 5 at 128 GB/s, roughly seven times. That is the tax on every inter-GPU exchange, and why the parallelism choice is not a matter of taste. vLLM’s documentation is direct: if the GPUs have no NVLink interconnect, use pipeline parallelism instead of tensor parallelism, for higher throughput and lower communication overhead. Tensor parallelism does two all-reduces in every layer; pipeline parallelism passes activations once per stage boundary.

So the ladder runs in order. Quantise before you scale out: FP8 or NVFP4 on one card beats BF16 split across two cards with no fast link. For more aggregate throughput, run independent replicas, one instance per card behind a load balancer, which removes inter-GPU traffic entirely and doubles availability. Only if the model genuinely will not fit should you split it, and then prefer pipeline parallel. Tensor parallelism stays supported, with TP2, TP4 and TP8 profiles in the NIM matrix; it simply pays a PCIe tax. The direction is documented; published measurements of its size on this card are few, and our PCIe and NVLink article collects them.

A sizing procedure you can run this week

None of this needs a laboratory: an afternoon and one card produce a number you can defend because you measured it.

1. Count concurrency, not people. Log requests in flight at the gateway for a week; take the peak and the 95th percentile.
2. Decide the context you will declare, which is not the model’s maximum: 8K instead of an inherited 40K multiplies concurrency by five.
3. Compute the weights: parameters × bytes per parameter, or simply the size of the checkpoint you will load. Decide the quantisation here: 141.2, about 72.7 and about 42.7 GB are the same 70B model.
4. Compute KV per token: 2 × layers × kv_heads × head_dim × bytes, taking kv_heads from num_key_value_heads and never from num_attention_heads.
5. Subtract and divide. 95.6 GiB × 0.9, less about 3 GiB, less the weights, divided by per-session KV. That is your floor.
6. Start the server and read the log. Launch vLLM with that model, quantisation and declared context and read the GPU KV cache size and Maximum concurrency values. If they disagree badly with step five, step five is wrong.
7. Measure latency at that load: time to first token and inter-token latency at realistic sequence lengths, against a first-token target and a per-user rate you will defend. Sweep the batch ceiling around the knee instead of accepting a default.
8. Decide availability separately. If the service must survive a driver update, the answer is two cards in two hosts whatever the arithmetic said.

The short version: an 8B-class assistant on one card serves tens of concurrent sessions at long context and well into the hundreds at realistic ones; a 32B model tens; a 70B model in FP8 single figures, which NVFP4 more than doubles. Eurokommerz supplies RTX PRO 6000 Blackwell cards and the servers they go in across the EU: send us the model, the declared context and the peak concurrency from your logs, and we will return the memory worksheet.

FAQ