How many people one RTX PRO 6000 Blackwell can actually serve, and how to work it out yourself

How many users is not a headcount

The question usually arrives as a staff number: four hundred employees, one card, will it hold. A GPU has no concept of employees. It holds sequences, meaning requests resident in the KV cache with tokens being generated for them at this instant. A hundred people with an assistant open in a browser tab are a hundred idle sessions and some much smaller number of simultaneous generations. Nobody publishes a ratio between the two, so we will not invent one: log requests in flight at your gateway and take the busy-hour peak. That is the only user number that enters the arithmetic.

Now the uncomfortable part, said early. NVIDIA has published no absolute tokens-per-second LLM figure for any edition of this card. Its inference performance page does carry a section headed RTX PRO 6000 Blackwell Server Edition, but every model in it is vision or image generation, measured in images or samples per second. The only official LLM claims are relative: up to five times an L40S, 5.6 times on Llama 3 70B, neither with a precision, a batch size or a sequence length attached. MLPerf Inference v5.1 holds no MLCommons-verified submission for this GPU, and Cisco’s eight-card figures for that round are self-declared, published as charts only, and stated by Cisco not to have been verified by MLCommons.

So every confident claim that an RTX PRO 6000 does N tokens per second traces back to a rental provider’s blog, and we will not repeat one with a caveat attached. The absence is the finding, and the replacement is better: vLLM computes the concurrency it can sustain and prints it at startup.

INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x

The first line reports the tokens that fit in the KV cache at once; the second, how many requests can be served concurrently if each requires the stated number of tokens, and it is simply the first divided by the tokens-per-request figure: 643,232 / 40,960 = 15.70. Read it on your card and you need nobody’s benchmark.

The memory budget: 96 GB minus the weights

Two things consume the 96 GB: the weights, fixed once the model and quantisation are chosen, and the KV cache, which grows with every user admitted, plus a few GB for activations, CUDA graphs and framework overhead. Because the weights do not move, the KV cache is what caps concurrency.

Weights are parameters multiplied by bytes per parameter: two for BF16, one for FP8, half a byte for NVFP4. Llama 3.1 8B is 16.1, 8.0 and 4.0 GB; Qwen3-32B 65.6, 32.8 and 16.4 GB; Llama 3.1 70B 141.2 GB in BF16, which does not fit, 70.6 at FP8 and 35.3 at NVFP4. NVIDIA’s NIM support matrix confirms the boundary: 70B runs at TP1 in FP8 and NVFP4, BF16 only at TP4 and TP8, and “BF16 TP2 with or without LoRA is not supported on NVIDIA RTX PRO 6000 Blackwell Server Edition”.

The KV cache is where sizing is won or lost:

KV bytes = 2 × layers × kv_heads × head_dim × bytes_per_element × seq_len × batch

The leading 2 is the key and the value; kv_heads is the number of key/value heads, not query heads; head_dim the per-head dimension; bytes_per_element 2 for an FP16 or BF16 cache and 1 for an FP8 one; seq_len the tokens held for that request; batch the concurrent sequences resident.

Here is the trap. NVIDIA’s inference optimisation blog writes the formula as “2 * (num_layers) * (num_heads * dim_head) * precision_in_bytes”. Read num_heads as the attention head count and you are wrong by the whole grouped-query attention ratio. Llama 3.1 70B has 64 query heads and 8 key/value heads: substituting 64 gives 2,560 KiB per token against the correct 320 KiB, an eightfold overestimate of the largest variable in the budget. vLLM names it num_kv_heads; in config.json, num_key_value_heads.

Head dimension is 128 in all three. Llama figures from Meta’s Llama 3 paper, Table 3; Qwen3-32B from its own config.json. KV per token is for an FP16 or BF16 cache; an FP8 cache halves every figure.

Grouped-query attention is why single-card serving works at all: without it a 70B model costs 20 GiB per 8K session instead of 2.5. The budget is 96 GiB times vLLM’s default gpu_memory_utilization of 0.9, so 86.4 GiB, less about 3 GiB of overhead, less the weights.

Budget: 96 GiB × 0.9, less about 3 GiB of overhead, less the weights. Sessions are concurrent requests each holding a full 8K context with an FP16 KV cache; the second figure is the same calculation with an FP8 KV cache.

Follow one row. Llama 3.1 70B at FP8 is 65.8 GiB of weights; 86.4 minus 65.8 minus 3 leaves 17.6 GiB; each 8K session at 320 KiB per token costs 2.5 GiB; 17.6 divided by 2.5 is 7. Seven concurrent full-length conversations from one card, fourteen with an FP8 cache, twenty with NVFP4 weights. Weight quantisation is the biggest single lever on concurrency.

Context length is the second lever and it is brutal. At a declared 32K the same arithmetic leaves 17 sessions for the 8B model in BF16 and 1 for the 70B in FP8; one 70B session at 128K costs 40 GiB of FP16 KV, more than that whole budget.

An FP8 KV cache, vLLM’s kv_cache_dtype="fp8_e4m3", halves every KV figure. And read the counts as a floor, not a ceiling: PagedAttention allocates blocks on demand, so sessions that never fill their declared context cost less.

Bandwidth sets the ceiling, and which bandwidth matters

Capacity decides how many users fit; bandwidth decides how fast each is served. Decoding re-reads the weights from memory at every token, so at low batch sizes the per-user rate is bounded by bandwidth divided by bytes of weights, not by any FLOPS figure on the datasheet. All three editions share the GB202 die, 96 GB of GDDR7 with ECC and 24,064 CUDA cores. They do not share a memory clock.

NVIDIA datasheets and product pages. Lenovo’s product guide quotes “Up to 1597 GB/s” and documents an optional 450 W power cap on the Server Edition; NVIDIA’s reference architecture says up to 1.6 TB/s per GPU.

The Server Edition is the one that goes in a rack, and it is the slower-memory one. Most specification tables under this product name quote 1,792 GB/s, the Workstation and Max-Q figure, about 11 per cent higher; carried into a server sizing calculation it overstates decode throughput by roughly 12 per cent.

The ceiling that follows is easy to compute. Llama 3.1 70B at FP8 is 70.6 GB of weights, and 1,597 divided by 70.6 is about 22 decode steps per second for a single unbatched user, before any efficiency loss. At NVFP4 the weights halve and the ceiling roughly doubles to about 45. That is arithmetic from the published bandwidth and the weight size, not a measurement, and no real system reaches it. Run the same division for your model.

How fast does it need to be? Brysbaert’s meta-analysis of 190 studies puts mean silent reading of non-fiction at 238 words per minute; at the usual 0.75 words per token, about 5.3 tokens per second, 6.7 for a fast reader. The industry ships against roughly 50 tokens per second per user and a time to first token at or below 250 ms, because users skim and abandon, and reasoning models emit thinking tokens nobody reads.

Batching: total throughput against per-user latency

Between the two ceilings sits the scheduler, and the lever is batch size. NVIDIA’s own description: at low concurrency latency is low and so is throughput; at high concurrency batching serves more requests efficiently and costs latency. The configurations where no metric improves without spoiling another are the Pareto front.

Before reading any throughput figure, separate two quantities that are routinely printed as one. Throughput per system is total output tokens per second across all requests running simultaneously; throughput per user is output length divided by end-to-end latency. NVIDIA describes a B200 as both a 60,000 tokens per second machine and a 1,000 tokens per second machine, in two configurations of the same silicon. A figure that does not state aggregate-or-per-user, the concurrency, the sequence lengths and the precision says nothing.

Nor does batching trade monotonically. NVIDIA’s own TensorRT-LLM tuning guide publishes the sweep.

NVIDIA’s TensorRT-LLM performance tuning guide, Llama 3.3 70B on four H100. Different hardware to the card discussed here; what transfers is the shape of the curve, not the absolute values.

Raising the ceiling from 64 to 512 bought 27 per cent more throughput at essentially no latency cost. Raising it from 512 to 2,048 made throughput worse, by about 17 per cent, and returned nothing in latency. Past the knee you lose on both axes, and the default max_batch_size in that stack is 2,048: an argument for sweeping it on your own model, not an argument against the default.

NVIDIA’s NIM reference architecture does publish concurrency, but on an H100 NVL: 137 users on Llama 3.1 8B at FP8, 278 on a mixed workload, near 1,000 ms to first token. That card has HBM3 at far higher bandwidth than GDDR7, so it is a shape, not a substitute number.

Slicing one card: MIG, vGPU and the licence

MIG works on all three editions: NVIDIA’s MIG user guide covers Workstation, Max-Q and Server Editions in one profile table, offering 1g.24gb with four instances, 2g.48gb with two and 4g.96gb with one. Note what is missing: there is no 3g profile. The granularity is 24, 48 or 96 GB and nothing between, so a 32 GB model cannot take a 24 GB slice and has to occupy a 48 GB one.

vGPU is stricter. The only RTX PRO 6000 entries on NVIDIA’s supported-GPU list are the Blackwell Server Edition, from vGPU 19.0, and its liquid-cooled variant, from 20.0; the Workstation and Max-Q editions appear nowhere in it, so if the plan involves virtual machines the Server Edition is the only option. For compute the relevant profiles are the C-series: one DC-96C per card, two DC-48C, three DC-32C, four DC-24C, six DC-16C, eight DC-12C, or twelve DC-8C at 8 GB each. The MIG-backed variants, which isolate rather than time-slice, top out at four.

One number needs killing before it reaches a design document. NVIDIA’s vGPU 19.0 blog says these GPUs support as many as 48 concurrent virtual machines on one GPU, then explains the arithmetic itself: four MIG instances multiplied by twelve VMs each. That is virtual-desktop density from small graphics profiles, not LLM serving, where the ceiling is twelve. Our guide on MIG and vGPU counts per card works the same distinction across the range. Partitioning never creates capacity: four slices hold four small models, not more users of one large one.

Licensing is counted two ways and they get mixed up constantly. NVIDIA AI Enterprise, which the C-series compute profiles require, is licensed per GPU: one for every GPU in the server that hosts the software. vApps, vPC and RTX vWS are sold per concurrent user, one CCU licence per concurrent virtual machine. So a card serving a model to 200 people over an API needs one AI Enterprise licence, because it is one GPU; concurrent-user counting applies to virtual desktops, not to HTTP clients of an inference endpoint.

When one card stops being enough

Four things force a second card; only three are about capacity.

1. The model does not fit. Llama 3.1 70B in BF16 is 141.2 GB, and NVIDIA’s NIM matrix lists it only at TP4 and TP8 while explicitly refusing BF16 TP2 on this GPU.
2. The context is long. One 70B session at 128K costs 40 GiB of FP16 KV; two of those plus FP8 weights are over the card.
3. Concurrency exceeds the KV budget. That is the arithmetic above, and quantisation is the first move against it.
4. Availability. One card is a single point of failure: a driver update, an ECC fault or a hung process takes the service down, and MIG and vGPU share its failure domain entirely. A second card, ideally in a second host, is the only answer, and it stands even when the arithmetic says one card is plenty.

What makes the second card different here is that there is no NVLink on any edition. Lenovo’s product guide states it as a flat No; NVIDIA’s specification table and architecture whitepaper omit it, and the reference architecture describes GPU placement purely in PCIe terms.

NVIDIA’s own H200 page prints both figures side by side: NVLink at 900 GB/s per GPU against PCIe Gen 5 at 128 GB/s, roughly seven times. That is the tax on every inter-GPU exchange, and why the parallelism choice is not a matter of taste. vLLM’s documentation is direct: if the GPUs have no NVLink interconnect, use pipeline parallelism instead of tensor parallelism, for higher throughput and lower communication overhead. Tensor parallelism does an all-reduce at every layer; pipeline parallelism passes activations once per stage boundary.

So the ladder runs in order. Quantise before you scale out: FP8 or NVFP4 on one card beats BF16 split across two cards with no fast link. For more aggregate throughput, run independent replicas, one instance per card behind a load balancer, which removes inter-GPU traffic entirely and doubles availability. Only if the model genuinely will not fit should you split it, and then prefer pipeline parallel. Tensor parallelism stays supported, with TP2, TP4 and TP8 profiles in the NIM matrix; it simply pays a PCIe tax. Nobody publishes the size of that tax on this card: the direction is documented, the magnitude is not.

A sizing procedure you can run this week

None of this needs a laboratory: an afternoon and one card produce a number you can defend because you measured it.

1. Count concurrency, not people. Log requests in flight at the gateway for a week; take the peak and the 95th percentile.
2. Decide the context you will declare, which is not the model’s maximum: 8K instead of an inherited 40K multiplies concurrency by five.
3. Compute the weights: parameters × bytes per parameter. Decide the quantisation here: 141.2, 70.6 and 35.3 GB are the same 70B model.
4. Compute KV per token: 2 × layers × kv_heads × head_dim × bytes, taking kv_heads from num_key_value_heads and never from num_attention_heads.
5. Subtract and divide. 96 GiB × 0.9, less about 3 GiB, less the weights, divided by per-session KV. That is your floor.
6. Start the server and read the log. Launch vLLM with that model, quantisation and declared context and read the two lines. If they disagree badly with step five, step five is wrong.
7. Measure latency at that load: time to first token and inter-token latency at realistic sequence lengths, against 250 ms and a per-user rate you will defend. Sweep the batch ceiling around the knee instead of accepting a default.
8. Decide availability separately. If the service must survive a driver update, the answer is two cards in two hosts whatever the arithmetic said.

The short version: an 8B-class assistant on one card serves tens of concurrent sessions at long context and well into the hundreds at realistic ones; a 32B model tens; a 70B model in FP8 single figures, which NVFP4 roughly triples. Eurokommerz supplies RTX PRO 6000 Blackwell cards and the servers they go in across the EU: send us the model, the declared context and the peak concurrency from your logs, and we will return the memory worksheet.

FAQ