How many users one RTX PRO 6000 Blackwell can serve, and how to work it out

Users per GPU means concurrent requests

By our users-per-card rule, one RTX PRO 6000 holds 6 concurrent full-length sessions of Llama 3.3 70B in FP8 at a declared 8K context, 25 of Qwen3-32B in FP8 and 68 of an 8B model in BF16, each with a 16-bit KV cache; an FP8 KV cache about doubles each figure, to 12 for the 70B model. vLLM prints the figure for your own model when it starts. The question is usually asked as a staff number, for example whether one card is enough for four hundred employees. A GPU serves sequences: requests resident in the KV cache, with tokens being generated for them at that moment. A hundred people with an assistant open in a browser tab are a hundred idle sessions and a much smaller number of simultaneous generations. We found no published ratio between the two. Log the requests in flight at your gateway and take the busy-hour peak; that peak is the user number for the sizing.

The published figures from NVIDIA and MLPerf® do not answer the question either. NVIDIA’s absolute LLM figures for the card are maximum-throughput numbers for the Server Edition in its TensorRT-LLM performance overview, measured with its own engine: Qwen3-30B-A3B in FP4 on one GPU, for example, at 9,938 output tokens per second per GPU with 1,000-token input and output. We found no per-user figure from NVIDIA. Its inference performance page does carry a section headed RTX PRO 6000 Blackwell Server Edition, but every model in it is vision, image or video generation, measured in images or samples per second or videos per minute. NVIDIA’s other LLM claims are relative, such as up to five times an L40S, with no precision, batch size or sequence length attached. MLPerf® Inference v6.0 of April 2026 holds MLCommons-verified results for this GPU, but they are throughputs of whole multi-GPU servers under the benchmark’s latency limits. The newer round, v6.1 of 16 September 2026, has no interactive Llama 2 70B result for the card, so the interactive figure here stays with v6.0. On Llama 2 70B in the interactive scenario, which allows 450 ms to the first token and 40 ms per output token at the 99th percentile, one eight-card Server Edition system reached 6,238.12 tokens per second with FP4 weights (entry 6.0-0004).

MLPerf® Inference v6.0 Datacenter, Closed division, Available category, Llama 2 70B (llama2-70b-99), Interactive scenario, entry 6.0-0004, retrieved from mlcommons.org on 24 September 2026. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

vLLM gives a better basis, because it computes the concurrency it can sustain on your card and prints it at startup. The two lines look like this; the values illustrate the format only and match none of the configurations sized below:

INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x

The first value is the number of tokens that fit in the KV cache at once. The second is how many requests can be served concurrently if each requires the stated number of tokens; for a model with full attention in every layer it is the first divided by the tokens-per-request figure: 643,232 / 40,960 = 15.70. Our guide to the vLLM KV cache size and Maximum concurrency lines explains how vLLM computes them and which flags move them. Read on your own card, these two values replace someone else’s benchmark for sizing.

The memory budget: 96 GB minus the weights

The 96 GB holds the weights, which are fixed once the model and quantisation are chosen, and the KV cache, which grows with every user admitted, plus a few GB for activations, CUDA graphs and framework overhead. Because the weights are fixed, the KV cache is what limits concurrency.

Weights are parameters multiplied by bytes per parameter: two for BF16, one for FP8, and for NVFP4 half a byte plus one FP8 scale per 16 weights, with the embeddings and output layer usually kept in BF16 in both FP8 and NVFP4 checkpoints. NVIDIA’s NVFP4 checkpoint of Llama 3.3 70B, the same architecture as 3.1 70B, is 42.7 GB, about 0.61 bytes per parameter. In BF16, FP8 and NVFP4, Llama 3.1 8B is 16.1, 9.1 (NVIDIA’s FP8 checkpoint) and, by our arithmetic, about 6.0 GB. Qwen3-32B is 65.6, 34.3 (Qwen’s FP8 checkpoint) and 20.7 GB (NVIDIA’s NVFP4 checkpoint). Llama 3.1 70B is 141.2 GB in BF16, which does not fit, about 72.7 at FP8 and about 42.7 at NVFP4 (NVIDIA’s Llama 3.3 70B checkpoints). NVIDIA’s NIM support matrix, as of October 2026, lists the RTX PRO 6000 Server Edition among the verified GPUs for Llama 3.3 70B, with BF16, FP8 and NVFP4 profiles from TP1 to TP8 that it does not tie to a GPU; TP is the number of cards the model is split across. By the weights alone, FP8 and NVFP4 fit one card and BF16 needs two.

The KV cache decides how many users fit; its size is:

KV bytes = 2 × layers × kv_heads × head_dim × bytes_per_element × seq_len × batch

The leading 2 is the key and the value; kv_heads is the number of key/value heads, not query heads; head_dim the per-head dimension; bytes_per_element 2 for an FP16 or BF16 cache and 1 for an FP8 one; seq_len the tokens held for that request; batch the concurrent sequences resident.

The head count in this formula is easy to get wrong. NVIDIA’s inference optimisation blog writes the formula as “2 * (num_layers) * (num_heads * dim_head) * precision_in_bytes”. Reading num_heads as the attention head count overstates the result by the whole grouped-query attention ratio. Llama 3.1 70B has 64 query heads and 8 key/value heads: substituting 64 gives 2,560 KiB per token against the correct 320 KiB, an eightfold overestimate of the largest variable in the budget. vLLM names it num_kv_heads; in config.json, num_key_value_heads.

Head dimension is 128 in all three. Llama figures from Meta’s Llama 3 paper, Table 3; Qwen3-32B from its own config.json. KV per token is for an FP16 or BF16 cache; an FP8 cache halves every figure.

Grouped-query attention makes single-card serving practical: without it a 70B model needs 20 GiB per 8K session instead of 2.5. The budget starts from the 95.6 GiB the driver reports (nvidia-smi, FB Memory Total); NVIDIA’s vPC sizing guide, updated on 19 August 2026, states that the ECC of Blackwell-generation GPUs is built into GDDR7 memory and always enabled without reducing usable VRAM. We plan with a gpu_memory_utilization of 0.9, vLLM’s default up to release 0.19 (0.92 since 0.20.0 of April 2026), which gives 86.0 GiB; from that we subtract about 3 GiB of overhead and the weights.

Budget: 95.6 GiB × 0.9, less about 3 GiB of overhead, less the weights of the published checkpoints, which keep the embeddings and output layer in BF16. Sessions are concurrent requests each holding a full 8K context with an FP16 KV cache; the second figure is the same calculation with an FP8 KV cache.

Take the Llama 3.3 70B row at FP8 as an example (Llama 3.1 and 3.3 70B share the architecture in the table above). The weights are 67.7 GiB in NVIDIA’s checkpoint; 86.0 minus 67.7 minus 3 leaves 15.3 GiB; each 8K session at 320 KiB per token needs 2.5 GiB; 15.3 divided by 2.5 is 6. One card therefore holds six concurrent full-length conversations, twelve with an FP8 cache and seventeen with NVFP4 weights. Weight quantisation has the largest single effect on concurrency.

Context length has the second-largest effect. At a declared 32K the same arithmetic leaves 17 sessions for the 8B model in BF16 and 1 for the 70B in FP8; one 70B session at 128K costs 40 GiB of FP16 KV, more than that whole budget.

An FP8 KV cache, vLLM’s kv_cache_dtype="fp8_e4m3", halves every KV figure. Since release 0.32.0 of 18 October 2026, vLLM also runs an NVFP4 KV cache on SM12x cards such as this one; published accuracy figures for it are still few, and our page on the NVFP4 KV cache on the RTX PRO 6000 Server Edition collects them, so the tables here plan with FP8. The counts are a floor: PagedAttention allocates blocks on demand, so sessions that never fill their declared context use less memory.

Our GPU servers for private LLMs, among them the four-card inference node, are sized from the same inputs. Tell us your model and the context you plan to declare, and we reply within one business day with this calculation and a configuration.

Memory bandwidth and per-user speed by edition

Capacity decides how many users fit; bandwidth decides how fast each is served. Decoding re-reads the weights from memory at every token, so at low batch sizes the per-user rate is bounded by bandwidth divided by bytes of weights, not by any FLOPS figure on the datasheet. All three editions share the GB202 die, 96 GB of GDDR7 with ECC and 24,064 CUDA cores, but not the memory clock.

NVIDIA datasheets and product pages. Lenovo’s product guide quotes “Up to 1597 GB/s” and documents an optional 450 W power cap on the Server Edition; NVIDIA’s reference architecture says up to 1.6 TB/s per GPU.

The Server Edition, the one for rack servers, has the slower memory. Most specification tables under this product name quote 1,792 GB/s, the Workstation and Max-Q figure, which is about 12 per cent higher; used in a server sizing calculation, it overstates decode throughput by the same 12 per cent.

The ceiling that follows is easy to compute. Llama 3.1 70B at FP8 reads about 70.6 GB of weights per token, and 1,597 divided by 70.6 is about 23 decode steps per second for a single unbatched user, before any efficiency loss. At NVFP4 each token reads about 40.6 GB, the 4-bit layers plus the output layer in BF16 (the embedding table is only looked up), and the ceiling rises to about 39. That is arithmetic from the published bandwidth and the weight size, not a measurement, and no real system reaches it. Run the same division for your model.

Reading speed is a useful lower reference for the generation rate users need. A 2019 meta-analysis of 190 reading studies in the Journal of Memory and Language puts mean silent reading of non-fiction at 238 words per minute; at the usual 0.75 words per token, that is about 5.3 tokens per second, or 6.7 for a fast reader. Interactive services are built well above that, because users skim and abandon, and reasoning models emit thinking tokens nobody reads. The interactive scenario of MLPerf® Inference v6.0 for Llama 2 70B, for example, requires the first token within 450 ms and at most 40 ms per output token at the 99th percentile, which is 25 tokens per second per user.

Batching: total throughput against per-user latency

Between these two limits the scheduler sets the trade-off, mainly through the batch size. The configurations where no metric improves without spoiling another are the Pareto front.

Before reading any throughput figure, separate two quantities that are routinely printed as one. Throughput per system is total output tokens per second across all requests running simultaneously; throughput per user is output length divided by end-to-end latency. NVIDIA describes one DGX B200 server with eight Blackwell GPUs both as a machine delivering over 1,000 tokens per second to each user and as a 72,000 tokens per second machine on Llama 4 Maverick, in two configurations of the same silicon. A throughput figure can only be compared if it states whether it is aggregate or per user, the concurrency, the sequence lengths and the precision.

The trade-off is not monotonic either, as the sweep in NVIDIA’s own TensorRT-LLM tuning guide shows.

NVIDIA’s TensorRT-LLM performance tuning guide, Llama 3.3 70B on four H100. Different hardware to the card discussed here; what transfers is the shape of the curve, not the absolute values.

Raising the ceiling from 64 to 512 gave 27 per cent more throughput at almost no latency cost. Raising it from 512 to 2,048 made throughput worse, by about 17 per cent. Past the knee, a higher ceiling costs throughput and gives nothing back in latency. The default max_batch_size in that stack is 2,048; this result does not make the default wrong, but it is a reason to sweep the setting on your own model.

Slicing one card: MIG, vGPU and the licence

MIG works on all three editions: NVIDIA’s MIG user guide covers Workstation, Max-Q and Server Editions in one profile table, offering 1g.24gb with four instances, 2g.48gb with two and 4g.96gb with one. There is no 3g profile, so the granularity is 24, 48 or 96 GB with nothing in between: a 32 GB model cannot take a 24 GB slice and has to occupy a 48 GB one.

vGPU is stricter. The only RTX PRO 6000 entries on NVIDIA’s supported-GPU list are the Blackwell Server Edition, from vGPU 19.0, and its liquid-cooled variant, from 20.0; the Workstation and Max-Q editions appear nowhere in it, so if the plan involves virtual machines the Server Edition is the only option. For compute the relevant profiles are the C-series: one DC-96C per card, two DC-48C, three DC-32C, four DC-24C, six DC-16C, eight DC-12C, or twelve DC-8C at 8 GB each. The MIG-backed variants, which isolate rather than time-slice, top out at four.

The figure of 48 virtual machines per card needs care before it goes into a design document. NVIDIA’s vGPU 19.0 blog says these GPUs support as many as 48 concurrent virtual machines on one GPU, then explains the arithmetic itself: four MIG instances multiplied by twelve VMs each. That is virtual-desktop density from small graphics profiles; for LLM serving the limit is twelve. Our guide on MIG and vGPU counts per card makes the same distinction across the range. Partitioning does not add capacity: four slices hold four small models, not more users of one large model.

NVIDIA counts AI Enterprise licences and desktop vGPU licences differently. NVIDIA AI Enterprise, which the C-series compute profiles require, is licensed per GPU: one for every GPU in the server that hosts the software. vApps, vPC and RTX vWS are usually licensed per concurrent user, one CCU licence per concurrent virtual machine, although NVIDIA also sells some vGPU editions per GPU. So a card serving a model to 200 people over an API needs one AI Enterprise licence if it runs NVIDIA AI Enterprise software such as NIM, because it is one GPU, and none if it runs an open-source engine such as vLLM on bare metal; concurrent-user counting applies to virtual desktops, not to HTTP clients of an inference endpoint.

We supply NVIDIA AI Enterprise licences with the cards, on the same invoice. If you are not sure whether your setup needs one, describe it in the form below and we will tell you.

When one card stops being enough

Four situations call for a second card; three are about capacity and one is about availability.

1. The model does not fit. Llama 3.1 70B in BF16 is 141.2 GB; split over two cards it leaves about 17 GiB of cache per card by the rule above.
2. The context is long. One 70B session at 128K costs 40 GiB of FP16 KV, more than the 15.3 GiB left beside FP8 weights; even with an FP8 cache (20 GiB) a single such session needs NVFP4 weights or a second card.
3. Concurrency exceeds the KV budget. That is the arithmetic above, and quantisation is the first move against it.
4. Availability. One card is a single point of failure: a driver update, an ECC fault or a hung process takes the service down, and MIG and vGPU share its failure domain entirely. A second card, ideally in a second host, is the only way to remove that risk, even when the arithmetic says one card is enough.

No edition has NVLink, and that changes how a second card is used. Lenovo’s product guide lists NVLink as “No” and NVIDIA’s product brief for the Server Edition as “Not supported”; we found no mention of it in NVIDIA’s specification table or architecture whitepaper, and the reference architecture describes GPU placement purely in PCIe terms.

NVIDIA’s own H200 page gives both figures side by side: NVLink at 900 GB/s per GPU against PCIe Gen 5 at 128 GB/s, roughly seven times. Every exchange between RTX PRO 6000 cards goes over the slower link, so the choice of parallelism matters. vLLM’s documentation recommends tensor parallelism within one node, but a note on uneven splits adds that on GPUs without NVLink, pipeline parallelism gives higher throughput and lower communication overhead. Tensor parallelism does two all-reduces in every layer; pipeline parallelism passes activations once per stage boundary.

The order of steps follows from this. Quantise before you scale out: FP8 or NVFP4 on one card beats BF16 split across two cards with no fast link. For more aggregate throughput, run independent replicas, one instance per card behind a load balancer, which removes inter-GPU traffic entirely and doubles availability. Split the model only if it does not fit, and then prefer pipeline parallelism for throughput and tensor parallelism where the latency of each request matters. Tensor parallelism stays supported, with TP2, TP4 and TP8 profiles in the NIM matrix, but its all-reduces cross PCIe, at a cost in speed. The direction of that cost is documented; published measurements of its size on this card are few, and our PCIe and NVLink article collects them. A server with four cards, the inference node among our AI server configurations, follows the same arithmetic: four copies of Llama 3.3 70B in FP8 hold 24 concurrent 8K sessions with a 16-bit cache and 48 with an FP8 cache, and two tensor-parallel pairs hold 78 and 156, because each pair stores the weights once. Each pair synchronises over PCIe in every layer, and by default vLLM disables its custom all-reduce for more than two GPUs without NVLink, which favours two TP=2 pairs over one TP=4 copy; vLLM 0.30.0 (release notes of 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for such servers (pull request 53576), with little published about its gain so far. Throughput at peak concurrency is not measured for these layouts; we would test two TP=2 instances against four single-card copies and one TP=4 copy on your model in a load test.

The same rule scales to whole servers. Two, four and eight cards hold 166.1, 332.2 and 664.3 GiB of weights and KV cache, and the layout decides how much of it is cache: every copy stores its own weights, so fewer, larger copies hold more conversations, at the price of all-reduces over PCIe.

Our estimates by the users-per-card rule (0.9 × 95.6 GiB, less 3 GiB per card, less the weights), concurrent conversations at full context length, with an FP8 KV cache for Llama 3.3 70B, Qwen3-235B-A22B-2507 and Llama 4 Maverick and a 16-bit cache for gpt-oss-120b and Qwen3.8-27B, as in our guides to two, four and eight Server Edition cards; the eight-card NVFP4 and Qwen3.8-27B figures are eight single-card copies by the same arithmetic. Weights: Llama 3.3 70B 67.7 GiB in FP8 and 39.8 GiB in NVFP4, gpt-oss-120b 60.8 GiB, Qwen3.8-27B 28.8 GiB, Qwen3-235B-A22B-2507 219.8 GiB, Llama 4 Maverick about 417 GB.

Our guides to two, four and eight Server Edition cards show these layouts per model, with power, chassis and published measurements.

Sizing procedure in eight steps

The procedure needs one card for an afternoon and ends with a number measured on your own model.

1. Count concurrency, not people. Log requests in flight at the gateway for a week; take the peak and the 95th percentile.
2. Decide the context you will declare, which is not the model’s maximum: 8K instead of an inherited 40K multiplies concurrency by five.
3. Compute the weights: parameters × bytes per parameter, or the size of the checkpoint you will load. Decide the quantisation here: 141.2, about 72.7 and about 42.7 GB are the same 70B model.
4. Compute KV per token: 2 × layers × kv_heads × head_dim × bytes, taking kv_heads from num_key_value_heads and never from num_attention_heads.
5. Subtract and divide. 95.6 GiB × 0.9, less about 3 GiB, less the weights, divided by per-session KV. That is your floor.
6. Start the server and read the log. Launch vLLM with that model, quantisation and declared context and read the GPU KV cache size and Maximum concurrency values (use vLLM 0.31.0 or later, which fixes advisory GHSA-h3rc-6mm3-gc2m, preferably 0.32.0; as of October 2026). If they disagree badly with step five, step five is wrong.
7. Measure latency at that load: time to first token and inter-token latency at realistic sequence lengths, against a first-token target and a per-user rate you will defend. Sweep the batch ceiling around the knee instead of accepting a default.
8. Decide availability separately. If the service must survive a driver update, the answer is two cards in two hosts whatever the arithmetic said.

By our users-per-card rule, one card serves an 8B model in BF16 to 17 full-length sessions at 32K and 68 at 8K (136 with an FP8 cache), Qwen3-32B in FP8 to 25 at 8K, and Llama 3.3 70B in FP8 to 6, or 12 with an FP8 cache; NVFP4 weights raise the 70B figures to 17 and 34.

What we supply

We supply the RTX PRO 6000 Blackwell in all three editions, with manufacturer warranty on one EU contract and invoice, as cards or in GPU servers built to order with two to eight cards per node, and the NVIDIA AI Enterprise licences where your setup needs them.

FAQ