vLLM KV cache size and Maximum concurrency: reading the start-up log on RTX PRO 6000

What the two log lines say

When vLLM starts, it reports how many tokens of key/value cache fit in the GPU memory it has claimed, and how many requests of the declared maximum length that is. The first figure, GPU KV cache size, is a token count for the whole instance; the second, Maximum concurrency, is that count divided by the tokens one request may occupy, so 15.70x means fifteen and a bit requests at full length could sit in the cache at the same time. It is a worst case for the context you declared, not the number of users a server can carry, and it moves with every flag that changes memory or context.

The lines look like this, in the example our guide to users per RTX PRO 6000 quotes (the values illustrate the format only):

GPU KV cache size: 643,232 tokens

Maximum concurrency for 40,960 tokens per request: 15.70x

For a model with full attention in every layer the second is the first divided by the stated length: 643,232 / 40,960 = 15.70. The length in that line is the server's --max-model-len, and if you did not set it, vLLM derives it from the model configuration, as its engine-argument reference for 0.32.0 states. A model with a 128K context therefore gets a concurrency figure for 131,072 tokens per request unless you say otherwise, which is why many first start-ups print a disappointing number.

The line counts requests that use their whole context. Real chat requests rarely do: a question with a retrieved document might take 6,000 tokens of a declared 32,768. vLLM's PagedAttention allocates cache blocks as a request grows, so the cache holds more short requests than the printed figure suggests. When it fills, vLLM pauses requests instead of failing them; its optimisation guide for 0.32.0 describes a warning that a sequence "is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space" and says such requests "are recomputed when sufficient KV cache space becomes available again". Preemptions therefore show up as latency, and our guide to vLLM monitoring metrics explains how to alert on vllm:num_preemptions_total and on cache usage.

How vLLM arrives at the token figure

The calculation starts with a share of the card's memory. --gpu-memory-utilization is "the fraction of GPU memory to be used for the model executor", 0.92 if unspecified, and the reference adds that it "is a per-instance limit": two vLLM instances on one GPU would each be set to 0.5. vLLM 0.20.0 of April 2026 raised the default from 0.9 to 0.92. From that share vLLM subtracts what it measures at start-up.

vLLM 0.32.0 engine arguments, optimisation and conserving-memory guides (pages dated 20 August and 9 October 2026), read 24 October 2026; component names as in Anyscale's GPU guidance page for vLLM, which lists "model weights", "non_torch_memory", "PyTorch activation peak memory" and the rest "reserved for the KV Cache".

Two of these shares surprise people. The activation peak grows with --max-num-batched-tokens, the number of tokens processed in one step. A user on vLLM's forum reported in April 2026, with vLLM 0.18.0 on a 24 GB A30, available KV cache memory of 11.32 GiB at a budget of 16K tokens, 9.64 GiB at 32K, 6.24 GiB at 64K and 4.53 GiB at 80K; a larger prefill budget buys faster first tokens at the cost of cache. CUDA graphs also take memory: vLLM's conserving-memory guide says graphs "take up extra memory in the GPU" and that the capture sizes by default go "up to max_num_seqs". --enforce-eager disables graph capture completely, at a cost in decode speed.

If you would rather fix the cache than the fraction, --kv-cache-memory-bytes sets the "Size of KV Cache per GPU in bytes", and the reference notes that when set it "ignores gpu_memory_utilization". That is useful for two instances on one card or when another process holds GPU memory.

The token count then follows from the model's architecture. The bytes per token are 2 × layers × key/value heads × head dimension × bytes per element; for Llama 3.3 70B, with 80 layers, 8 key/value heads and a head dimension of 128, that is 320 KiB per token in BF16 and 160 KiB in FP8. Dividing the leftover memory by that figure gives the GPU KV cache size.

Worked numbers for one RTX PRO 6000 and one H200 NVL

We plan with our users-per-card rule: 0.9 × the driver-visible memory, less 3 GiB of overhead per card, less the weights. It sits slightly below vLLM's 0.92 default on purpose. For Llama 3.3 70B in NVIDIA's FP8 checkpoint (67.7 GiB of weights), the rule gives these log values, by our estimate:

our arithmetic by the users-per-card rule (95.6 GiB driver-visible per RTX PRO 6000, 140.4 GiB per H200 NVL, × 0.9, less 3 GiB per card, less 67.7 GiB of weights; the second row uses 0.92), FP8 KV cache at 160 KiB per token. vLLM's own activation and graph measurement replaces our 3 GiB, so the printed values differ.

Three things follow. At 8,192 tokens per request, one Server Edition card holds 12 full-length requests with an FP8 cache, the figure our sizing pages use, and 6 with a 16-bit cache. At the model's full 131,072 tokens it holds less than one: the cache cannot keep a single maximum-length request, so declare the context your users need. And two cards in a tensor-parallel pair hold 78 rather than 24, because the 67.7 GiB of weights is stored once across both cards; with tensor parallelism each card keeps its share of the key/value heads for every token, so the printed token count describes the whole instance.

The H200 NVL figure of 44 comes from its larger memory, 140.4 GiB visible to the driver: one card holds more of these requests than two RTX PRO 6000 Server Edition cards running separate copies, which hold 24. Our comparison of RTX PRO 6000 and H200 NVL for LLM inference weighs speed, NVLink and power.

We build servers with RTX PRO 6000 Server Edition and H200 NVL cards and size them from these figures; send us your model, its precision and the context you plan to declare, and our team replies with the expected KV cache and a configuration.

Flags that move Maximum concurrency

Some flags change the memory the cache gets, others only change the divisor. Only the first kind lets the server carry more work.

vLLM 0.32.0 engine arguments and release notes (18 October 2026, "NVFP4 KV cache now runs on SM8x and SM12x", extended to SM90 in the same entry); NVFP4 factor by our arithmetic (0.5625 against 1 byte per value, 90 against 160 KiB per token for Llama 3.3 70B). Read 24 October 2026.

--max-model-len accepts human-readable values such as 8k, and "-1 or 'auto'" makes vLLM "choose the maximum model length that fits in GPU memory", using the model's maximum if it fits and otherwise "the largest length that can be accommodated", in the reference's words. That is the documented way out when a start-up fails because a full-length request does not fit. It hides the decision from your users, though: a client that sends a longer prompt gets an error, so set the length deliberately in production.

--kv-cache-dtype fp8 halves the bytes per token compared with a 16-bit cache. Since 0.32.0, vLLM also runs nvfp4 as a KV cache dtype on SM12x cards such as the RTX PRO 6000, which by our arithmetic takes Llama 3.3 70B on one card from 12 to 21 requests of 8,192 tokens. The accuracy cost is a separate question; our article on the NVFP4 KV cache on the RTX PRO 6000 Server Edition collects the published measurements.

--max-num-seqs is different in kind. It caps how many sequences the scheduler runs in one step and sizes the CUDA graphs, but reserves no cache. vLLM's scheduler reference calls its built-in 128 a value "mainly for convenience when testing" and says the real default is set when the engine configuration is created; vLLM's gpt-oss guide describes it as "a large number like 1024 on GPUs with large memory sizes". If Maximum concurrency at your real prompt lengths is lower than --max-num-seqs, the cache fills first and requests are preempted; if it is higher, the batch cap queues requests first. Lowering --max-num-seqs reduces graph memory and so adds a little cache.

Sliding-window, hybrid and MoE models

The simple division holds for models with full attention in every layer, such as Llama 3.3 70B. Newer models mix layer types. vLLM's hybrid KV cache manager design page (dated 19 August 2026) says sliding-window layers reserve "slots only for the most recent sliding_window_size tokens" and free the blocks outside the window, while full-attention layers keep all of them; all layer types share one memory pool with one page size. For such a model the bytes per token fall as the context grows, and the concurrency line is not the token line divided by the length. gpt-oss-120b, which alternates full and windowed attention, needs about 4.5 GiB of 16-bit cache for one 128K conversation against 40 GiB for Llama 3.3 70B, as our guide to long-context LLM hardware shows. Mamba layers keep a state per request rather than per token, and the same design page notes that vLLM raises the attention block size so that its pages match the larger Mamba state. For these models, read the concurrency line rather than computing it.

Mixture-of-experts models change the weights, not the cache. Such a model reads only its active experts per token but stores all of them, and its cache is set by its attention layers alone. Our page on gpt-oss-120b hardware requirements works through one such model.

From the log line to a number of users

Maximum concurrency answers how many full-length requests fit at once. The number you need for sizing is requests in flight at the busiest moment, measured at the gateway, and how long each one is. Our guide to private AI pilots explains how to measure both. Three checks connect them.

1. Declare a realistic context. Set --max-model-len to the longest prompt plus answer your application sends, not to the model's maximum; the printed concurrency then refers to a request you will see.
2. Compare with peak requests in flight. If the measured peak is below the printed concurrency at that context, the cache will not run out even when every request is long. If it is above, preemptions start once enough long requests coincide.
3. Watch the live figures. Cache usage and preemptions at /metrics show how close the real mix comes to the limit; short requests let the server run well above the printed figure without preemption.

A server that must carry more long requests at once needs more cache: an FP8 or NVFP4 cache, NVFP4 weights, a tensor-parallel pair, or a card with more memory. Our users-per-card guide gives the figures per model and layout, and our guide on KV cache offloading to CPU memory and NVMe covers the case where cached prefixes, not running requests, run out of room. Our engineering partner Vixen.UNO deploys models with vLLM as part of our Private AI/ML service; write to us with the start-up log of your current server, and we size the next one from it.

FAQ