vLLM CUDA out of memory: causes and fixes by start-up phase, on RTX PRO 6000 and H200 NVL

Where vLLM runs out of memory

vLLM reports CUDA out of memory almost always while it starts, before it serves a single request: it checks that the share of the card it was given is free, loads the weights, measures its working memory and fills the rest with KV cache, then captures CUDA graphs. Each phase fails with its own message, and each has a different fix. Find the last phase the log reached, and you know which flag to change and whether the card is too small.

Once the server is up, out-of-memory errors are rarer. vLLM's optimisation guide for 0.32.0 says it "pre-allocates GPU cache" from the configured share, so a full cache pauses requests instead of crashing the engine. Our guide to vLLM KV cache size and Maximum concurrency explains those pauses and the log lines behind them. This page is the failure path: what each error means and what to do about it, with figures for the RTX PRO 6000 Blackwell Server Edition and the H200 NVL. All vLLM behaviour below is from the documentation of release 0.32.0 (18 October 2026) and NVIDIA's NIM 2.0.13 troubleshooting page, last updated 19 October 2026, read on 25 October 2026.

message fragments as quoted in vLLM GitHub issues 35313 (February 2026) and 19407, NVIDIA NIM 2.0.13 "Troubleshooting GPU Memory Out-of-Memory Errors" (19 October 2026), vLLM 0.32.0 conserving-memory guide (9 October 2026). Exact wording can change between releases. Read 25 October 2026.

Phase 1: another process holds GPU memory

--gpu-memory-utilization is, in the words of vLLM's engine-argument reference, "the fraction of GPU memory to be used for the model executor", 0.92 if unspecified. The reference calls it "a per-instance limit" and gives the example of two vLLM instances on one GPU at 0.5 each, which only adds up if the fraction refers to the whole card. The documentation of vLLM-Omni, a project built on vLLM, writes the rule out as requested_memory = total_gpu_memory × gpu_memory_utilization.

At start-up vLLM compares that request with what is free. A vLLM issue of February 2026 shows the result when it does not fit: "Free memory on device cuda:0 (74.9/142.5 GiB) on startup is less than desired GPU memory utilization (0.9, 128.25 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes." The numbers show the rule: 0.9 × 142.5 GiB = 128.25 GiB, against 74.9 GiB free.

On a 96 GB card, which the driver reports as 95.6 GiB, the default asks for 0.92 × 95.6 = 88.0 GiB, by our arithmetic. Anything else holding more than about 7.6 GiB of the card stops the start: a Jupyter kernel with a loaded tensor, a second container, an earlier vLLM process that did not exit, or on a workstation the desktop. nvidia-smi lists the processes and their memory. The fix is to stop the other process, or to give every instance on the card its own share so that the shares add up to less than 1. --kv-cache-memory-bytes sets the cache per GPU in bytes and "ignores gpu_memory_utilization", which suits a card shared with a process of known size.

One upgrade trap belongs here. vLLM 0.20.0 of April 2026 raised the default from 0.9 to 0.92, so an instance without an explicit setting claims about 1.9 GiB more of a 95.6 GiB card than before. Two services that just fitted next to each other can fail after the upgrade without any change to their own configuration. Set the share explicitly in every service file.

Phase 2: torch.OutOfMemoryError while loading weights

If the log ends while loading the weights with torch.OutOfMemoryError: CUDA out of memory, NVIDIA's NIM page names the cause plainly: the card "does not have enough memory to hold the model weights at the selected" precision and tensor-parallel degree. Its formula is weight memory per GPU = parameters × bytes per parameter / TP, with 2 bytes for BF16 and 1 for FP8.

Llama 3.3 70B shows the arithmetic. In BF16 its weights take about 131 GiB, more than one RTX PRO 6000 and, once overhead and cache are counted, more than one H200 NVL as well: our users-per-card rule allows 0.9 × 140.4 − 3 = 123.4 GiB for weights and cache on the H200 NVL. NVIDIA's FP8 checkpoint takes 67.7 GiB and fits one card of either type. In BF16 the model needs two RTX PRO 6000 Server Edition cards with --tensor-parallel-size 2, about 65.5 GiB per card.

Two details catch people. A checkpoint that is quantised on the fly loads at 16-bit first: our guide to how much VRAM an LLM needs quotes NVIDIA's note that such an FP8 profile needs room for the BF16 weights at start-up. And tensor parallelism divides the weights, not the overhead: every rank keeps its own CUDA context, activations and graphs. On an eight-card server with RTX PRO 6000 Server Edition, the choice between one copy at TP=8 and several smaller copies is a question of speed as much as memory; our guide to the RTX PRO 6000 without NVLink covers what PCIe all-reduce costs.

Phase 3: no room for the KV cache

When the weights load but the remainder cannot hold one request of the declared length, vLLM stops with a message of this kind, as quoted in vLLM issue 19407: "To serve at least one request with the models's max seq len (...), (... GiB KV cache is needed, which is larger than the available KV cache memory (... GiB)." The same message continues "Based on the available memory, the estimated maximum model length is ...", and ends with "Try increasing gpu_memory_utilization or decreasing max_model_len". Users on vLLM's forum also report the shorter "No available memory for the cache blocks" when nothing at all is left for the cache.

The length vLLM checks is --max-model-len, which the reference says is "automatically derived from the model config" if you do not set it. For Llama 3.3 70B that is 131,072 tokens. One request of that length needs 40 GiB of cache at 16 bits and 20 GiB with an FP8 cache, and by our users-per-card rule one RTX PRO 6000 Server Edition has about 15.3 GiB left after the FP8 weights. The start fails, and the estimated maximum length in the message, by our estimate about 100,500 tokens with an FP8 cache, is the useful number in it.

The fixes, in the order we would try them:

1. Declare the context you need. --max-model-len 32768, or auto, which in the reference's words picks "the maximum model length that fits in GPU memory". Clients that send longer prompts then get an error, so choose the value deliberately.
2. Halve the cache. --kv-cache-dtype fp8 stores keys and values in 8 bits; since 0.32.0, nvfp4 is a further option on SM12x cards such as the RTX PRO 6000.
3. Reduce the step budget. A large --max-num-batched-tokens raises the activation peak that vLLM measures before sizing the cache; the KV cache guide above quotes a forum measurement in which the cache shrank from 11.32 to 4.53 GiB as the budget grew from 16K to 80K tokens.

NVIDIA's NIM page adds a warning that matters here: lowering --gpu-memory-utilization "reduces the KV-cache budget and can make a KV-capacity failure worse". Raising it from 0.92 towards 0.95 gains about 2.9 GiB on a 95.6 GiB card, by our arithmetic, at the cost of headroom for allocations the profiler does not see.

We build RTX PRO 6000 Server Edition and H200 NVL servers and size them for the context your users need; send us the model, its checkpoint and the start-up log, and our team replies with the configuration that holds it.

Phase 4: CUDA graph capture and warm-up

If the log shows the cache size and then fails, often after "Graph capturing finished" or in compile_or_warm_up_model, NVIDIA's NIM page attributes it to graph capture or warm-up needing "more memory than remains available", and adds that "there is no single amount of headroom that works for every profile". vLLM's conserving-memory guide says CUDA graphs "take up extra memory in the GPU", and that the capture sizes by default go "up to max_num_seqs".

Three fixes, in increasing cost to speed:

• lower --gpu-memory-utilization a little, which leaves more headroom outside vLLM's budget; NIM gives 0.9 to 0.85 as an illustration, "not a validated value";
• capture fewer graphs, with a smaller --max-num-seqs or an explicit list of cudagraph_capture_sizes in the compilation configuration, as the guide shows with sizes 1, 2, 4, 8 and 16;
• start with --enforce-eager to confirm that graphs are the cause. It disables capture completely and reduces decode speed, so use it to diagnose, not as the production setting.

A variant of this failure is fragmentation. The PyTorch message then reports a large amount "reserved by PyTorch but unallocated" and suggests PYTORCH_ALLOC_CONF=expandable_segments:True. NIM calls this "particularly common with hybrid architectures" and notes that the setting "does not add GPU capacity".

Multimodal models: images, video and the encoder cache

Vision-language models add memory that text models do not have. vLLM's memory profiling at start-up takes into account the images, videos and audio a prompt may carry, so a model that accepts several high-resolution images or a video reserves room for them. The conserving-memory guide gives two levers. --limit-mm-per-prompt caps items per prompt, for example three images and one video, and setting a modality to 0 frees its memory; {"image": 0} runs a multimodal model for text only. Some models also accept processor arguments that shrink inputs, for the Qwen2-VL series max_pixels, by default 1280 × 28 × 28.

Two notes from the same guide prevent wrong conclusions. The size hints in --limit-mm-per-prompt "only affect activation memory profiling", and the encoder cache is set by the real inputs at run time, so a server that started cleanly can still run short when users send larger images than you profiled for. And mm_processor_cache_gb, 4 GiB by default, is a cache in CPU memory: lowering it helps when the host runs out of RAM, not the GPU. Our guide to vision-language models for document AI sizes these models per card.

Several models on one card, and MIG slices

Running two models on one RTX PRO 6000 works if each instance gets its own explicit share and the shares, together with any other process, stay below the card's memory. Without MIG the instances share one memory space, and a burst in one can starve the other at its next start.

MIG gives each instance a fixed slice. On every RTX PRO 6000 edition NVIDIA's MIG guide offers 1g.24gb four times, 2g.48gb twice or 4g.96gb once, as our page on RTX PRO 6000 MIG profiles lists. Each slice has its own memory, so one instance cannot take memory from another. An 8B model in BF16, about 16 GB of weights by NVIDIA's NIM example, fits a 24 GB slice with a short context by our estimate; Llama 3.3 70B in FP8 needs the whole card. NVIDIA's MIG guide states that "NCCL is currently not supported with MIG", so tensor parallelism, which runs its collective operations over NCCL, cannot join slices. Our step-by-step MIG guide sets the slices up.

When the fix is a larger card or a second one

Every flag above trades something: context length, cache precision, speed or the number of requests in flight. If the model only starts with eager mode, a context shorter than your documents or a 4-bit checkpoint you have not validated, the card is too small for the job. We plan with our users-per-card rule: 0.9 × the memory the driver reports, less 3 GiB per card, less the weights. For Llama 3.3 70B it gives these figures, by our estimate:

our arithmetic by the users-per-card rule (95.6 GiB driver-visible per RTX PRO 6000, 140.4 GiB per H200 NVL), FP8 KV cache at 1.25 GiB per 8,192-token request; BF16 weights of about 131 GiB as in our VRAM guide. vLLM measures its own overhead, so its log values differ.

The H200 NVL holds more such requests on one card than two RTX PRO 6000 cards running separate copies, which hold 24, and it adds NVLink bridges for larger models. The RTX PRO 6000 Server Edition offers 96 GB per card with FP4 Tensor Cores, which our comparison of RTX PRO 6000 and H200 NVL for LLM inference weighs against bandwidth and NVLink. Our guide to users per RTX PRO 6000 gives the figures for other models.

Our engineering partner Vixen.UNO deploys models with vLLM on these servers as part of our Private AI/ML service; write to us with the error, the model and the number of users, and we reply with the setting or the configuration that fixes it.

How to fix a vLLM OOM, check by check

1. Find the phase. The last line before the error says whether vLLM was checking free memory, loading weights, sizing the cache or capturing graphs.
2. List the processes. nvidia-smi on the host shows every process on the card; on a MIG card, check the instance the container sees.
3. Set the share explicitly. Write --gpu-memory-utilization into every service file, so a changed default in an upgrade cannot move it.
4. Declare the context. Use the longest prompt plus answer your application sends, not the model's maximum.
5. Check the checkpoint. The format you load, FP8, NVFP4 or BF16, sets the weight memory; a profile that quantises at load time needs 16-bit room first.
6. Diagnose graphs with eager mode, then return. Production servers keep CUDA graphs, with fewer capture sizes if needed.
7. Record the working set. Our guide to operating a private LLM platform puts out-of-memory at start and after an upgrade into its runbook table.

FAQ