KV cache offloading to CPU memory and NVMe: when it helps an LLM server and how to size it

KV cache offloading: what it does and when it helps

KV cache offloading moves cache blocks that the GPU does not need for the current step from GPU memory to CPU memory, and from there to NVMe or shared storage, so that they can be reused later instead of recomputed. The reuse comes from multi-turn chats, long system prompts shared by many requests, agents that resend their history and RAG prompts over the same documents. On a hit, the engine copies the stored blocks back to the GPU and skips that part of the prefill, so the first token arrives sooner and the GPUs spend less compute on repeated prompts.

Offloading does not make token generation faster. vLLM’s documentation on automatic prefix caching, updated on 13 October 2026, states that it “does not reduce the time of generating new tokens (the decoding phase)”. vLLM’s offloading guide, updated on 15 October 2026, describes the connector as one “which extends the prefix cache by offloading completed KV blocks to slower but larger tiers.” By the reading of our guide to long-context LLM hardware, the conversation being generated keeps its cache in GPU memory, so offloading does not raise the longest context a card can serve; vLLM’s HiSparse for GLM-5.3, described there, is the exception.

How big the KV cache gets: users times context

The cache grows with the number of conversations times their length. For a model with full attention in every layer, one token takes 2 × layers × KV heads × head dimension × bytes per value, the method of our guide to how much VRAM an LLM needs. Qwen3-32B’s config.json on Hugging Face lists 64 layers, 8 KV heads and a head dimension of 128. With an FP8 cache, set in vLLM with --kv-cache-dtype fp8, that gives 2 × 64 × 8 × 128 × 1 byte = 128 KiB per token, and 1 GiB for one full 8K conversation of 8,192 tokens. The model’s 16-bit default doubles both figures.

A shared system prompt costs little, because prefix caching keeps one copy of it for every request that starts with it. Per-user history is different, as each conversation carries its own cache, and that is what offloading keeps between turns.

Worked example: 500 users on four RTX PRO 6000 Server Edition

Take Qwen3-32B in FP8, one copy per card on a server with four RTX PRO 6000 Server Edition, and 500 users who each keep a conversation of up to 8K tokens. Our sizing rule takes 0.9 of the 95.6 GiB the driver reports, minus 3 GiB for CUDA graphs and activations, minus the 32.0 GiB of FP8 weights. That leaves 51 GiB of cache per card, or 51 conversations at full length, and 204 on the four cards.

The other 296 conversations are idle between turns, and without offloading the engine evicts their blocks when new requests need the space, and a returning user pays a full prefill again. To keep all 500 warm, the CPU tier has to hold 500 GiB, about 537 GB. vLLM’s offloading guide asks for a CPU tier larger than the GPU cache, because “a smaller CPU tier just mirrors what the GPU already holds and adds no hit rate.” With four copies that means about 125 GiB of pinned memory per instance, each above its own 51 GiB on the GPU.

Pinned memory cannot be swapped out, so the offload buffer comes on top of the operating system, containers, a RAG index and the page cache that holds the 34.3 GB checkpoint. Our guide to CPU, RAM and storage for a 4-GPU server plans this server with 768 GB in 24 DIMMs of 32 GB, twice its GPU memory, and with about 1 TB for a KV tier that holds all 500 users. In 768 GB the 500 GiB buffer would leave only about 230 GB for everything else, so we plan the next even step, 24 DIMMs of 48 GB or 1,152 GB, which leaves about 615 GB beside the buffer. The 512 GB of the inference node on our AI servers page is a starting configuration, and the page states that “the sizing conversation decides the final build”. Spread the memory evenly across both sockets, so that each socket holds the buffers of its own two cards.

With a 16-bit cache every figure doubles, to 2 GiB per conversation and 1,000 GiB for 500 users, which takes 24 DIMMs of 64 GB. An FP8 cache is the first saving to test on your evaluation set.

We size system memory and the offload buffer with the GPUs. Tell us your model, context length and user count in the form below, and the configuration and quote follow within one business day.

CPU memory, PCIe and NVMe: capacity and bandwidth per tier

Every reload crosses the PCIe link between the card and its processor, and that link sets the speed. NVIDIA lists 128 GB/s for PCIe Gen5 on the H200 NVL, which is 64 GB/s in each direction of an x16 link, and the RTX PRO 6000 Server Edition uses the same Gen5 interface. AMD lists 614 GB/s of memory bandwidth per socket for an EPYC 9005 processor with 12 channels of DDR5-6400, more than the four links of a four-card server carry together.

NVIDIA RTX PRO 6000 Server Edition page (updated 10 March 2026) and H200 page (updated 27 August 2026); AMD EPYC 9355 product page; vLLM blog of 8 January 2026 (H100 system); capacities and reload times are our arithmetic.

A Gen5 x4 NVMe drive has a quarter of an x16 link, so its datasheet figure sits below 16 GB/s, and vLLM stages every file-tier transfer through CPU memory, since “Only the CPU primary tier has direct GPU access.” The NVMe tier suits conversations from earlier in the day and long documents that return now and then, not users who answer within a minute.

On a two-socket server, check with nvidia-smi topo -m which NUMA node each card belongs to and start each instance with its buffer on that node, for example with numactl --cpunodebind=0 --membind=0. Write volume matters as well. vLLM’s blog of 10 September 2026 says its tiering “cascades chunks to all configured secondary tiers simultaneously”, so every newly offloaded prompt block is also written to NVMe. At 1 GiB per 8K prompt, 2,000 new conversations a day write about 2.1 TB. Compare that with the rated drive writes per day of the drive you plan.

vLLM KV cache offloading: native connector and flags

vLLM has shipped CPU offloading since release 0.11.0, according to its blog of 8 January 2026, and the tiered framework since 0.22, according to the September post. The names below are those of the current release, 0.31.0 of 5 October 2026, as its engine arguments and offloading guide define them; the developer documentation of 15 October 2026 reads the same.

1. The short form is --kv-offloading-size 125, the buffer in GiB. By default it is unset and “no KV offloading is enabled”, and “When TP > 1, this is the total buffer size summed across all TP ranks.”
2. --kv-offloading-backend chooses native, the default, or lmcache.
3. The full form is --kv-transfer-config with "kv_connector": "OffloadingConnector", "kv_role": "kv_both" and, inside kv_connector_extra_config, the required cpu_bytes_to_use, which the guide defines as “Total bytes of host memory reserved for the CPU tier across all workers (not per-worker).” For 125 GiB that is 134217728000.
4. For NVMe, set "spec_name": "TieringOffloadingSpec" and list a tier with "type": "fs" and a root_dir on the drive under secondary_tiers.

The option offload_prompt_only defaults to true, so “only prompt (prefill) blocks are offloaded; decode blocks are skipped.” By our reading, a returning conversation finds its earlier prompts in the CPU tier, and the last answer is computed again unless it is still in GPU memory. The CPU tier belongs to one instance. With four copies of the model, a returning user gets a hit only on the copy that holds the conversation, which is what KV-aware routing in NVIDIA Dynamo and llm-d provides. A file system tier on the same root_dir can serve all instances, which the guide says “works by default”, unless the xxhash hash options are set without a common PYTHONHASHSEED. Dynamo’s page on native vLLM offloading, for release 1.5.1, uses the same OffloadingConnector and adds --router-mode kv on its frontend.

LMCache, NVIDIA Dynamo KVBM and SGLang HiCache

Project documentation, source code at the release tags and PyPI, read on 16 October 2026; the LMCache and SGLang guides carry no version number.

LMCache, under the Apache 2.0 licence, can run as a separate multiprocess server that vLLM workers reach over ZMQ or gRPC through the LMCacheMPConnector; its documentation also mentions the in-process LMCacheConnectorV1. That documentation carries no version number, so the names here are as read on 16 October 2026, with 0.5.6 current. --l1-size-gb sets the CPU tier and repeated --l2-adapter options add storage. Its PyPI page says it “reduces TTFT” for long-context, multi-turn and RAG workloads.

In Dynamo 1.5.1, the KV Block Manager defines DYN_KVBM_CPU_CACHE_GB as the “Size of the host (G2, pinned CPU memory) cache in GB” and recommends a disk cache at least as large as the CPU cache. A disk offload filter is on by default, and O_DIRECT and GPUDirect Storage are options. NVIDIA’s KVBM overview says that offloading “is most effective when KV Cache exceeds GPU memory and cache reuse outweighs the overhead of transferring data.”

SGLang’s HiCache extends RadixAttention to host memory and storage. In SGLang 0.5.22, --hicache-ratio sets “the ratio of the size of host KV cache memory pool to the size of device pool”, 2.0 by default in cache mode, and --hicache-size in GB overrides it. The HiCache guide notes that “L1 and L2 are private to a single inference instance; only L3 can be shared.” Our comparison of vLLM, SGLang and TensorRT-LLM covers the engines themselves.

Published results and when offloading does not pay

vLLM’s blog of 8 January 2026 reports, for Llama 3.1 8B on one H100 80 GB with 500 GB of host memory, that “loading KV values from the CPU reduces TTFT by X2-X22, depending on the prompt size”, and throughput gains of up to nine times as the CPU hit rate rises. It also states that the connector “has minimal effect on TTFT for cache misses.” The September post measured the prefill workers of a disaggregated setup for Qwen3.6-35B-A3B on two H100, with a file system tier on local NVMe. Beyond 128 conversations, that tier was “more than doubling throughput compared to the alternatives.” Neither setup uses an RTX PRO 6000 or an H200 NVL, so read these figures as direction for your own test.

Offloading pays when prompts repeat and the cache of returning sessions does not fit in GPU memory. It adds little for single-turn requests with unique prompts, for short prompts whose prefill is cheap, or when every conversation already fits on the GPUs. The January post called hybrid models “currently not optimized for the offloading connector”; the September post lists DeepSeek V4, GLM 5.3 and Nemotron 3 among the hybrid architectures the tiered framework supports, so test such a model on the release you run.

Our engineering partner Vixen.UNO deploys vLLM on-premise as part of our Private AI/ML service. Describe your prompt lengths and how often users return in the form below; the first call is free of charge.

What we supply

We build AI servers to order with 2 to 8 GPUs per node, with DDR5 ECC memory sized to the GPU pool and NVMe tiers for models, indexes and scratch, assembled and burn-in tested. For an offloading setup we size the GPUs, system memory and NVMe drives together and check the rack, power and airflow before we quote. The cards in this article, the RTX PRO 6000 Server Edition and the H200 NVL with NVLink bridges, are part of our professional GPU range, with manufacturer warranty on one EU contract and invoice. Serving engines, routing and RAG are our Private AI/ML service, with engineering by our partner Vixen.UNO.

FAQ