NVMe storage for an LLM server: model weights, load times, RAG indexes, KV cache offload

How much NVMe an LLM server needs

An LLM server needs local NVMe for three things: the model files it serves, the RAG index next to the models, and, where returning conversations are many, a KV cache offload tier. NVIDIA's guides set the floor at one NVMe drive of at least 1 TB per CPU socket for inference and 2 TB per socket for deep learning, and checkpoints of current open models run from about 31 GB to 690 GB each. For an inference server with NVIDIA RTX PRO 6000 Blackwell Server Edition or NVIDIA H200 NVL cards, a mirrored model store of a few terabytes plus a separate boot pair covers most company deployments, with an extra drive or two for the offload tier.

The capacity is the easy part. What decides the layout is where the original copy of each file lives, how fast a model has to be back after a restart, and how many writes a cache tier puts on its drives. The sections below give the figures for each, with load times as lower bounds from link and drive rates, labelled as our arithmetic.

Model store: checkpoint sizes of current models

A checkpoint is the set of safetensors files the serving engine loads. Its size follows parameters times bytes per parameter, plus scales and embeddings, and the Hugging Face file listing gives the exact figure.

Hugging Face file listings, read on 24 October 2026 for our sibling articles (GB = 10⁹ bytes; Qwen3-235B-A22B-2507 is the Instruct-2507 release); Qwen3-32B and Llama 3.3 70B BF16 as in our guides to KV cache offloading and to backing up an AI server. GPU layouts for 96 GB RTX PRO 6000 Server Edition cards by our sizing rule, as in our articles on two, four and eight cards.

Three things add to the sum of the models in service. During an update the old and the new version sit side by side until the new one passes its tests, so plan twice the size of the largest model. Evaluation candidates come and go: a team testing three 70B checkpoints in BF16 holds about 423 GB of them by our arithmetic. And embedding and reranker models are small, about 1.2 GB for Qwen3-Embedding-0.6B and 15 GB for the 8B model in BF16, as our guide to embedding and reranker servers shows.

A worked example, by our arithmetic: an inference node with four Server Edition cards that serves gpt-oss-120b and Qwen3-235B-A22B in FP8 next to an 8B embedding model holds 65.3 + 236 + 15 GB, about 316 GB, and about 632 GB while all three are being updated. A test slot for one 141 GB BF16 model brings it to about 773 GB. NVIDIA's floor of 1 TB per socket would hold it, but on single drives without a mirror and with little room to grow, so we plan such a node with a mirrored pair of 3.84 TB drives for models and index.

How fast a model loads from NVMe, a NAS or the page cache

A restart of the serving engine reads the whole checkpoint again, and on a server with several copies of a model each copy reads its share. The floors below are the transfer alone: the checkpoint size divided by the data rate of the source.

Our arithmetic: 14.0 GB/s sequential read per drive as Micron lists for its 9550 PCIe Gen5 NVMe SSD (four drives 56 GB/s), 25 and 100 GbE line rates divided by eight. Lower bounds; file system, engine start-up and weight processing come on top.

The floor is rarely what a default setup reaches. NVIDIA's Dynamo documentation (release 1.6.0) describes vLLM's default loader as one that "memory-maps each safetensors shard and copies tensors out largely single-stream", and measured about 124 MB/s effective on a cloud Lustre file system. At that rate a 72.7 GB checkpoint would take nearly ten minutes by our arithmetic. In the same document a 554 GiB checkpoint loaded across two nodes in 76 minutes before and 5.3 minutes after switching to the Run:ai Model Streamer and striping the files over all storage targets. The page calls the default loader latency-bound, so the gap is largest on remote storage, and network file systems need parallel reads to come near their bandwidth.

vLLM's load settings, as documented on 17 October 2026, offer the levers. By default safetensors files are memory-mapped and loaded lazily, with prefetching switched on automatically when an NFS file system is detected and the checkpoint fits into free RAM. The prefetch strategy reads the files into the operating system's page cache first, and eager reads each file into CPU memory before loading, which the documentation recommends for network file systems. The runai_streamer load format reads safetensors with many concurrent requests. Once a checkpoint is in the page cache, a restart or a second copy on another card reads it from RAM, which is one reason NVIDIA's certified-systems guide asks for host memory of twice the GPU memory, as our guide to how much RAM a GPU server needs explains.

For inference servers this gives a clear rule: keep the models in service on local NVMe and fetch new versions from the shared repository in the background. Our guide to CPU, RAM and storage for a 4-GPU server compares local NVMe and a NAS in more detail.

We build LLM servers with NVMe sized to the models you run and the ones you plan to test; send us the models and how often you update them, and our team replies with drive count, capacity and layout.

RAG indexes and document stores

A RAG system adds a vector index, the parsed text of the documents and usually the original files. The vectors themselves are small. As our vector database comparison works out, a million 1,024-dimensional float32 vectors take 4.10 GB before any index, so ten million chunks are about 41 GB of vector data before indexes and row overhead. An HNSW index adds its graph, and pgvector's HNSW index stores each vector a second time.

The index usually lives in RAM while it serves queries, and NVMe holds its persistent copy, the database's write-ahead log and the document store. Milvus also documents DISKANN, which keeps its graph on disk for datasets "too big to fit in memory"; that is the case where NVMe read latency enters every query. For a company corpus of a few million chunks, the RAG data fits on the same mirrored pair as the models. It should be mirrored and backed up, because rebuilding it means parsing and embedding the corpus again: about 17 hours for 3 million chunks on one L4 by our embedding guide's arithmetic. Our guide to backing up an AI server covers what to copy and how.

KV cache offloading to NVMe

KV cache offloading keeps the cache of conversations that are not generating right now outside GPU memory, first in CPU memory and then on NVMe, so that a returning user's prompt does not have to be computed again. Our guide to KV cache offloading to CPU memory and NVMe covers the flags in vLLM, LMCache, NVIDIA Dynamo and SGLang; here the question is what the NVMe tier needs.

Figures from our KV cache offloading guide (vLLM documentation and blog posts of September and October 2026); drive-writes figure by our arithmetic: 2.15 TB ÷ 3.84 TB.

Three properties follow from vLLM's design. Every file-tier transfer passes through CPU memory, since in vLLM's words "Only the CPU primary tier has direct GPU access", so the NVMe tier suits conversations from earlier in the day, not users who answer within a minute. Every newly offloaded prompt block is also written to NVMe, so the tier wears its drives at the rate of new conversations, and the endurance rating of the drive has to match. And the tier is a cache: if a drive fails, the server computes the prompts again. It does not need mirroring, and striping two drives can double its bandwidth. NVIDIA's Dynamo KV Block Manager recommends a disk cache at least as large as the CPU cache and offers O_DIRECT and GPUDirect Storage for it.

NVMe per socket in NVIDIA's guides, and what the servers offer

NVIDIA's configuration guide for NVIDIA-Certified Systems, updated on 30 September 2026, asks for one NVMe drive of at least 1 TB per CPU socket. Its Enterprise Reference Architecture for RTX PRO AI Factory, updated on 18 May 2026, sets a minimum of 1 TB per CPU socket for inference servers and 2 TB per socket for deep-learning servers, and states that "NIC and NVMe drives should be under the same PCIe switch or PCIe root complex as the GPUs". On a two-socket server that means data drives on both processors, each next to the cards that read from it, so that weights and offloaded cache do not cross the link between the sockets.

Dell PowerEdge XE7740 Technical Guide Rev. A04 (April 2026) and XE7745 Technical Guide Rev. A09 (September 2026); HPE ProLiant Compute DL380a Gen12 QuickSpecs; Lenovo product guide LP2128 (updated 20 October 2026); Supermicro SYS-422GA-NRT datasheet; all read on 24 October 2026.

Eight front bays are the common figure for these GPU servers. Dell lists E3.S drives from 1.6 to 15.36 TB for the XE7740 and XE7745 and no storage controller for either; the XE7745 guide states that storage controllers are not supported in this release, and both mark Dell's internal, external and software RAID controllers as not available. Dell's guides offer no RAID for the data bays of these servers, so mirroring them is a task for the operating system, for example Linux software RAID. HPE offers Intel VROC with RAID 0, 1, 5 and 10 for up to eight direct-attached NVMe bays, depending on the licence. All four put the boot drives on M.2 modules separate from the data bays, with their own mirroring where the maker offers it.

RAID and mirroring: which tier gets what

The protection follows from where the original of the data on each drive lives.

Our recommendation, following the split of NVIDIA's DGX H200 (operating system on two 1.92 TB drives in RAID 1, eight 3.84 TB data drives in RAID 0 as a data cache) as described in our CPU, RAM and storage guide.

The model store looks like a cache, since every checkpoint can be downloaded again, but it often is not one. Our backup guide shows that a gated model can become unavailable, that your own fine-tuned weights and LoRA adapters exist nowhere else, and that a 141 GB checkpoint takes almost 19 minutes to copy over a 1 Gbit/s line. A mirrored pair keeps the server serving through a drive failure, and a copy off the server covers the rest. Published builds follow the same pattern: CoreWeave's eight-card RTX PRO 6000 Server Edition instance has two 7.68 TB drives in RAID 1, and Google's G4 shapes come with 3, 6 and 12 TB of local SSD for two, four and eight cards.

Training is the case for striping. A resumable checkpoint of a full Llama 3.3 70B fine-tune is about 990 GB, as our CPU and RAM guide counts. At the 10.0 GB/s of sequential writes Micron lists for the 9550, one drive needs at least 99 seconds to write it and six drives striped at least 17 seconds, by our arithmetic. A job that checkpoints every hour spends that time each hour, so the scratch tier is sized for speed and the finished checkpoints are copied to mirrored or shared storage.

Storage in our reference configurations

Our reference configurations on AI servers carry two, four or eight NVMe drives. A layout that fits the tiers above, as our example:

• Entry server, 2 × RTX PRO 6000 Server Edition, 2 × NVMe: one mirrored pair for models, RAG index and documents; the boot drives on the server's separate boot module.
• Inference node, 4 × RTX PRO 6000 Server Edition, 4 × NVMe: a mirrored pair for models and index, two drives striped or single as the KV offload tier, one per socket.
• Training node, 8 × H200 NVL, 8 × NVMe: a mirrored pair for models and finished checkpoints, six drives striped as scratch for datasets and running checkpoints, spread over both sockets.

Capacities follow the models: 3.84 TB per drive covers the worked example above with room for growth, and the deep-learning floor of 2 TB per socket is met by the scratch tier many times over. Our engineering partner Vixen.UNO deploys vLLM and RAG platforms on-premise as part of our Private AI/ML service; tell us which models, documents and users the server will handle, and we size GPUs, memory and NVMe together.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight RTX PRO 6000 Server Edition or H200 NVL cards and data-centre NVMe drives laid out for model store, RAG data, offload tier and scratch. Each server is assembled and tested before delivery, and every component carries the manufacturer warranty on one EU contract and invoice.

FAQ