CPU, RAM and storage for a 4-GPU AI server: cores per GPU, system RAM vs VRAM, NVMe or NAS

CPU, RAM and storage for a 4-GPU server: the published ratios

A 4-GPU AI server needs at least 24 physical CPU cores, system memory of at least twice its total GPU memory and one NVMe drive of 1 TB or more per CPU socket. These are the starting values in NVIDIA’s configuration guide for NVIDIA-Certified Systems, last updated on 30 September 2026. For four 96 GB RTX PRO 6000 Server Edition cards that means 768 GB of RAM, and for four 141 GB H200 NVL cards 1,128 GB. NVIDIA presents these figures as a starting point to adapt to the workload, and its reference architecture and DGX systems set the ratios differently.

NVIDIA-Certified Systems Configuration Guide (30 September 2026); NVIDIA Enterprise Reference Architecture for RTX PRO AI Factory, components, written for an eight-GPU RTX PRO Server (18 May 2026); DGX H100/H200 user guide (26 January 2026) and DGX B300 user guide (5 October 2026); Intel’s specification of the Xeon 6776P (64 cores). DGX B300 GPU memory: 2.3 TB in the user guide, 2.1 TB on the product page. DGX per-GPU figures are our division by eight.

Both DGX systems are eight-GPU SXM systems for training and inference, so a four-card PCIe server sits at the lower end of each column.

System RAM vs VRAM: what each one holds

VRAM, the memory on the GPU, holds what the GPU computes with: the model weights, the KV cache of active conversations and the activations. System RAM holds everything around it. On an LLM server that is the operating system, the serving engine’s processes, the model files on their way to the cards and transfer buffers; a RAG server adds the vector database, the document parser and the gateway. GPU memory per model and user count is sized in buying an AI server: what to specify.

vLLM memory-maps safetensors weights by default, so the files pass through the Linux page cache. While unused memory allows, they stay there, and a restart or a second copy of the same model on the next card can read them from RAM instead of from disk or the network.

In vLLM’s documentation as of October 2026, two settings place GPU data in system memory on purpose. --cpu-offload-gb reserves the given number of GiB per GPU for model weights, which is a fallback for a model that does not fit the cards and not a sizing target. --kv-offloading-size sets a buffer in GiB for KV cache blocks, summed across all tensor-parallel ranks, so that returning conversations find their cache in RAM instead of recomputing it. When that pays off is covered in KV cache offloading to CPU memory and NVMe; here it counts as part of the RAM budget.

How many CPU cores per GPU

The serving engine sets the floor. vLLM’s tuning guide counts one API server process, one engine core process and one worker process per GPU, so a server with N GPUs needs at least 2 + N physical cores. For data-parallel serving the guide gives the minimum as A + DP + N, plus one when DP is above one, where A is the number of API servers (by default equal to DP) and N the total number of GPUs. Four data-parallel copies on four cards therefore need 13 physical cores, or 10 with one API server. Four separate instances behind a load balancer need 12. The guide warns: “Underprovisioning CPU cores is a common source of performance degradation, especially in virtualized environments.”

NVIDIA’s figures leave room for the rest of the host. The certified-systems guide asks for six physical cores per GPU, 24 for four cards. The RTX PRO reference architecture, written for an eight-GPU server, asks for seven per GPU, two more per GPU for the operating system or virtualisation and two per MIG instance, so four cards come to 36 cores, on bare metal or in virtual machines, before any MIG partitioning. The extra cores serve tokenisation, document parsing for RAG, the vector database, monitoring agents and, during fine-tuning, the data loader workers.

PCIe lanes and memory channels: one socket or two

Four GPUs at Gen5 x16 take 64 PCIe lanes. One AMD EPYC 9005 processor has 128, so a single socket also has the lanes for four NVMe drives at x4 and two network cards at x16, 112 lanes in all. NVIDIA’s RTX PRO reference architecture accepts a single CPU socket as its minimum, while the certified-systems guide starts from two. In a two-socket EPYC server, part of each processor’s lanes links the two sockets, and AMD gives up to 160 lanes for devices in total; how many GPUs fit in one server explains the lane budget for larger builds.

A second socket adds memory channels and room to place devices next to each processor. Each EPYC 9005 socket has up to 12 DDR5-6400 channels, and the reference architecture asks for at least one DIMM in every channel, evenly across the sockets. Two sockets give 24 channels, so 24 DIMMs of 32 GB make 768 GB, and 24 of 64 GB make 1,536 GB. NVIDIA’s floor of 128 GB per GPU, 512 GB for four cards, fills only 16 of the 24 channels with 32 GB modules. NVIDIA’s guide also asks for GPUs spread evenly across sockets and root ports, with “NICs and NVMe drives” placed “within the same PCIe switch or root complex as the GPUs”. In a four-card server that means two GPUs, one network card and the data drives next to each processor, so a GPU reads its data without crossing the socket link.

Local NVMe or NAS for model weights and datasets

Local drives hold what the cards use now; a NAS or NFS share holds what several servers use or what must survive the loss of one.

Floors are our arithmetic from link rates: a Gen5 x16 link carries about 64 GB/s per direction, so x4 about 16 GB/s; 25 and 100 GbE line rates divided by eight. Drive and file system overheads add to these times. Checkpoint: NVIDIA’s Llama 3.3 70B FP8 on Hugging Face.

Four servers pulling a checkpoint at once share the NAS’s own uplink, so inference servers keep the models they serve on local NVMe and fetch new versions from the repository. Fine-tuning rereads its dataset and writes large checkpoints, so it needs a local scratch tier.

NVIDIA’s GPUDirect Storage reads from storage into GPU memory by DMA and “avoids a bounce buffer through the CPU”. vLLM’s documentation describes a weight loader built on it, the fastsafetensors library.

NVIDIA’s DGX H200 puts its operating system on two 1.92 TB NVMe drives in RAID 1 and its eight 3.84 TB data drives in RAID 0, labelled as a data cache. The same split suits a 4-GPU server, with the cache holding copies whose originals live on shared storage. Where the local model store is the only copy, put it on RAID 1 or RAID 10 and keep a checksummed copy off the server, as described in backing up an AI server.

We build AI servers with scratch, model store and index tiers on data-centre NVMe drives. Describe where your models and datasets are stored today in the form below, and we size the local tiers and the network link to your storage.

Worked example: 4-GPU inference and fine-tuning servers

The inference server runs an assistant with RAG for 500 employees on Qwen3-32B in FP8, one copy per RTX PRO 6000 Server Edition behind a load balancer. With an FP8 cache, each card holds 51 concurrent sessions at a full 8K context by the sizing rule of our buying guide, 204 for the server. Five hundred people with access produce far fewer generations at once, and a pilot’s busy-hour logs show whether 204 is enough. Two 32-core processors cover the 12 cores of four vLLM instances, the 36 of the reference architecture and the RAG services. 768 GB of RAM, twice the GPU memory, fills every channel with one 32 GB DIMM and leaves room for the page cache, the vector index and a KV offload buffer for part of the users. Keeping all 500 conversations warm in CPU memory needs a 500 GiB buffer, so for that case our KV cache offloading guide plans about 1 TB, 24 DIMMs of 48 GB.

The fine-tuning server runs four H200 NVL on a 4-way NVLink bridge, 564 GB of GPU memory, for a full fine-tune of Llama 3.3 70B with its 70.6 billion parameters. The ZeRO paper counts 16 bytes per parameter for mixed-precision training with Adam. That is 2 bytes for the 16-bit weights, 2 for the gradients and 12 for the optimiser states, an FP32 copy of the weights plus momentum and variance. For this model that is about 1,130 GB, twice what the four cards hold. DeepSpeed’s ZeRO-Offload “offloads the optimizer memory and computation from the GPU to the host CPU”. Its paper keeps the 16-bit weights on the GPUs and moves the FP32 model states and the 16-bit gradients to CPU memory, 14 bytes per parameter or about 990 GB here. ZeRO stage 3, with which DeepSpeed also allows this offload, splits the 141 GB of 16-bit weights across the four cards. By our arithmetic a resumable checkpoint, the 16-bit weights plus the optimiser states, is also about 990 GB. A LoRA run keeps the base weights frozen and needs a fraction of this memory.

Our configuration for the two workloads above, built from NVIDIA’s guide and reference architecture, vLLM’s CPU guidance, AMD’s EPYC 9005 datasheet and the memory counts of the ZeRO and ZeRO-Offload papers; the network rows follow our 70B worked example and NVIDIA’s guide.

The fine-tuning server gets more cores because ZeRO-Offload runs the optimiser step on the processors. The two-card case, one processor and at least the total GPU memory, is worked through in our 70B GPU server example.

We size memory to the GPU pool and check the rack, power and airflow before we quote. Send us the cards, the models and your peak sessions through the form below for a configuration and quote within one business day.

What we supply

We build AI servers to order around the workload, with AMD EPYC or Intel Xeon Scalable processors, DDR5 ECC memory, data-centre NVMe drives for the scratch, model store and index tiers, and 25 to 400G network cards, assembled and burn-in tested. The inference node on that page starts from 512 GB, NVIDIA’s floor of 128 GB per GPU; the workload above needs 768 GB, or about 1 TB with the KV tier. The cards in this article, the RTX PRO 6000 Server Edition and the H200 NVL with NVLink bridges, are among the professional NVIDIA GPUs we supply, with manufacturer warranty on every component on one EU contract and invoice. Operating system, drivers, CUDA and a container runtime are installed on request.

FAQ