Fine-tuning server, 2 to 8 GPUs, 8B to 70B: RTX PRO 6000 Server Edition or H200 NVL

What a fine-tuning server does, in plain words

A fine-tuning server adapts an existing open model, such as Llama 3.3 70B or a Qwen3 model, to a company's documents, terminology or tasks. A server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards does this for models from 8B to 70B with LoRA and QLoRA; four cards run several such jobs at once or one larger run; eight H200 NVL with NVLink bridges are the configuration for full fine-tuning of 32B and 70B models. The method decides the size of the server more than the model does.

LoRA and QLoRA train a small adapter on top of a frozen model and need a fraction of the memory of a full fine-tune, which retrains every weight. For Llama 3.3 70B the model states, meaning weights, gradients and optimiser states, come to 42.8 GB for QLoRA, 144.4 GB for LoRA on a 16-bit base and 1,129 GB for a full fine-tune, by the per-parameter rules in our guide to GPU memory for fine-tuning. Whether a company needs fine-tuning at all, or retrieval over its documents, is the subject of our fine-tuning or RAG comparison.

For a director the decision comes down to two questions: which methods the team will use in the next two years, and how many runs should be able to run at the same time. The configurations below answer both.

Three configurations: 2, 4 and 8 cards

These are the three starting points on our AI servers page. Memory and processor cores follow NVIDIA's configuration guide for NVIDIA-Certified Systems: at least twice the GPU memory in system memory, at least six cores per GPU and two sockets.

Reference configurations of our AI servers page, October 2026; more memory on request. Runs by our arithmetic in the next table, model states only.

Two variants sit between them. Two H200 NVL with a 2-way bridge hold an 8B full fine-tune on one card and a 70B LoRA at 72.2 GB per card over NVLink. Eight RTX PRO 6000 Server Edition cards run eight QLoRA jobs or a full 32B fine-tune at 65.5 GB per card, over PCIe.

Which runs fit which configuration

The table divides the model states of each run by the cards it uses and compares the result with the nominal memory of the card: 96 GB on an RTX PRO 6000 Server Edition and 141 GB on an H200 NVL. What is left holds activations, the CUDA context and buffers, which grow with batch size and sequence length.

Our arithmetic: 16 bytes per parameter for full fine-tuning with 32-bit Adam and about 10 with 8-bit Adam (ZeRO paper's count), a frozen BF16 base at 2 bytes and an NF4 base at 0.516 bytes for linear layers, rank-16 adapters on all linear layers; Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B with 8.03, 32.76 and 70.55 billion parameters.

Read the table from the bottom up. With 32-bit Adam a full 70B fine-tune needs 141.1 GB per H200 NVL, the whole card before any activation, so it needs 8-bit Adam, which cuts the states to 88.2 GB per card, or an optimiser in host memory, covered below. A full 32B fine-tune needs eight H200 NVL with 32-bit Adam, or four RTX PRO 6000 Server Edition cards with 8-bit Adam. Everything above those rows runs on two cards of either kind.

We build these servers with two, four or eight cards; send us the models, the method and the sequence length, and our team replies with the configuration that holds your runs and the memory left per card.

Data parallel or FSDP

There are two ways to use several cards for one run. In data parallel training, every card holds the whole run and trains on its share of the batches; after each step the cards add up their gradients in an all-reduce. PyTorch's FSDP documentation describes this as NO_SHARD, where states are "replicated across ranks similar to PyTorch's DistributedDataParallel API". In sharded training, FSDP's FULL_SHARD mode splits "parameters, gradients, and optimizer states" across the cards and gathers the parameters before the forward and again before the backward pass.

Use data parallel whenever the run fits one card with room for activations. The traffic is small for adapters: a rank-16 LoRA on Llama 3.3 70B trains 207.1 million weights, 0.41 GB of BF16 gradients per step, and the ZeRO paper counts 2Ψ of data movement for the all-reduce, about 0.83 GB per card per step. At 64 GB/s per direction on PCIe Gen5 that is around 13 ms, by our arithmetic. Two, four or eight RTX PRO 6000 Server Edition cards scale QLoRA and LoRA runs this way without NVLink.

Use FSDP when the states do not fit one card. It saves memory and costs traffic: the ZeRO paper puts a fully sharded run at 1.5 times the traffic of data parallel training, and for a 70B LoRA the weights of the frozen base cross the link on every pass. On two cards each card receives about 141 GB per forward and backward pass, on four about 212 GB. That is at least 2.2 and 3.3 seconds per pass over PCIe and about 0.3 seconds over a 2-way NVLink bridge at 450 GB/s, by our arithmetic; FSDP hides part of it behind computation.

QLoRA can be sharded too. The bitsandbytes documentation describes FSDP-QLoRA, which trains "LLMs up to 70B parameters on a dual 24GB GPU system", on the condition that bnb_4bit_quant_storage matches the data type of the other layers. On a server with 96 GB cards there is rarely a reason to: a 70B QLoRA fits one card with about 53 GB left, and data parallel training is simpler.

Across servers, PyTorch offers HYBRID_SHARD, which applies full sharding "within a node" and replicates "across nodes", so the expensive all-gathers stay inside each server and only gradients cross the network. That is the layout for two or more eight-card nodes on the 400G network of our training configuration.

NVLink bridges on the H200 NVL, PCIe on the RTX PRO 6000 Server Edition

The H200 NVL connects two or four cards with an NVLink bridge at 900 GB/s per GPU, against 128 GB/s for PCIe Gen5, both counted in two directions. A bridge joins four cards at most, so the eight-card server has two NVLink domains with PCIe between them. Dell's technical guide for the PowerEdge XE7740 states that "Eight DW H200 GPUs need two bridges"; Puget Systems pairs two cards with one 2-way bridge, four with one 4-way bridge and eight with two 4-way bridges; Cisco's spec sheet for the UCS C845A M8 allows four 2-way or two 4-way bridges for eight cards. NVIDIA's reference architecture places bridged cards under the same processor socket, and Dell calls bridging across processors "possible but not recommended".

For fine-tuning this gives a simple rule. Keep each sharded run inside one NVLink domain. A 70B LoRA on four bridged H200 NVL holds 36.1 GB per card and moves its weights over NVLink; the other four cards run a second job. A run sharded over all eight cards crosses PCIe for part of its traffic, which is only worth it when the states need more than four cards, as a full 32B or 70B fine-tune does. Our guide to the H200 NVL NVLink bridge shows how to check the domains with nvidia-smi topo -m.

No RTX PRO 6000 edition has NVLink. The cards communicate over PCIe, peer to peer where the platform allows it. NVIDIA's NCCL documentation warns that "CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer" and that ACS on PCIe switches has to be disabled, so both belong in the acceptance test, as our RTX PRO 6000 P2P guide explains. For data-parallel LoRA and QLoRA this costs little. For FSDP runs the H200 NVL is the faster platform, as our H200 NVL vs RTX PRO 6000 comparison shows with tensor rates and traffic.

Host memory: what the processors' RAM has to hold

NVIDIA's configuration guide for NVIDIA-Certified Systems, updated on 30 September 2026, asks for system memory of "2x times of total GPU memory". Our reference configurations meet it; the table shows how each size fills the 24 memory channels of two AMD EPYC processors with one module per channel.

NVIDIA-Certified Systems configuration guide, read on 24 October 2026; module counts by our arithmetic for 24 memory channels. Cisco's spec sheet for the UCS C845A M8 recommends at least three times the HBM for H200 NVL, 4,096 GB for eight cards.

Fine-tuning adds a use for host memory that inference does not have: the optimiser. The ZeRO-Offload paper keeps the 16-bit parameters on the GPU and "allocates all the fp32 model states along with the fp16 gradients on the CPU memory", 14 bytes per parameter, and keeps "a single copy of the optimizer states on the CPU memory regardless of the data parallel degree". By our arithmetic that is 112.4 GB for an 8B model, 458.6 GB for a 32B model and 987.7 GB for Llama 3.3 70B. DeepSpeed's documentation lists optimiser offload as valid for ZeRO stages 1, 2 and 3, and in stage 3 "the 16-bit model parameters are partitioned across the processes", so a full 70B fine-tune on eight H200 NVL holds about 17.6 GB of weights per card and about 1 TB in host memory. The 2.3 TB of our training node holds that with room for data loading and the page cache; the 768 GB of a four-card server would not. The processor then performs the parameter update, as the paper describes.

The other host rules are simpler. NVIDIA's guide asks for at least six physical cores per GPU, 12, 24 and 48 cores for the three configurations, and two sockets. Our guides to how much RAM a GPU server needs and to CPU, RAM and storage for a 4-GPU server go into processor choice and memory channels.

NVMe: datasets, checkpoints and offload

The checkpoints, more than the training data, set the storage size of a fine-tuning server. A resumable checkpoint of a full fine-tune stores the 32-bit master weights and both Adam moments, 12 bytes per parameter by the ZeRO paper's accounting, plus the 16-bit weights at 2 bytes. By our estimate that is about 112 GB for an 8B model, 459 GB for a 32B model and 988 GB for a 70B model, and a team that keeps the last three checkpoints of a 70B run needs about 3 TB for them alone. A LoRA checkpoint is small: the rank-16 adapter of a 70B model has 207.1 million weights, 0.41 GB in BF16, about 3.3 GB with its optimiser states.

NVIDIA's RTX PRO AI Factory reference architecture of 18 May 2026 recommends 2 TB of NVMe per processor socket for deep learning, against 1 TB for inference, plus a 1 TB boot drive. The eight drives of our training node give room for datasets, checkpoints and base models on separate drives. DeepSpeed can also offload optimiser states and parameters to NVMe, which ZeRO-Infinity introduced; it trades step time for memory and suits runs that do not fit even the host memory. Our guide to NVMe storage for an LLM server covers drive types and endurance.

Power and rack position

Training keeps the cards near full power for hours or days, unlike a chat service that idles between requests. Plan the feed for the full load.

Our estimate: 600 W per card from NVIDIA's documents; two AMD EPYC 9135 at 200 W each in the two-card server, two EPYC 9355 at 280 W in the four-card server and two 64-core EPYC 9555 at 360 W in the eight-card server (AMD's default TDP); plus 15 per cent for memory, drives, network cards, fans and conversion losses; a measurement on the delivered system confirms the figure.

We plan 80 per cent of a feed's rating: 2.94 kW on 230 V at 16 A, 5.89 kW at 32 A and 8.87 kW on a 400 V three-phase feed at 16 A. The two-card server fits a 16 A feed, the four-card server needs 32 A or three-phase at 600 W per card, and the eight-card server needs three-phase power or several feeds, with each side of an A/B pair able to carry the whole server. Dell's guide for the XE7740 limits the inlet temperature to 30 °C with H200 NVL, against 35 °C for other cards, so check the room as well as the feed. Our guide to GPU server power on 16 A, 32 A and three-phase works through redundant supplies and cords.

Which configuration to start with

A first project or one department. Two RTX PRO 6000 Server Edition cards. They run LoRA and QLoRA on every model size in this article, two jobs at once, and the same server serves the result. Our page to buy a 2× RTX PRO 6000 Server Edition server lists the variants.

Several teams or a queue of experiments. Four Server Edition cards: four jobs side by side, or one 70B LoRA sharded over four cards when a team needs it. Two four-card servers instead of one eight-card server give two failure domains, as our comparison of one 8-GPU server or two 4-GPU servers explains.

Full fine-tuning of 32B or 70B models. Eight H200 NVL with two 4-way bridges, 2.3 TB of host memory and 400G for a second node later. Our page on the eight-card H200 NVL server covers what it runs when it serves models.

The card choice behind these configurations is in our comparison of the best GPU for fine-tuning in 2026.

Ordering a fine-tuning server from us

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. Sizing is free. The server is assembled and tested before delivery, and we deliver it in the EU on one contract and invoice, with manufacturer warranty on every component and the assembly covered by us; other destinations go through our logistics partner. On request we install the operating system, drivers, CUDA and a container runtime, and commission the server on site. The training platform on top, MLOps on Kubernetes, is part of our Private AI/ML service, built by our engineering partner Vixen.UNO. Quantities and lead time are confirmed in writing with the quote.

For an exact configuration we need the base models, the method per model, the sequence length, how many runs should run at once and the power feed at the rack position. Describe these in the form below, and we answer with a configuration and a written quote.

FAQ