Unsloth, Axolotl, TRL or LLaMA-Factory: which one uses every card of a 4- or 8-GPU server

Which fine-tuning framework uses all GPUs of the server

On a server with four or eight GPUs, Axolotl, TRL and LLaMA-Factory use every card for one run, either as data-parallel copies or with the model sharded across the cards through FSDP or DeepSpeed ZeRO. Unsloth documents data-parallel training on several cards and a mode that splits one model across cards, and its documentation still calls official multi-GPU support upcoming. For LoRA and QLoRA runs that fit one card, all four frameworks keep the cards busy; for runs that do not fit one card, choose Axolotl, TRL or LLaMA-Factory.

This page compares the software. The memory each run needs, and which server holds it, is worked out in our guides to GPU memory for fine-tuning and to the fine-tuning server with 2, 4 or 8 GPUs; the choice of card is compared in H200 NVL or RTX PRO 6000 for fine-tuning and the best GPU for fine-tuning in 2026, and single-device runs on DGX Spark. The figures below come from each project's documentation, README and release notes as read on 24 October 2026. These projects release often, so check the current notes before you plan a run.

Project documentation, README files and GitHub release pages, read on 24 October 2026. Unsloth's licence split is quoted from its README.

Three ways to put several cards to work

A fine-tuning framework can use the cards of one server in three ways, and the differences between the projects are mostly about which of them they support.

One job per card. Four cards run four independent jobs, each started with its own CUDA_VISIBLE_DEVICES. Every framework can do this, because each job sees one GPU. It suits a team that runs many small experiments, such as several QLoRA runs with different data.

Data parallel (DDP). Each card holds a full copy of the run and trains on its own share of the batches; after each step the cards add up their gradients. Unsloth's DDP guide puts it as "one copy of the model on each GPU device, feeding each copy distinct samples". The run must fit one card. For a rank-16 LoRA on Llama 3.3 70B the gradients come to about 0.83 GB per card per step, about 13 ms over PCIe Gen5 by the arithmetic on our fine-tuning server page, so data parallel training scales well without NVLink.

Sharded (FSDP2 or DeepSpeed ZeRO-3). The weights, gradients and optimiser states are split across the cards and gathered when a layer needs them. This is the way to train a run that does not fit one card, such as a LoRA on a 16-bit 70B base at 144.4 GB of model states, or a full fine-tune. It moves much more data: the 70B LoRA on four cards receives about 212 GB per card on every forward and backward pass, by our arithmetic in the same guide.

A fourth arrangement, the device map, places a model's layers on several cards so that it fits at all. Unsloth offers it, as described below. It is a way to fit a model, and its speed is something to measure.

We supply servers with two, four or eight cards for all three ways of working; tell us how many runs your team starts at once and how large they are, and we size the cards and host memory to them.

Unsloth: fast on one card, DDP on several

Unsloth's multi-GPU page states that it "currently supports multi-GPU setups through libraries like Accelerate and DeepSpeed" and that "you can already leverage parallelism methods such as FSDP and DDP". The page then gives only the DDP procedure: write a training script and run accelerate launch train.py or torchrun --nproc_per_node N_GPUS train.py. It also says the team is "working hard to make multi-GPU support much simpler" and will be "announcing official multi-GPU support for Unsloth soon". We found no Unsloth page that documents an FSDP or ZeRO-3 setup.

The DDP guide shows the measured case. Qwen3-8B on two NVIDIA H100 80 GB cards used "~19GB of VRAM per H100 GPU" at about 1.1 iterations per second, and Unsloth writes that throughput "increases ~linearly with the number of GPUs", with "a small, but increasing penalty for the distributed comms" as cards are added. DDP is "auto-enabled when training with >1 GPU".

For models larger than one card, Unsloth's page offers device_map = "balanced": "we will split the model for you on each GPU", shown with Llama 3.3 70B in 4-bit. Its latest GitHub release on 24 October 2026, v0.1.806-beta of 2 September, adds "Improved multi-GPU planning, memory fitting, and split-model training" and "Train larger models across multiple GPUs with automatic placement". Neither page says how the cards share the computation in this mode.

The licence matters for companies. Unsloth's README states: "The core Unsloth package remains licensed under Apache 2.0, while certain optional components, such as the Unsloth Studio UI are licensed under the open-source license AGPL-3.0." Have your legal counsel check the AGPL terms before you offer the Studio UI to other users over a network.

Use Unsloth on a multi-GPU server when each run fits one card: QLoRA up to 70B and LoRA up to 32B on a 96 GB card, one job per card or DDP across all cards.

Axolotl: FSDP2, DeepSpeed and context parallelism

Axolotl's multi-GPU documentation lists three options and notes that you "generally cannot combine these strategies": DeepSpeed, which "supports ZeRO stages 1-3"; FSDP, "PyTorch's native sharding implementation (Recommended)"; and DDP, the "Default if neither of the above are selected". FSDP means FSDP2 now. The release notes of version 0.20.0, dated 30 September 2026, state that "FSDP2 is the default and the only supported version". For DeepSpeed the page advises "Start from Stage 1 -> Stage 2 -> Stage 3", and the configuration takes a key such as deepspeed: deepspeed_configs/zero1.json.

Long sequences are split across cards with context parallelism, set with context_parallel_size; Axolotl prefers Ulysses and falls back to Ring attention. Version 0.20.0 adds packed sequences under "Ulysses, Ring, or USP context parallelism" and expert parallelism through "a plain all_to_all backend", and the same notes allow LoRA training "on a frozen native NVFP4 base under FSDP2, DeepSpeed ZeRO-1/2/3, or tensor parallelism". The README lists N-D parallelism, which combines context parallelism, tensor parallelism and FSDP within one node or across nodes, and multi-node training with torchrun or Ray.

Two requirements from the README: Python 3.12 or later and PyTorch 2.13.0 or later. Version 0.18.0 added support for "consumer Blackwell (RTX 50xx / sm120)"; NVIDIA gives the RTX PRO Blackwell cards the same compute capability 12.0, which our page on Blackwell RTX software support covers. Axolotl is licensed under Apache 2.0.

TRL: Hugging Face's trainers on Accelerate

TRL is the library behind SFTTrainer, DPOTrainer and GRPOTrainer, and several other tools build on it; its README says it integrates Unsloth for "accelerating training using optimized kernels". The README states that TRL "natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP". Its distributed-training guide uses accelerate config and accelerate launch train.py, notes that this "automatically distributes the workload across all available GPUs", and gives the batch rule: per-device batch size × number of devices × gradient accumulation steps. The DeepSpeed page uses examples/accelerate_configs/deepspeed_zero2.yaml as its example. The latest release on 24 October 2026 was 1.16.0 of 15 October. Not every trainer has the same status: TRL's documentation lists DistillationTrainer with the main trainers, while GKDTrainer, GOLDTrainer and AsyncDistillationTrainer are experimental and are imported from trl.experimental, as our page on LLM distillation on premises describes.

The long-context guide is the most detailed multi-GPU document of the four projects. It pairs context parallelism (parallelism_config_cp_size) with FSDP2 and accelerate 1.11, and Ulysses sequence parallelism (parallelism_config_sp_size) with DeepSpeed 0.18.1 and accelerate 1.12; "cp_size requires FSDP2 and sp_size only runs under DeepSpeed". Sequence parallelism scales "up to the number of KV heads", eight in its example. The guide's example needs one node with eight H100s and trains Qwen3-8B; at 131k tokens, it reports, a step takes 34.6 s across two GPUs, 17.8 s across four and 9.5 s across eight. Context parallelism rules out packing and models with sliding-window attention; the guide names gpt-oss, Gemma 3 and 4 and Qwen3.5 and later. Our page on GRPO and reinforcement learning covers TRL's GRPOTrainer with vLLM on the same servers.

LLaMA-Factory: one CLI, several back ends

LLaMA-Factory wraps training into one command, llamafactory-cli train with a YAML file, and supports full, freeze, LoRA and QLoRA tuning as well as continued pretraining. Its examples README starts multi-GPU runs with FORCE_TORCHRUN=1, gives a DeepSpeed ZeRO-3 example (qwen3_lora_sft_ds3.yaml), multi-node runs with NNODES and NODE_RANK, elastic restarts and a Ray example on four GPUs. The repository also ships an Accelerate FSDP configuration, examples/accelerate/fsdp_config.yaml, with FULL_SHARD. The README's news lists "FSDP+QLoRA that fine-tunes a 70B model on 2x24GB GPUs" (20 March 2024) and a "Megatron-core training backend with mcore_adapter" (26 October 2025). We found no FSDP2 statement in its README or examples. The licence is Apache 2.0.

The README's memory table, which it marks as estimated, is a quick check: it gives 18 × the parameter count in GB for full training in mixed precision, 2 × for LoRA and 0.5 × for 4-bit QLoRA, and lists 1,200 GB, 160 GB and 48 GB for 70B models. The model states in our guide for a 70B model, 1,128.8, 144.4 and 42.8 GB, are a little lower.

Four RTX PRO 6000 Server Edition cards over PCIe

No edition of the NVIDIA RTX PRO 6000 Blackwell has NVLink, so on a four-card server every byte between the cards crosses PCIe Gen5. The choice of framework follows from the size of the run.

Model states from our fine-tuning memory guide (rank-16 adapters, 16 bytes per parameter with 32-bit Adam, about 10 with 8-bit Adam); memory left against the listed 96 GB per card. Framework column from the documentation above.

Data-parallel runs fit this server best, because only gradients cross the link. Sharded runs work, but each pass moves the frozen base over PCIe: about 212 GB per card for the 70B LoRA, at least 3.3 seconds per pass at 64 GB/s by our arithmetic, part of which FSDP overlaps with computation. Before sharded runs, check peer-to-peer transfers: NVIDIA's NCCL documentation states that "CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer", and our RTX PRO 6000 P2P guide shows the checks. Our four-card Server Edition page covers the rest of the server.

Eight H200 NVL with two NVLink domains

An NVLink bridge joins at most four H200 NVL, so an eight-card server has two domains of four bridged cards with PCIe between them. On this server two sharded jobs, each confined to one domain, avoid PCIe for their heaviest traffic: a 70B LoRA at 36.1 GB per card on cards 0 to 3, a second run on cards 4 to 7. Check which cards share a bridge with nvidia-smi topo -m before you set CUDA_VISIBLE_DEVICES, as our H200 NVL NVLink bridge guide shows.

Runs that need all eight cards are full fine-tunes: a 32B model at 65.5 GB per card with 32-bit Adam, a 70B model at 88.2 GB per card with 8-bit Adam, by the figures on our fine-tuning server page. In Axolotl, TRL and LLaMA-Factory these run with FSDP2 or ZeRO-3 across both domains. For two or more such servers, Axolotl documents multi-node training with torchrun or Ray, TRL with Accelerate and Slurm, and LLaMA-Factory with NNODES and its Megatron-core backend. Our page on the eight-card H200 NVL server describes the hardware.

Which framework for which team

One run per card, many experiments. Unsloth or any of the others, one job per card. Unsloth's kernels make each single-card run faster, and four or eight cards run four or eight jobs.

LoRA and QLoRA that fit one card, one run as fast as possible. DDP in any of the four. Unsloth's DDP is documented and measured; in Axolotl it is the default.

Runs that do not fit one card. Axolotl or TRL with FSDP2, or DeepSpeed ZeRO-3 in any of the three. Axolotl gives the most configuration options in one YAML file; TRL gives the most control in Python; LLaMA-Factory gives one CLI for many model families.

Long sequences. TRL or Axolotl with context parallelism, on an FSDP2 or DeepSpeed setup as each project requires.

The framework runs on the operating system, drivers and CUDA of the server. The training platform around it, with queues, data access and MLOps on Kubernetes, is part of our Private AI/ML service, built by our engineering partner Vixen.UNO. We supply the servers with RTX PRO 6000 Server Edition or H200 NVL cards, assembled and tested before delivery, on one EU contract and invoice; quantities and lead time are confirmed in writing with the quote. Describe your runs and the framework your team uses, and we reply within one business day.

FAQ