LLM distillation on-premise: hardware for teacher and student, trainers and licences

What LLM distillation does: a teacher, a student and your tasks

Knowledge distillation trains a small model, the student, to reproduce what a large model, the teacher, does on your own tasks. The result is a model of 8B or less that answers your support tickets, extracts your invoice fields or classifies your documents close to the way a 70B or 120B model does, and runs on a fraction of the hardware. On-premise the work runs in two phases on GPU servers you own, so the company data in the prompts stays inside your network: the teacher generates answers or probabilities as a batch job, and the student is trained on them.

Distillation narrows a model to the tasks in its training data. A student trained on support tickets does not become a general assistant, and it is usually weaker than the teacher outside those tasks, so the method pays off for a few high-volume, well-defined tasks. Our comparison of fine-tuning and RAG covers when to train a model at all; this article covers the two phases of distillation, the hardware each needs and what the teacher's licence says.

Two ways to distil: teacher outputs or teacher logits

Sequence-level distillation is supervised fine-tuning on texts the teacher wrote. You send your prompts to the teacher, keep the answers that pass your checks, and train the student on the prompt-answer pairs with an ordinary SFT trainer. TRL's documentation describes this case in the options of its experimental GKDTrainer: seq_kd performs "Sequence-Level KD (can be viewed as supervised FT on teacher-generated output)". The teacher and the student can come from different model families, because only text passes between them.

Logit distillation trains the student on the teacher's probabilities for every next token, which carry more information than one sampled answer. TRL implements it in several trainers (documentation read on 24 October 2026):

Hugging Face TRL documentation (main branch; latest release 1.16.0), read on 24 October 2026. The experimental trainers are imported from trl.experimental, for example trl.experimental.gkd. GKD and DistillationTrainer implement on-policy distillation after arXiv 2306.13649; GOLD extends Universal Logit Distillation.

The DistillationTrainer page states that the teacher "must share the student's vocabulary". Its method is on-policy: the student generates its own completions and learns to match the teacher's next-token distribution over them, with a chunked Jensen-Shannon loss "so the teacher's dense distribution is never materialized in full". GOLDTrainer "supports student/teacher pairs with different tokenizers" by aligning text spans and merging their logits, for example "LLaMA students with Qwen teachers"; it sits in TRL's experimental namespace, where "APIs may change without notice". NVIDIA's Model Optimizer offers distillation and Minitron-style pruning with distillation for Hugging Face, Megatron-Bridge and Megatron-LM, for teams that train on NVIDIA's stack.

For most company tasks, sequence-level distillation is the place to start. It needs no special trainer, any teacher licence permitting, and its training data can be read, filtered and corrected by people before a single GPU-hour goes into the student.

Phase 1: the teacher generates training data

Teacher generation is batch inference: a fixed set of prompts, no user waiting, sized by tokens per second. Our guide to batch LLM inference on-premise describes the vLLM settings and the method; the published throughput of a comparable model gives a first estimate.

MLPerf® Inference: Datacenter v6.1, closed division, available category, gpt-oss-120b, offline scenario, entry 6.1-0050, HPE ProLiant Compute DL380a Gen12 with eight RTX PRO 6000 Blackwell Server Edition, 15,256 tokens per second, retrieved from mlcommons.org on 24 October 2026. Result verified by MLCommons Association. Hours are our arithmetic, not an MLPerf metric; the margin plans with half the published rate, as in our batch inference guide; answer counts and lengths are example values. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The 2,000-token row is there for reasoning teachers, which write a long chain of thought before the answer; keep or strip it depending on whether the student should learn to reason. Generate more than you need, because filtering removes part of it: answers that fail a schema check, contradict the source document or repeat each other. A dense 70B teacher in FP8 holds 12 sessions of 8,192 tokens on one RTX PRO 6000 Server Edition by our users-per-card rule, so a single card can generate while the rest of the server serves users, at a lower rate.

Storing logits instead of text changes the scale. A full next-token distribution over a vocabulary of 128,256 tokens in 16-bit values is about 256 KB per token, so 120 million tokens would need about 31 TB, by our arithmetic. Keeping only the top 8 tokens, the default that AsyncDistillationTrainer sends, at an assumed 8 bytes per entry comes to about 64 bytes per token, or 7.7 GB. This is why TRL's logit trainers query the teacher during training instead of reading a stored file.

We supply servers with two to eight RTX PRO 6000 Server Edition cards or H200 NVL for both phases; send us your teacher, your task and the number of examples, and our team replies with a configuration.

Phase 2: training the student

Sequence-level distillation trains the student like any fine-tune, and our guide to GPU memory for fine-tuning gives the figures for Llama 3.1 8B: 128 GB of model states for a full fine-tune with mixed-precision Adam, about 80 GB with 8-bit Adam, which fits one 96 GB RTX PRO 6000 Server Edition, 32.1 GB per card with FSDP over four cards, and 16.7 GB for LoRA. A student that should reproduce the teacher closely on its tasks is usually trained in full; LoRA fits small budgets and many separate tasks. Which training framework spreads such a run over several cards is compared in our page on Unsloth, Axolotl and TRL for multi-GPU fine-tuning.

The compute is modest. With about 6 FLOP per parameter and training token, the approximation of the scaling-laws paper (arXiv 2001.08361), 200 million tokens of prompts and answers trained for three epochs come to 6 × 8.03 × 10⁹ × 6 × 10⁸ ≈ 2.9 × 10¹⁹ FLOP for an 8B student. On eight H200 NVL at an assumed 40 per cent of NVIDIA's 835.5 dense BF16 TFLOPS per card, that is about three hours, by our estimate.

Logit distillation adds the teacher's forward pass for every training token. The same paper puts the forward pass at about a third of training compute, 2 FLOP per parameter and token, so a 70B teacher adds about 2 × 70.55 × 10⁹ ≈ 1.4 × 10¹¹ FLOP per token, three times the 4.8 × 10¹⁰ of the 8B student's training step. On-policy trainers also let the student generate its completions during training. By our estimate a logit-distillation run with a 70B teacher therefore needs about four times the compute of sequence-level training on the same tokens, before generation.

Memory follows the layout. With the teacher loaded in the training process, a 70B teacher in BF16 adds about 141 GB of weights to the student's states; sharded over eight H200 NVL, that is about 17.6 GB of teacher weights per card next to 16 GB of student states. With AsyncDistillationTrainer the teacher runs as its own vLLM server, which its documentation says "can be a much larger model than would otherwise fit alongside the student"; teacher, student generation and trainer run on separate GPUs, the trainer supports only FSDP2, and vLLM 0.22.0 or later is required. On an eight-card RTX PRO 6000 Server Edition server, one layout by our example puts Llama 3.3 70B in FP8 on two cards, the student's vLLM server on one and the trainer on five, with a student whose tokenizer files match the teacher's.

The teacher's licence and the student

Whether the teacher's outputs may train another model is a question for the teacher's licence. Read the licence file of the exact checkpoint; the terms below were read on 24 October 2026.

Llama 3.3 LICENSE file on Hugging Face, clause 1.b.i; Gemma Terms of Use, Section 1.1(e) and (f), 3.3; gpt-oss-120b model card on Hugging Face. Our summary; your legal department reads the full terms.

The Llama 3.3 licence says that if you use the Llama Materials "or any outputs or results" of them to create, train or fine-tune an AI model "which is distributed or made available", its name must begin with "Llama". By our reading, a student kept for internal use is neither distributed nor made available; one offered to customers or published is. Google's Gemma Terms of Use define Model Derivatives to include any model trained to perform similarly to Gemma, "including distillation methods" and the generation of "synthetic data Outputs by Gemma for training that model", so the student inherits the terms' use restrictions; the same terms state that "Outputs are not deemed Model Derivatives" and that Google "claims no rights in Outputs". The gpt-oss-120b card describes a "Permissive Apache 2.0 license".

The student keeps its own licence as well: an 8B Llama student remains under the Llama licence of its base model, a Qwen or Mistral student under Apache 2.0 where its card says so. Our comparison of open LLM licences for company use sets out the other licences. On the AI Act, the training compute of a typical distillation run is far below the one-third criterion of the Commission's guidelines for modified models, as our article on AI Act fine-tuning and GPAI providers works through.

What the student saves when it serves

The point of distillation is the serving hardware. Our published sizing for both models, by our users-per-card rule:

Our estimates: conversations from FACTS-based sizing in our VRAM guide (70B) and small-model guide (8B, 9.1 GB NVIDIA FP8 checkpoint, 128 KiB of 16-bit cache per token; on the L4 a budget of 17.3 GiB with ECC on and 18.6 GiB with ECC off, as in that guide); 25.7 GiB per RTX PRO 4500 Server Edition by our rule. Speed ceilings are bandwidth divided by the weights read per token (1,597 GB/s ÷ 70.6 GB, 300 GB/s ÷ 9.1 GB, 800 GB/s ÷ 9.1 GB), upper bounds, not measurements.

A 72 W L4 serving the student has a higher single-user ceiling than an RTX PRO 6000 Server Edition serving the teacher, because it reads an eighth of the bytes per token. Eight L4 or eight RTX PRO 4500 Server Edition cards in one server then hold eight student copies, or a mix of students for different tasks. Our guide to small language models on L4 and RTX PRO 2000 covers branch and edge servers for such models.

Checking the student before it replaces the teacher

Keep a test set the student never saw in training, drawn from real requests, and score teacher and student on it with the same metric: exact match for extraction fields, accuracy for classes, graded answers for text. Add cases outside the training distribution, since a narrow student fails there first, and route those requests to the teacher or a general model. Repeat the comparison whenever the teacher, the prompts or the task change, and keep the generated training set with its filters, so the student can be retrained.

Our Private AI/ML service sets up the teacher generation, the training and the serving of the student on your own servers, together with our engineering partner Vixen.UNO. Describe the task and the request volume you want to move to a smaller model, and we propose the hardware for both phases.

FAQ