Continued pretraining on company data: GPU time, memory and data preparation on 8 GPUs

Can a company run continued pretraining on its own server

Continued pretraining, also called domain-adaptive pretraining, means training an open base model further on a large amount of raw text, such as a company's manuals, reports and correspondence, with the same next-token objective it was pretrained with. On one eight-card server it is feasible for models up to about 32B parameters and token counts in the low billions: by our arithmetic an 8B model takes about two days per 10 billion tokens on eight NVIDIA H200 NVL. Plan the data preparation as a project of its own.

The calculation behind this is short, and it decides the server. This page works it through for 8B, 32B and 70B models on eight H200 NVL and on eight NVIDIA RTX PRO 6000 Blackwell Server Edition cards, then covers memory, data preparation and what continued pretraining does and does not teach. The supervised fine-tuning of smaller datasets is the subject of our guide to the fine-tuning server with 2, 4 or 8 GPUs, and a full fine-tune of one model in detail is on our page on full fine-tuning of Qwen3.8-27B.

For a director, three numbers matter: how many tokens of usable text the company has, which model size it wants to adapt, and how many days the server may be busy. Start with the first. Anthropic's guidance quoted in our fine-tuning or RAG comparison puts 200,000 tokens at about 500 pages, so 10 billion tokens would be about 25 million pages by our arithmetic. Count the tokens of a sample with the model's own tokeniser before planning a run.

GPU time by 6·N·D

The training compute of a dense transformer is about six floating-point operations per parameter and token. EleutherAI's "Transformer Math 101" writes it as C ≈ 6PD, with about 2PD for the forward pass and 4PD for the backward pass. For Llama 3.1 8B, with 8.03 billion parameters, 10 billion tokens come to 6 × 8.03 × 10⁹ × 10¹⁰ ≈ 4.8 × 10²⁰ FLOP. For comparison, Meta's model card states that Llama 3.1 was pretrained on "~15 trillion tokens" and lists 1.46 million H100 GPU hours for the 8B model.

To turn FLOPs into days, divide by the peak rate of the cards and by the share of it a run achieves, its model FLOP utilisation (MFU). NVIDIA's Megatron-LM README reports "up to 47% Model FLOP Utilization (MFU) on H100 clusters", with MFU rising "from 41% for the smallest model considered to 47–48% for the largest models" in its weak-scaling runs. Those are tuned runs on large H100 clusters. On a PCIe server with NVLink bridges or without NVLink, and with general-purpose frameworks, plan lower. We show 40 per cent as the optimistic case and 20 per cent as a cautious one; your own short test run gives the real figure.

The peak rates are dense BF16: 835.5 TFLOPS per H200 NVL, half the 1,671 TFLOPS NVIDIA lists with sparsity, and 503.8 TFLOPS for the RTX PRO 6000 Workstation Edition from NVIDIA's architecture whitepaper. NVIDIA gives no dense rate for the Server Edition, which runs at lower clocks, so its figures below are a ceiling.

Our arithmetic: C = 6 × parameters × tokens (8.03, 32.76 and 70.55 billion parameters); time = C ÷ (8 × dense BF16 peak × MFU), at MFU 40 % and 20 %. Peaks 835.5 TFLOPS (H200 NVL) and 503.8 TFLOPS (Workstation Edition, a ceiling for the Server Edition).

The time scales linearly with tokens. One billion tokens on the 8B model takes about 5 to 10 hours on eight H200 NVL and at least 8 to 17 hours on eight Server Edition cards. FP8 training doubles the peak rate on both cards, but we found no published utilisation figures for FP8 training on them, so the table stays in BF16.

Activation recomputation adds work. EleutherAI's post bounds it by "a full additional forward pass". MFU counts only the model's 6PD, so recomputation shows up as a lower MFU, not as extra FLOPs in the formula, and the table does not add it again.

We build eight-card training servers with H200 NVL or RTX PRO 6000 Server Edition cards; send us your token count and the model you want to adapt, and our team replies with the run time we estimate and the configuration for it.

Memory for full-parameter training

This section assumes that all weights are trained, as in the original pretraining. We use the same rule as our guide to GPU memory for fine-tuning: 16 bytes per parameter with mixed-precision Adam, which EleutherAI's post splits into 2 bytes of 16-bit weights, 2 bytes of gradients and 12 bytes of optimiser states, including a 32-bit copy of the weights. Sharded with FSDP or DeepSpeed ZeRO-3 across eight cards:

Our arithmetic from the model states on our fine-tuning pages; 8-bit Adam at about 10 bytes per parameter; memory left against listed capacity (96 and 141 GB). Activations come out of what is left.

Pretraining runs on long, packed sequences, so activations take a large share of what is left; activation checkpointing trades them for the recomputation above. The 8B model leaves room on both servers. The 32B model fits eight Server Edition cards with about 30 GB per card for activations. The 70B model is a job for eight H200 NVL with 8-bit Adam, or with the optimiser in host memory, which our fine-tuning server page puts at about 988 GB for Llama 3.3 70B; on eight Server Edition cards, 7.8 GB left per card is not a plan.

Traffic between the cards is smaller than it looks for long runs. A sharded step gathers the 16-bit weights twice and reduces the gradients once, about 3 × 16.1 GB × 7/8 ≈ 42 GB per card per micro-batch for the 8B model, about 0.66 seconds at 64 GB/s over PCIe Gen5. A micro-batch of 16,384 tokens per card takes about 3.9 seconds of computation at 40 per cent of 503.8 TFLOPS. By our arithmetic the traffic is therefore about a sixth of the computation even without NVLink, and FSDP overlaps part of it. On eight H200 NVL, the cards inside each domain of four are bridged with NVLink, but a run sharded over all eight still crosses PCIe between the two domains, so the PCIe figure is the planning figure there too, as our page on the H200 NVL NVLink bridge explains.

LoRA instead of full training

LoRA can be used for continued pretraining as well. Unsloth's continued-pretraining guide adds lm_head and embed_tokens to the trained modules, uses rank 16 and gives the embeddings a learning rate "2-10x smaller", for example 5e-5 for the adapters and 5e-6 for the embeddings, for cases such as teaching a model a new language. The memory need drops to the frozen base plus the adapters.

The trade-off is documented. The paper "LoRA Learns Less and Forgets Less" (May 2024) compared both methods on programming and mathematics, including continued pretraining on 20 billion unstructured tokens, and found that "LoRA substantially underperforms full finetuning" in standard low-rank settings, while it "better maintains the base model's performance on tasks outside the target domain". The compute per token falls to about 4N with the base frozen, forward plus input gradients, by the same arithmetic as our gpt-oss fine-tuning page; a LoRA run of the 8B model on 10 billion tokens therefore takes about two thirds of the time in the table above.

Data preparation: deduplicate, filter, tokenise

The pipeline from documents to training tokens is a project of its own. It runs on processors and, for some steps, on GPUs.

Extract and clean. Convert PDFs, office documents and e-mail to plain text, remove boilerplate such as signatures, headers and navigation, and drop documents your legal counsel and data-protection officer have not cleared for training.

Deduplicate. Company archives hold many copies and near-copies of the same document. NVIDIA's NeMo Curator, Apache 2.0, offers "exact/fuzzy/substring deduplication" on GPUs through RAPIDS and Ray; its README reports fuzzy deduplication of a RedPajama v2 subset in 0.65 hours on three H100 nodes against 10.7 hours for a processor-based alternative, and its GPU text path needs about 16 GB of GPU memory. Hugging Face's datatrove, also Apache 2.0, is "a library to process, filter and deduplicate text data at a very large scale" with MinHash, sentence and exact-substring examples, and runs locally, on Slurm or on Ray.

Filter and classify. NeMo Curator lists classification, quality filtering, ensemble quality classifiers and language detection for text. Decide which document types belong in the corpus: a thousand copies of a form letter teach the model little.

Tokenise and pack. Tokenise with the base model's tokeniser and store the token IDs. By our arithmetic, 10 billion tokens take 40 GB at 4 bytes per token ID; 2 bytes suffice only for vocabularies of at most 65,536 entries, and Llama 3.1 has 128,256. Checkpoints take more space than the data: a resumable checkpoint of an 8B full training holds about 112 GB at 14 bytes per parameter, by the figures on our fine-tuning server page.

Forgetting, learning rate and data mixing

A model trained only on company text loses some of its general ability. The study "Simple and Scalable Strategies to Continually Pre-train Large Language Models" (2024) found that "a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data" matched training from scratch, tested up to a 10B-parameter model, "saving significant compute compared to re-training". Replay means mixing general text of the kind the model was originally trained on into your corpus; the abstract gives no fixed share, so test the mix on a small run with your evaluation set.

The gains are real for language and style. "Don't Stop Pretraining" (2020) found that "a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains" across four domains, among them biomedical and computer science publications.

Facts are another matter. The EMNLP 2024 study discussed in our fine-tuning or RAG comparison ran unsupervised continued pretraining on new material and concluded that models "struggle to learn new factual information" this way; retrieval did better. Use continued pretraining to teach the language of your domain and keep retrieval, as in our guide to RAG on company data, for the facts that change. Whether a modification of this size has consequences under the EU AI Act is covered in our page on the AI Act and fine-tuning; your legal counsel confirms the assessment.

Which server for continued pretraining

8B models, up to a few billion tokens. Eight RTX PRO 6000 Server Edition cards, at least 3.5 days per 10 billion tokens by our arithmetic, with 80 GB per card left for activations. Our page to buy an 8× RTX PRO 6000 Server Edition server lists the configuration.

8B to 32B models, or repeated runs. Eight H200 NVL with two 4-way NVLink bridges, 2.3 TB of host memory and eight NVMe drives, our training configuration: about 2.1 days per 10 billion tokens for the 8B model at 40 per cent MFU, 8.5 days for the 32B model.

70B models. Eight H200 NVL with 8-bit Adam or optimiser offload, 18 days or more per 10 billion tokens, or two such servers on the 400G network. Our page on the eight-card H200 NVL server describes the hardware, and our page to buy an 8× H200 NVL server the order.

Eight H200 NVL at 600 W with two 64-core AMD EPYC 9555 processors draw about 6.3 kW at the wall by our estimate (cards and processors plus 15 per cent for the rest), so the 50-hour run of the 8B model uses about 315 kWh; a measurement on the delivered system confirms the figure.

Ordering a training server from us

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. The server is assembled and tested before delivery, delivered in the EU on one contract and invoice, with manufacturer warranty on every component and the assembly covered by us. On request we install the operating system, drivers, CUDA and a container runtime. The data pipeline and training platform on top are part of our Private AI/ML service, built by our engineering partner Vixen.UNO. Quantities and lead time are confirmed in writing with the quote. Describe your corpus, the base model and the time the run may take, and we answer with a configuration and a written quote.

FAQ