Fine-tuning or RAG: which one puts your company’s knowledge into a model

What RAG and fine-tuning each change

Use RAG for company knowledge that changes or has to be cited, and fine-tuning for output format, tone, task behaviour and domain skill; the two can be combined. Fine-tuning changes the model’s parameters, while RAG changes what the model receives at query time.

Retrieval-augmented generation, defined in the original RAG paper of May 2020, pairs the model’s parametric memory with a searchable non-parametric one. In its authors’ words, providing provenance for a model’s decisions and updating its world knowledge were open research problems. Anthropic’s glossary describes the mechanism in operational terms: documents retrieved at runtime are placed in the context window. No weights change.

Fine-tuning works the other way. NVIDIA’s customisation guide states that fine-tuning updates the pretrained weights, while prompt-based techniques leave them frozen. Anthropic’s glossary describes the result as a model that represents and imitates the patterns of the tuning dataset, which means it learns patterns rather than facts.

LoRA, from a June 2021 paper, freezes the base model and injects trainable low-rank matrices into each transformer layer. Against GPT-3 175B fine-tuned with Adam, the paper reports 10,000× fewer trainable parameters and GPU memory cut by a factor of three: training memory fell from 1.2 TB to 350 GB and the checkpoint to 35 MB. It also reports a 25 per cent training speed-up and no added inference latency. Two conditions of these figures matter for any comparison. The parameter reduction was measured at rank 4 on the query and value projections alone, whereas Hugging Face’s PEFT documentation notes that adapting all linear layers, as QLoRA does, “can provide performance equal to a fully finetuned model”. The baseline is also the authors’ own.

QLoRA, from a May 2023 paper, applies LoRA to a base model quantised to four bits. The paper tuned a 65B model on one 48 GB GPU while preserving 16-bit task performance; 16-bit tuning of the same model needed more than 780 GB. It uses three mechanisms: a four-bit data type called NormalFloat, a second quantisation pass that reduces the quantisation constants from 0.5 to 0.127 bits per parameter, and paged optimisers. Its figure of 99.3 per cent of ChatGPT’s performance applies to the Vicuna benchmark only.

Decision table: which technique solves which problem

The table maps each problem to the technique that solves it; the last column says how firmly the published results settle it.

The access-control row is our reasoning rather than a finding of the papers below: retrieval can be filtered per user, weights cannot, which is why our RAG guide applies the filter at retrieval.

OpenAI explicitly advises against fine-tuning for facts. One of its cookbook articles says “We generally do not recommend it as a way to teach the model knowledge”, and its accuracy guide advises optimising the context when knowledge is missing from training, out of date or proprietary. Anthropic’s glossary defines both techniques without ranking them, and we found no NVIDIA rule for choosing parameter-efficient tuning over full tuning. The research below supports the same rule with measurements. The only published threshold we found is Anthropic’s, written for Claude in September 2024: for a corpus under 200,000 tokens, about 500 pages, put the whole corpus in the prompt and build no RAG. On a private Llama 3.3 70B that much context does not fit: the model’s context window is 128K tokens (131,072), and a full window takes 20 GiB of FP8 KV cache per concurrent request by our arithmetic, at 160 KiB per token. For larger corpora, or where that cache does not fit the memory of your GPUs, our guide to RAG on company data covers how to build retrieval.

If you choose RAG, assistants that answer from your documents, cite the source and respect each user’s access rights are part of our private AI/ML service. Describe the documents and who may see which of them; the first call is free of charge.

What fine-tuning costs in GPU memory

Training memory is dominated by model states, and the published multipliers differ: 16 or 18 bytes per parameter, which moves every figure by an eighth.

The ZeRO paper breaks mixed-precision Adam down into two bytes per parameter of fp16 weights, two of fp16 gradients, and an fp32 master copy, momentum and variance at four bytes each, sixteen in total. Hugging Face’s memory-anatomy page lists the same components and totals 18, assuming fp32 gradients rather than fp16. The QLoRA paper gives only a lower bound: 16-bit tuning of a 65B model “requires more than 780 GB”, which is 65 billion times 12 bytes, the count without that master copy. We therefore work with a range of 16 to 18 bytes per parameter.

Our arithmetic from the multipliers above, not vendor or paper figures, for Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B at the parameter counts Hugging Face reports. GB = 10⁹ bytes; model states only, activations on top. LoRA and QLoRA with rank-16 adapters on all linear layers at 16 bytes per trained parameter, on a frozen bf16 base or, for QLoRA, on linear layers at 0.516 bytes per parameter (NF4 with double quantisation, both of which have to be switched on explicitly in Hugging Face Transformers) with the embeddings and output layer in bf16. Our fine-tuning memory guide shows the working.

With 32-bit Adam, therefore, no full fine-tune of an 8B, 32B or 70B model fits on one 96 GB card, even before activations, which grow with batch size and sequence length. A 70B LoRA does not fit either, while a 70B QLoRA does. The belief that a 7B model can be fully tuned on one card confuses inference memory with training memory: the model needs 14 GB to serve at FP16 and about 112 GB to tune fully, a ratio of eight to one. An eight-bit optimiser reduces the gap. Hugging Face puts 8-bit AdamW at 2 bytes per parameter of optimiser state instead of 8, which brings the 8B model’s states to about 80 GB by the ZeRO count, within one 96 GB card before activations.

QLoRA saves memory at the cost of speed, and NVIDIA publishes both figures on one page. It states that QLoRA is “up to” 60 per cent more memory-efficient than LoRA, a vendor claim about its own product, and that the saving shrinks as batch size grows. Directly below, it states that QLoRA training is 50 to 200 per cent slower than LoRA.

NVIDIA’s scheduling matrix for NeMo Customizer shows how many GPUs it allocates to each tuning job, and up to what sequence length.

NVIDIA’s NeMo Microservices configuration matrix, version 25.12.0: configurations are labelled “+40GB”, and the page names L40, A100 and H100 without assigning them to rows.

NVIDIA schedules two GPUs for an 8B LoRA job that the weight arithmetic puts at about 17 GB. The difference is activations, framework overhead and service defaults, and it is what you have to procure for. The matrix lists no all-weights configuration for any 70B model. Serving works the other way round: vLLM serves many LoRA adapters against one copy of the base weights, and each adapter is small, 35 MB in the LoRA paper’s GPT-3 example and, by our arithmetic, about 84 MB for a rank-16 adapter on all linear layers of Llama 3.1 8B at 16-bit. An additional tenant therefore adds little memory.

What RAG costs to run: embedding, reranking and the KV cache

RAG needs no training run; its costs arise in operation, in the embedding service, the reranker, the vector store and the KV cache.

The embedding model is a GPU service of its own. NVIDIA’s retrieval NIM matrix asks a minimum of 3.6 GiB of GPU memory for Llama Nemotron Embed 1B v2 in the unoptimised FP16 profile, and the figures on that page vary sixteenfold, from 2 GiB to 33 GiB for bge-m3. NVIDIA’s RAG blueprint runs further services next to the language model: embedding and reranking microservices, document extraction and a vector store, which has been Elasticsearch by default since release 2.6.0, with Milvus as an option. Its default deployment asks for three GPUs, two of them for the language model, its Kubernetes chart for eight, and each optional GPU service, such as captioning or audio, adds one more.

Chunking defaults differ even within one library: LlamaIndex’s library constant is 1,024 tokens with 20 tokens of overlap, while its own sentence splitter uses 200, and LangChain’s base splitter defaults to 4,000 characters with 200, while its own tutorial uses 1,000 and 200. NVIDIA’s RAG primer contains an error that is often repeated: it says OpenAI’s embedding models encode up to 1,536 tokens and truncate silently beyond that. Neither part holds for OpenAI’s API, which rejects longer input with an error rather than truncating it. OpenAI documents 8,192 tokens of input, and 1,536 is the output vector dimension of its small model.

Reranking improves retrieval measurably. Anthropic measured a top-20 retrieval failure rate of 5.7 per cent on its test sets of code, fiction and research papers. It fell to 3.7 per cent with contextual embeddings, to 2.9 with contextual BM25 added and to 1.9 when the top 150 results were reranked to 20. These are the relative reductions of 35, 49 and 67 per cent that are usually quoted without the 5.7 per cent base. For the vector store, Milvus’s documentation counts 4 bytes per dimension for 32-bit vectors, so a million 128-dimensional vectors take 512 MB before index overhead; the raw store can be estimated and the index measured.

The KV cache decides the serving cost. NVIDIA’s KV cache formula is two, times the layer count, times the hidden size, times the bytes per element, per token. For Llama 2 7B at 16-bit that is 2 × 32 layers × 4,096 × 2 bytes, or 512 KiB per token. Retrieved passages are tokens like any others, so 4,000 tokens of context per request costs about 2 GiB of KV cache per concurrent request, and about 62 GiB at 32 concurrent requests: two thirds of a 96 GB card spent on retrieved text before a token is generated. That multiplication is ours, and it assumes multi-head attention as in Llama 2 7B. Current models of that size use grouped-query attention, with far fewer key-value heads and a proportionally lower cost per token. Llama 3.1 8B has 8 key-value heads against 32 query heads, per Meta’s Llama 3 paper. By our arithmetic, the same 4,000 tokens then cost about 0.5 GiB at 16-bit, 128 KiB per token, and about 15.6 GiB at 32 concurrent requests. Run the formula against your own model’s configuration. Our article on how many users one RTX PRO 6000 can serve works the arithmetic end to end.

We build GPU servers for private LLMs and RAG, sized by model size and concurrent users. Write to us with the model and the retrieved context per request, and we reply with a configuration and a quote within one business day.

What published studies found: retrieval against fine-tuning

An EMNLP 2024 study, “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs”, tested Mistral 7B, Llama2 7B and Orca2 7B on MMLU subsets and on a current-events task on knowledge the models could not have seen. With Mistral 7B on current events the base model scored 0.481, base plus retrieval 0.875, the fine-tuned model 0.504 and the fine-tuned model with paraphrases 0.588. It concludes that “LLMs struggle to learn new factual information through unsupervised fine-tuning”.

The study’s fine-tuning arm was unsupervised continued pretraining on raw text, not supervised tuning on question and answer pairs, which is what most companies do; the paper says so in its abstract. Applying the result to supervised tuning is a common misreading. The study shows something narrower: a continued-pretraining run on a document corpus does not reliably put those facts into the weights.

The counter-result is an agriculture case study of January 2024, which used supervised tuning and found the two additive: over 6 percentage points from fine-tuning and a further 5 from retrieval on top, retrieval effective on contextually relevant data and tuning at teaching domain skills. The two results do not agree. The EMNLP study’s tables show base plus retrieval ahead of fine-tuned plus retrieval, 0.681 against 0.659 on anatomy at zero shot with Mistral 7B; the agriculture study’s tables show the gains adding up. The studies used different tasks, domains and tuning regimes.

Two further studies are relevant. A SIGIR-AP 2024 study ran twelve models from 80M to 11B parameters and found retrieval ahead by a large margin on low-popularity facts; a small fine-tuned model with retrieval matched a much larger one, 1.6B against 8B: the logic behind running a private model rather than a cloud API. And a TMLR paper of August 2024, “LoRA Learns Less and Forgets Less”, compared LoRA with full fine-tuning on programming and mathematics: at standard low ranks LoRA substantially underperforms it, the mechanism proposed being that full tuning learns weight changes of 10 to 100 times the rank. LoRA’s own paper claims parity or better. The two peer-reviewed papers disagree, so test a higher rank before you rule LoRA out.

Failure modes and maintenance of fine-tuning and RAG

Fine-tuning’s characteristic failure is catastrophic forgetting, and the finding most relevant for planning concerns model size. A 2023 study of catastrophic forgetting in continual fine-tuning measured it across domain knowledge, reasoning and reading comprehension in models from 1B to 7B and found it general and, within that range, intensifying as the models grew. Up to 7B, a bigger base model was no mitigation; instruction tuning before the domain tune did help. LoRA is a partial answer, with a documented trade-off: the TMLR paper that found it weaker on the target domain found it preserving abilities outside that domain better than weight decay or dropout.

RAG’s failures are named and counted. A CAIN 2024 experience report from three case studies lists seven: content missing from the corpus, the right document ranked too low, the right document dropped in consolidation, the answer in context but not extracted, output ignoring the requested format, the wrong specificity, and an answer correct but incomplete. The fifth, wrong format, is a fine-tuning problem within a RAG failure taxonomy. The report concludes that a RAG system can only be validated in operation, and the ongoing work is tuning its parameters: chunk size, embedding, chunking, retrieval and consolidation strategy, context size and prompts.

A longer context window does not compensate for weak retrieval. The TACL paper “Lost in the Middle” found performance highest when the relevant passage sits at the start or end of the context and degraded in the middle, even in long-context models: adding fifty chunks to the context instead of fixing the ranking can make results worse. Both paths need evaluation. OpenAI advises building evaluations before tuning; for retrieval, Ragas at EACL 2024 scores relevance, faithfulness and generation quality without ground truth.

Combining RAG and fine-tuning, and how much training data you need

NVIDIA recommends a sequence rather than a choice: start with RAG, then customise if you need more domain-specific accuracy. Its RAG primer states that the two are not alternatives and describes a combination that is often overlooked: parameter-efficient tuning inside a RAG system, refining the language model or the embedding model. Tuning the retriever rather than the generator is discussed less often.

Taken together, the sources support this order: evaluations first; prompt and context engineering; measure the corpus and skip retrieval only where it fits the model’s context window and your KV cache budget per request; build retrieval above that; only then reach for LoRA to fix format, tone, task compliance and domain skill; tune the embedding model last. OpenAI adds one instruction: carry the prompt that worked best before tuning into every training example.

For teaching behaviour, relatively little data is needed. OpenAI states a hard minimum of 10 examples, reports improvements from 50 to 100, and recommends starting with 50 well-crafted demonstrations; its quality bar is that a smaller amount of good data beats a larger amount of poor data, and that examples must be self-contained, consistent and in the format expected at inference. LIMA fine-tuned a 65B model on 1,000 curated prompts and was preferred or equivalent to GPT-4 in 43 per cent of comparisons; its authors conclude that the results “strongly suggest” almost all knowledge is learned during pretraining and only limited instruction data is needed for good output. The QLoRA authors agree that quality beats dataset size. OpenAI’s figures come from its self-serve fine-tuning platform, which OpenAI is winding down. According to its deprecations page, organisations that had not run fine-tuning before have been unable to create fine-tuning jobs since 7 May 2026, and existing customers can create no new jobs from 6 January 2027.

Facts need far more data than behaviour. The EMNLP 2024 study found accuracy rising monotonically with the number of paraphrases used: each fact has to be restated many times over, and the result still falls short of retrieval. For scale, the TMLR paper’s two data regimes were about 100,000 prompt and response pairs for instruction tuning and about 20 billion unstructured tokens for continued pretraining; those are the sizes it tested, not minimums. For two figures we found no published number, and we do not estimate them. We found no minimum example count in NVIDIA’s NeMo documentation, and Anthropic’s July 2024 announcement of Claude 3 Haiku fine-tuning in Amazon Bedrock asked only for high-quality prompt and completion pairs, with no quantity (its own API offers no fine-tuning as of October 2026); that announcement’s accuracy figure, from 81.5 to 99.6 per cent on moderating online comments, is Anthropic’s own test on a single task and says nothing about how many examples a task needs.

What we do

We supply the GPUs and servers for either path across the EU and size them from your corpus, your model and your peak concurrency rather than from the name of a technique. Our engineering partner Vixen.UNO builds and supports the platform on top as part of our private AI/ML service, from RAG assistants that respect each user’s access rights to MLOps on Kubernetes, under one contract with Eurokommerz.

FAQ