Fine-tuning or RAG: which one actually puts your company’s knowledge into a model

What each technique actually changes

One technique changes the model’s parameters. The other changes what you put in front of them at query time. Two consultants can argue for an hour without settling that.

Retrieval-augmented generation, defined in the original RAG paper of May 2020, pairs the model’s parametric memory with a searchable non-parametric one. The paper’s own framing is the useful part: providing provenance for a model’s decisions and updating its world knowledge were, in its authors’ words, open research problems. Anthropic’s glossary puts the mechanism operationally, documents retrieved at runtime into the context window. Not one weight moves.

Fine-tuning is the opposite. NVIDIA’s customisation guide draws the line plainly: fine-tuning updates the pretrained weights where prompt-based techniques leave them frozen. Anthropic’s glossary says what that buys, a model that represents and imitates the patterns of the tuning dataset. Patterns, not facts.

LoRA, from a June 2021 paper, freezes the base and injects trainable low-rank matrices into each transformer layer. Its headline claim: against GPT-3 175B fine-tuned with Adam it cut trainable parameters by 10,000× and GPU memory by three times, training memory from 1.2 TB to 350 GB and the checkpoint to 35 MB. Two conditions are usually dropped: that parameter cut was measured at rank 4 on the query and value projections alone, where Hugging Face’s PEFT documentation notes that adapting all linear layers, as QLoRA does, “can provide performance equal to a fully finetuned model”, and the baseline is the authors’ own. It also reports a 25 per cent training speed-up and no added inference latency.

QLoRA, from a May 2023 paper, puts LoRA on a base quantised to four bits: a 65B model tuned on one 48 GB GPU while preserving 16-bit task performance, where 16-bit tuning of it needed more than 780 GB. Its mechanisms: a four-bit type called NormalFloat, a second quantisation pass dropping the quantisation constants from 0.5 bits per parameter to 0.127, and paged optimisers. Its 99.3 per cent of ChatGPT figure is the Vicuna benchmark only.

The decision table, and which rows are settled

Map the problems, not the techniques; the last column is what to read.

The access-control row is our reasoning rather than a finding of the papers below: retrieval can be filtered per user, weights cannot, which is why our RAG guide applies the filter at retrieval.

You will also hear that the model providers say fine-tuning is the wrong tool for facts. OpenAI does: one of its cookbook articles says “We generally do not recommend it as a way to teach the model knowledge”, and its accuracy guide says to optimise the context when knowledge is missing from training, out of date or proprietary. Anthropic’s glossary defines both techniques without ranking them, and we found no NVIDIA rule for choosing parameter-efficient tuning over full tuning. The research below backs the same rule with measurements. The one published threshold is Anthropic’s: under 200,000 tokens of corpus, about 500 pages, put the lot in the prompt and build no RAG. Our guide to RAG on company data covers the rest.

What fine-tuning costs in GPU memory

Training memory is dominated by model states, and the multiplier is disputed: the dispute moves every figure by a third.

The ZeRO paper breaks mixed-precision Adam down: two bytes per parameter of fp16 weights, two of fp16 gradients, then an fp32 master copy, momentum and variance at four bytes each: sixteen in total. Hugging Face’s memory-anatomy page lists the same components and totals 18, assuming fp32 gradients rather than fp16. The QLoRA paper implies 12: its 780 GB for a 65B model is exactly 65 billion times 12 bytes, omitting that master copy. Working range: 16 to 18 bytes per parameter.

Our arithmetic from the multipliers above, not vendor or paper figures, for Llama 3.1 8B, Qwen3-32B and Llama 3.3 70B at the parameter counts Hugging Face reports. GB = 10⁹ bytes; model states only, activations on top. LoRA and QLoRA with rank-16 adapters on all linear layers at 16 bytes per trained parameter, on a frozen bf16 base or, for QLoRA, on linear layers at 0.516 bytes per parameter with the embeddings and output layer in bf16. Our fine-tuning memory guide shows the working.

So: with 32-bit Adam, no full fine-tune of an 8B, 32B or 70B model fits on one 96 GB card, before any activation, and activations grow with batch size and sequence length. LoRA on 70B does not fit either. QLoRA on 70B does. The belief that a 7B fully tunes on one card confuses inference with training memory: that model is 14 GB to serve at FP16 and about 112 GB to fully tune, a ratio of eight to one. An eight-bit optimiser narrows the gap: Hugging Face puts 8-bit AdamW at 2 bytes per parameter of optimiser state instead of 8, which brings the 8B model’s states to about 80 GB by the ZeRO count, inside one 96 GB card before activations.

QLoRA’s saving has a price, and NVIDIA publishes both halves on one page. That QLoRA is “up to” 60 per cent more memory-efficient than LoRA is a vendor claim about its own product, and the saving shrinks as batch size grows. Directly below it, and not marketing: QLoRA training is 50 to 200 per cent slower than LoRA.

The document almost nobody quotes: NVIDIA’s scheduling matrix, on L40 cards of 48 GB.

NVIDIA’s NeMo Microservices configuration matrix; aggregate VRAM is our own multiplication.

NVIDIA schedules two L40 cards, 96 GB, for an 8B LoRA job that the weight arithmetic puts at about 17 GB; the gap is activations, framework overhead and service defaults, and it is what you procure against. And NVIDIA lists no all-weights configuration for any 70B model. Serving runs the other way: vLLM holds many LoRA adapters against one copy of the base weights, and an adapter is small: 35 MB in the LoRA paper’s GPT-3 example, about 84 MB for a rank-16 adapter on all linear layers of Llama 3.1 8B at 16-bit by our arithmetic, so a tenant costs little.

What RAG costs to run, which nobody budgets

RAG is called the cheap option because it needs no training run. That is not where its money goes.

The embedding model is a GPU service of its own: NVIDIA’s retrieval NIM matrix asks a minimum of 3.6 GiB of GPU memory for its 1B embedding model in the unoptimised FP16 profile, and the spread on that page is sixteenfold, from 2 GiB to 33 GiB for bge-m3. Around it sit more services: NVIDIA’s RAG blueprint runs embedding and reranking microservices, document extraction and a Milvus vector store next to the language model. Its default deployment asks for three GPUs, two of them for the language model, its Kubernetes chart for eight, and each optional GPU service, such as captioning or audio, adds one more.

Chunking defaults are not even self-consistent: LlamaIndex’s library constant is 1,024 tokens with 20 tokens of overlap where its own sentence splitter uses 200, and LangChain’s base splitter defaults to 4,000 characters with 200 where its own tutorial uses 1,000 and 200. One repeated error to correct: NVIDIA’s RAG primer says OpenAI’s embedding models encode up to 1,536 tokens and truncate silently beyond. The truncation is real; the number is not. OpenAI documents 8,192 tokens of input, and 1,536 is the output vector dimension of its small model.

Re-ranking pays. Anthropic measured a top-20 retrieval failure rate of 5.7 per cent on its own corpora, falling to 3.7 with contextual embeddings, 2.9 with contextual BM25 added and 1.9 when the top 150 results were reranked to 20: the relative reductions of 35, 49 and 67 per cent quoted without that 5.7 per cent base. We found no per-vector byte count in Milvus’s documentation, so the store gets measured.

Now the figure that decides serving cost. NVIDIA’s KV cache formula is two, times the layer count, times the hidden size, times the bytes per element, per token. For Llama 2 7B at 16-bit that is 2 × 32 layers × 4,096 × 2 bytes, or 512 KiB per token. Retrieved passages are tokens like any others, so 4,000 tokens of context per request costs about 2 GiB of KV cache per concurrent request, and about 62 GiB at 32 concurrent requests: two thirds of a 96 GB card spent on retrieved text before a token is generated. That multiplication is ours, and it assumes multi-head attention as in Llama 2 7B. Current models of that size use grouped-query attention, with far fewer key-value heads and a proportionally lower cost per token: Llama 3.1 8B has 8 key-value heads against 32 query heads, per Meta’s Llama 3 paper, so by our arithmetic the same 4,000 tokens cost about 0.5 GiB at 16-bit, 128 KiB per token, and about 15.6 GiB at 32 concurrent requests. Run the formula against your own model’s configuration. Our article on how many users one RTX PRO 6000 can serve works the arithmetic end to end.

What the published comparisons actually found

An EMNLP 2024 study, “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs”, tested Mistral 7B, Llama2 7B and Orca2 7B on MMLU subsets and on a current-events task on knowledge the models could not have seen. With Mistral 7B on current events the base model scored 0.481, base plus retrieval 0.875, the fine-tuned model 0.504 and the fine-tuned model with paraphrases 0.588. Its conclusion: “LLMs struggle to learn new factual information through unsupervised fine-tuning”.

The load-bearing word is “unsupervised”. That study’s fine-tuning arm was continued pretraining on raw text, not supervised tuning on question and answer pairs, which is what most companies do; the paper says so in its abstract. Generalising it to supervised tuning is the commonest misreading in the field. What it establishes is narrower: pouring a document corpus into a continued-pretraining run does not put those facts reliably into the weights.

The counter-result is an agriculture case study of January 2024, which used supervised tuning and found the two additive: over 6 percentage points from fine-tuning and a further 5 from retrieval on top, retrieval effective on contextually relevant data and tuning at teaching domain skills. The two do not reconcile and we will not pretend they do. The EMNLP study’s tables show base plus retrieval beating fine-tuned plus retrieval, 0.681 against 0.659 on anatomy at zero shot with Mistral 7B; the agriculture study’s show them stacking. Different tasks, domains and tuning regimes.

Two more results. A SIGIR-AP 2024 study ran twelve models from 80M to 11B parameters and found retrieval ahead by a large margin on low-popularity facts; a small fine-tuned model with retrieval matched a much larger one, 1.6B against 8B: the logic behind running a private model rather than a cloud API. And a TMLR paper of August 2024, “LoRA Learns Less and Forgets Less”, compared LoRA with full fine-tuning on programming and mathematics: at standard low ranks LoRA substantially underperforms it, the mechanism proposed being that full tuning learns weight changes of 10 to 100 times the rank. LoRA’s own paper claims parity or better. Both are peer-reviewed and disagree directly, so raise the rank before concluding.

What breaks, and what each one demands afterwards

Fine-tuning’s characteristic failure is catastrophic forgetting, and the finding that should stop a project is about scale. A 2023 study of catastrophic forgetting in continual fine-tuning measured it across domain knowledge, reasoning and reading comprehension in models from 1B to 7B and found it general and intensifying as the models grew. A bigger base model is no mitigation; instruction tuning before the domain tune did help. LoRA is the partial answer and the trade is explicit: the TMLR paper that found it weaker on the target domain found it preserving abilities outside that domain better than weight decay or dropout.

RAG’s failures are named and counted. A CAIN 2024 experience report from three case studies lists seven: content missing from the corpus, the right document ranked too low, the right document dropped in consolidation, the answer in context but not extracted, output ignoring the requested format, the wrong specificity, and an answer correct but incomplete. Note the fifth: wrong format is a fine-tuning problem inside a RAG failure taxonomy. Its verdict is that a RAG system can only be validated in operation, and the monthly burden is its knob list: chunk size, embedding, chunking, retrieval and consolidation strategy, context size and prompts.

A longer context window does not rescue a weak retriever. The TACL paper “Lost in the Middle” found performance highest when the relevant passage sits at the start or end of the context and degraded in the middle, even in long-context models: stuffing in fifty chunks instead of fixing the ranker can make things worse. Evaluation is not optional either side. OpenAI says build evaluations before tuning; for retrieval, Ragas at EACL 2024 scores relevance, faithfulness and generation quality without ground truth.

The hybrid case, the order, and how much data you need

NVIDIA’s published position is an order rather than a choice: start with RAG, then customise if you need more domain-specific accuracy. Its RAG primer says outright that this is not one technique or the other, and names the combination most people miss: parameter-efficient tuning inside a RAG system, refining the language model or the embedding model. Tuning the retriever rather than the generator is the under-discussed option.

The order the sources support: evaluations first; prompt and context engineering; measure the corpus and skip retrieval entirely under roughly 200,000 tokens; build retrieval above that; only then reach for LoRA to fix format, tone, task compliance and domain skill; tune the embedding model last. OpenAI adds one instruction: carry the prompt that worked best before tuning into every training example.

On data the numbers are smaller than people expect. OpenAI states a hard minimum of 10 examples, reports improvements from 50 to 100, and recommends starting with 50 well-crafted demonstrations; its quality bar is that a smaller amount of good data beats a larger amount of poor data, and that examples must be self-contained, consistent and in the format expected at inference. LIMA fine-tuned a 65B model on 1,000 curated prompts and was preferred or equivalent to GPT-4 in 43 per cent of comparisons; its conclusion settles the question, that almost all knowledge is learned during pretraining and only limited instruction data is needed for good output. The QLoRA authors agree that quality beats dataset size.

For facts the requirement inverts. The EMNLP 2024 study found accuracy rising monotonically with the number of paraphrases used: each fact restated many times over, and you still lose to retrieval. For scale, the TMLR paper’s two data regimes were about 100,000 prompt and response pairs for instruction tuning and about 20 billion unstructured tokens for continued pretraining; those are the sizes it tested, not minimums. Two gaps we will not fill with a plausible number. We found no minimum example count in NVIDIA’s NeMo documentation, and Anthropic asks only for high-quality prompt and completion pairs, with no quantity; the classification accuracy figure in its July 2024 announcement, from 81.5 to 99.6 per cent on moderating online comments, is Anthropic’s own test on a single task and says nothing about how many examples a task needs.

Eurokommerz supplies the hardware this decision lands on, across the EU, and would rather size it from your corpus and concurrency than from a technique name. Our private AI and ML page covers the platforms we build on.

FAQ