Fine-tuning or RAG: which one actually puts your company’s knowledge into a model

What each technique actually changes

One technique changes the model’s parameters. The other changes what you put in front of them at query time. Two consultants can argue for an hour without settling that.

Retrieval-augmented generation, defined by Lewis and colleagues in May 2020, pairs the model’s parametric memory with a searchable non-parametric one. Their own framing is the useful part: providing provenance for a model’s decisions and updating its world knowledge were, in their words, open research problems. Anthropic’s glossary puts the mechanism operationally, documents retrieved at runtime into the context window. Not one weight moves.

Fine-tuning is the opposite. NVIDIA’s customisation guide draws the line plainly: fine-tuning updates the pretrained weights where prompt-based techniques leave them frozen. Anthropic’s glossary says what that buys, a model that represents and imitates the patterns of the tuning dataset. Patterns, not facts.

LoRA, from Hu and colleagues in June 2021, freezes the base and injects trainable low-rank matrices into each transformer layer. Its headline claim: against GPT-3 175B fine-tuned with Adam it cut trainable parameters by 10,000× and GPU memory by three times, training memory from 1.2 TB to 350 GB and the checkpoint to 35 MB. Two conditions are usually dropped: that parameter cut was measured at rank 4 on the query and value projections alone, where Hugging Face’s PEFT documentation now recommends adapting all linear layers, and the baseline is the authors’ own. It also reports a 25 per cent training speed-up and no added inference latency.

QLoRA, from Dettmers and colleagues in May 2023, puts LoRA on a base quantised to four bits: a 65B model tuned on one 48 GB GPU while preserving 16-bit task performance, where 16-bit tuning of it needed more than 780 GB. Its mechanisms: a four-bit type called NormalFloat, a second quantisation pass dropping the quantisation constants from 0.5 bits per parameter to 0.127, and paged optimisers. Its 99.3 per cent of ChatGPT figure is the Vicuna benchmark only.

The decision table, and which rows are settled

Map the problems, not the techniques; the last column is what to read.

The access-control row is reasoning, not a sourced claim: no primary source recommends metadata filtering as the authorisation mechanism, but retrieval can be filtered per user where weights cannot.

You will also hear that the model providers say fine-tuning is the wrong tool for facts. They do not. OpenAI’s optimisation, fine-tuning and best-practice pages contain no such statement; Anthropic’s glossary defines both and never compares them; NVIDIA publishes no rule for choosing parameter-efficient tuning over full tuning. The rule is well supported by the research below; it is not a vendor position. The one published threshold is Anthropic’s: under 200,000 tokens of corpus, about 500 pages, put the lot in the prompt and build no RAG. Our guide to RAG on company data covers the rest.

What fine-tuning costs in GPU memory

Training memory is dominated by model states, and the multiplier is disputed: the dispute moves every figure by a third.

The ZeRO paper from Rajbhandari and colleagues breaks mixed-precision Adam down: two bytes per parameter of fp16 weights, two of fp16 gradients, then an fp32 master copy, momentum and variance at four bytes each: sixteen in total. Hugging Face’s memory-anatomy page lists the same components and totals 18, assuming fp32 gradients rather than fp16. The QLoRA paper implies 12: its 780 GB for a 65B model is exactly 65 billion times 12 bytes, omitting that master copy. Working range: 16 to 18 bytes per parameter.

Our arithmetic from the multipliers above, not vendor or paper figures. GB = 10⁹ bytes; model states only, activations on top. LoRA assumes a frozen bf16 base and an adapter at NVIDIA’s under-1-per-cent ceiling; QLoRA 0.516 bytes per parameter.

So: no full fine-tune of an 8B, 32B or 70B model fits on one 96 GB card, before any activation, and activations carry a term quadratic in sequence length. LoRA on 70B does not fit either. QLoRA on 70B does. The belief that a 7B fully tunes on one card confuses inference with training memory: that model is 14 GB to serve at FP16 and about 112 GB to fully tune, a ratio of eight to one. An eight-bit optimiser might close the gap; no primary source states its bytes per parameter, so we print none.

QLoRA’s saving has a price, and NVIDIA publishes both halves on one page. That QLoRA is “up to” 60 per cent more memory-efficient than LoRA is a vendor claim about its own product, and the saving shrinks as batch size grows. Directly below it, and not marketing: QLoRA training is 50 to 200 per cent slower than LoRA.

The document almost nobody quotes: NVIDIA’s scheduling matrix, on L40 cards of 48 GB.

NVIDIA’s NeMo Microservices configuration matrix; aggregate VRAM is our own multiplication.

NVIDIA schedules two L40 cards, 96 GB, for an 8B LoRA job that the weight arithmetic puts at about 17 GB; the gap is activations, framework overhead and service defaults, and it is what you procure against. And NVIDIA lists no all-weights configuration for any 70B model. Serving runs the other way: vLLM holds many LoRA adapters against one copy of the base weights, so at 35 MB a tenant costs almost nothing.

What RAG costs to run, which nobody budgets

RAG is called the cheap option because it needs no training run. That is not where its money goes.

The embedding model is a GPU service of its own: NVIDIA’s retrieval NIM matrix asks a minimum of 3.6 GiB of GPU memory for its 1B embedding model in the unoptimised FP16 profile, and the spread on that page is sixteenfold, from 2 GiB to 33 GiB for bge-m3. Around it sit four more services: NVIDIA’s enterprise RAG architecture names the language model, embedding and reranking microservices, document extraction and Milvus, at a baseline of four GPUs, 40 vCPU and 200 GiB. Only one of those GPUs serves the language model.

Chunking defaults are not even self-consistent: LlamaIndex’s library constant is 1,024 tokens with 20 tokens of overlap where its own sentence splitter uses 200, and LangChain’s base splitter defaults to 4,000 characters with 200 where its own tutorial uses 1,000 and 200. One repeated error to correct: NVIDIA’s RAG primer says OpenAI’s embedding models encode up to 1,536 tokens and truncate silently beyond. The truncation is real; the number is not. OpenAI documents 8,192 tokens of input, and 1,536 is the output vector dimension of its small model.

Re-ranking pays. Anthropic measured a top-20 retrieval failure rate of 5.7 per cent on its own corpora, falling to 3.7 with contextual embeddings, 2.9 with contextual BM25 added and 1.9 when the top 150 results were reranked to 20: the relative reductions of 35, 49 and 67 per cent quoted without that 5.7 per cent base. Milvus publishes no per-vector byte count, so the store gets measured.

Now the figure that decides serving cost. NVIDIA’s KV cache formula is two, times the layer count, times the hidden size, times the bytes per element, per token. For Llama 2 7B at 16-bit that is 2 × 32 layers × 4,096 × 2 bytes, or 512 KiB per token. Retrieved passages are tokens like any others, so 4,000 tokens of context per request costs about 2 GiB of KV cache per concurrent request, and about 62 GiB at 32 concurrent requests: two thirds of a 96 GB card spent on retrieved text before a token is generated. That multiplication is ours, and it assumes multi-head attention as in Llama 2 7B. Current models of that size use grouped-query attention, with far fewer key-value heads and a proportionally lower cost per token, so run the formula against your own configuration; we could not verify the key-value head counts of current Llama 3.1 checkpoints, so we do not print them. Our article on how many users one RTX PRO 6000 can serve works the arithmetic end to end.

What the published comparisons actually found

Ovadia and colleagues, at EMNLP 2024, tested Mistral 7B, Llama2 7B and Orca2 7B on MMLU subsets and on a current-events task on knowledge the models could not have seen. With Mistral 7B on current events the base model scored 0.481, base plus retrieval 0.875, the fine-tuned model 0.504 and the fine-tuned model with paraphrases 0.588. Their conclusion: “LLMs struggle to learn new factual information through unsupervised fine-tuning”.

The load-bearing word is “unsupervised”. That study’s fine-tuning arm was continued pretraining on raw text, not supervised tuning on question and answer pairs, which is what most companies do; the paper says so in its abstract. Generalising it to supervised tuning is the commonest misreading in the field. What it establishes is narrower: pouring a document corpus into a continued-pretraining run does not put those facts reliably into the weights.

The counter-result is Balaguer and colleagues’ agriculture case study of January 2024, which used supervised tuning and found the two additive: over 6 percentage points from fine-tuning and a further 5 from retrieval on top, retrieval effective on contextually relevant data and tuning at teaching domain skills. The two do not reconcile and we will not pretend they do. Ovadia’s tables show base plus retrieval beating fine-tuned plus retrieval, 0.681 against 0.659 on anatomy at zero shot; Balaguer’s show them stacking. Different tasks, domains and tuning regimes.

Two more results. Soudani and colleagues at SIGIR-AP 2024 ran twelve models from 80M to 11B parameters and found retrieval ahead by a large margin on low-popularity facts; a small fine-tuned model with retrieval matched a much larger one, 1.6B against 8B: the logic behind running a private model rather than a cloud API. And Biderman and colleagues, in TMLR in August 2024, compared LoRA with full fine-tuning on programming and mathematics: at standard low ranks LoRA substantially underperforms it, the mechanism proposed being that full tuning learns weight changes of 10 to 100 times the rank. LoRA’s own paper claims parity or better. Both are peer-reviewed and disagree directly, so raise the rank before concluding.

What breaks, and what each one demands afterwards

Fine-tuning’s characteristic failure is catastrophic forgetting, and the finding that should stop a project is about scale. Luo and colleagues measured it across domain knowledge, reasoning and reading comprehension in models from 1B to 7B and found it general and intensifying as the models grew. A bigger base model is no mitigation; instruction tuning before the domain tune did help. LoRA is the partial answer and the trade is explicit: the TMLR paper that found it weaker on the target domain found it preserving abilities outside that domain better than weight decay or dropout.

RAG’s failures are named and counted. Barnett and colleagues, in a CAIN 2024 experience report from three case studies, list seven: content missing from the corpus, the right document ranked too low, the right document dropped in consolidation, the answer in context but not extracted, output ignoring the requested format, the wrong specificity, and an answer correct but incomplete. Note the fifth: wrong format is a fine-tuning problem inside a RAG failure taxonomy. Their verdict is that a RAG system can only be validated in operation, and the monthly burden is their knob list: chunk size, embedding, chunking, retrieval and consolidation strategy, context size and prompts.

A longer context window does not rescue a weak retriever. Liu and colleagues, in TACL, found performance highest when the relevant passage sits at the start or end of the context and degraded in the middle, even in long-context models: stuffing in fifty chunks instead of fixing the ranker can make things worse. Evaluation is not optional either side. OpenAI says build evaluations before tuning; for retrieval, Ragas at EACL 2024 scores relevance, faithfulness and generation quality without ground truth.

The hybrid case, the order, and how much data you need

NVIDIA’s published position is an order rather than a choice: start with RAG, then customise if you need more domain-specific accuracy. Its RAG primer says outright that this is not one technique or the other, and names the combination most people miss: parameter-efficient tuning inside a RAG system, refining the language model or the embedding model. Tuning the retriever rather than the generator is the under-discussed option.

The order the sources support: evaluations first; prompt and context engineering; measure the corpus and skip retrieval entirely under roughly 200,000 tokens; build retrieval above that; only then reach for LoRA to fix format, tone, task compliance and domain skill; tune the embedding model last. OpenAI adds one instruction: carry the prompt that worked best before tuning into every training example.

On data the numbers are smaller than people expect. OpenAI states a hard minimum of 10 examples, reports improvements from 50 to 100, and recommends starting with 50 well-crafted demonstrations; its quality bar is that a smaller amount of good data beats a larger amount of poor data, and that examples must be self-contained, consistent and in the format expected at inference. LIMA fine-tuned a 65B model on 1,000 curated prompts and was preferred or equivalent to GPT-4 in 43 per cent of comparisons; its conclusion settles the question, that almost all knowledge is learned during pretraining and only limited instruction data is needed for good output. The QLoRA authors agree that quality beats dataset size.

For facts the requirement inverts. Ovadia and colleagues found accuracy rising monotonically with the number of paraphrases used: each fact restated many times over, and you still lose to retrieval. The scale reference points are in the TMLR paper: roughly 100,000 prompt and response pairs to teach a job, 20 billion unstructured tokens to teach a field. Two gaps we will not fill with a plausible number. NVIDIA publishes no minimum example count in its NeMo documentation, and Anthropic asks only for high-quality prompt and completion pairs, with no quantity; the accuracy figures in its own announcement are customer marketing, on a classification task.

Eurokommerz supplies the hardware this decision lands on, across the EU, and would rather size it from your corpus and concurrency than from a technique name. Our private AI and ML page covers the platforms we build on.

FAQ