Fine-tuning embedding models and rerankers for RAG: training data, losses and the GPU

When fine-tuning for domain-specific embeddings pays off

Fine-tuning an embedding model for RAG means training it on pairs of questions and passages from your own documents, so that it places your vocabulary, product names and abbreviations correctly. The GPU needs are small: widely used multilingual models such as Qwen3-Embedding-0.6B and bge-m3 have 0.6B parameters or less and train on one card with 24 to 48 GB, such as an NVIDIA L40S or a MIG instance of an RTX PRO 6000 Server Edition. The cost lies elsewhere, in building the training pairs, testing the result and re-embedding the corpus.

Start with a measurement. Our guide to RAG on company data explains why wrong answers usually begin in retrieval and how to build an evaluation set from real questions. If that set shows the right passage missing from the top results for questions with internal terms, part numbers or domain language, the retriever is the part to train. If the passage is found but ranked too low, a fine-tuned reranker may be enough, and it avoids re-indexing. If the problem lies in the language model's behaviour, our fine-tuning or RAG comparison is the better starting point.

This page covers training. Which GPU serves the models in production, and how fast, is in our guide to embedding and reranker servers for RAG.

Training pairs from your own documents

Embedding models learn from examples of what belongs together. The minimum is a pair: an anchor, usually a question, and a positive, the passage that answers it. A triplet adds a negative, a passage that looks similar but does not answer. The sentence-transformers documentation, version 6.2.0 of 21 October 2026, takes the inputs from the dataset columns in order: "The names of these columns are irrelevant, only the order matters", and a label column has to be named "label", "labels", "score" or "scores". Extra columns such as an ID are treated as inputs, so remove them.

Most companies have no such pairs, but they have the passages. The sentence-transformers guide to domain adaptation describes Generative Pseudo Labeling (GPL), which builds training data from an unlabelled corpus in four steps: a T5 model "generates a possible query for the given text"; dense retrieval mines negative passages from the corpus; a cross-encoder scores all query-passage pairs, which catches mined passages that in fact answer the query; and the embedding model trains on these scores with MarginMSELoss. The guide reports that GPL can improve semantic search "up to 10 points". On a RAG server the language model that already answers users can take the T5's place and write the questions for each chunk; that is our suggestion, and the evaluation below shows whether it helps.

Hard negatives need care. Hugging Face's reranker example mines them with an existing embedding model through mine_hard_negatives, with num_negatives=5, range_max=100 and sampling_strategy="top": five of the top 100 results that are not the answer. On your documents, check a sample of mined negatives by hand, because a passage that answers the question as well as the labelled one would teach the model the wrong thing; GPL's cross-encoder step exists for this reason.

We supply single cards and servers for retrieval training and serving, from the L40S to the RTX PRO 6000 Server Edition; send us your model, corpus size and number of training pairs, and our team replies with a card and memory plan.

Losses and why batch size sets the memory

The sentence-transformers loss overview marks two losses as recommended defaults for both (anchor, positive) pairs and (anchor, positive, negative) triplets: MultipleNegativesRankingLoss and CachedMultipleNegativesRankingLoss. It says the first is "commonly used to train the top performing embedding models". The loss treats the other positives in the same batch as negatives for each anchor, so a larger batch gives each question more wrong answers to tell apart. The documentation adds that "MultipleNegativesRankingLoss benefits from no duplicate samples in a batch", which the batch sampler BatchSamplers.NO_DUPLICATES provides.

This is why batch size, more than model size, decides the memory of an embedding fine-tune. Activations grow with the number of texts in the batch times their length. The cached variant decouples the two. It follows the gradient caching paper of 2021, which "decouples backpropagation between contrastive loss and the encoder" and computes gradients for one part of the batch at a time, with almost constant memory. The documentation notes that the cached loss "is often used to increase the batch size, resulting in superior performance".

MatryoshkaLoss wraps one of these losses and trains embeddings "whose size can be truncated without notable losses in performance". Qwen3-Embedding already supports output dimensions from 32 to 1,024 on its 0.6B model; if your index uses shortened vectors, train and evaluate at that size.

Keep the query format the model expects. Qwen3-Embedding's card writes queries as Instruct: {task_description}\nQuery:{query} and documents without an instruction. By our reasoning, the fine-tuning data should use the same format, so that training and production see the same input.

GPU memory per embedding model

For full fine-tuning we use the same rule as our fine-tuning memory guide: 16 bytes per parameter for weights, gradients and Adam states in mixed precision. The sentence-transformers documentation says that "Loading in fp32 is preferred for training if your memory can handle it"; in 32-bit, weights, gradients and Adam states come to the same 16 bytes. Activations come on top and depend on batch size and sequence length.

Safetensors sizes from the Hugging Face file lists of the Qwen models, read on 24 October 2026; bge-m3's 568M parameters from our embedding page (NVIDIA's NIM support matrix). Full fine-tune = 16 bytes per parameter by our arithmetic, model states only.

Set against the cards, the small models fit everywhere and the large ones need a large card.

Listed capacity of each card or instance (NVIDIA datasheets and MIG guide as on our product and MIG pages) minus model states by our arithmetic; 8-bit Adam at about 10 bytes per parameter, 75.7 GB for the 8B model. Activations come out of what is left.

For the 0.6B models the card is chosen by batch size and convenience. A 24 GB MIG instance on the RTX PRO 6000 Server Edition that serves your language model can train the retriever overnight in its own memory partition, as our RTX PRO 6000 MIG profiles page describes; the L40S and the RTX PRO 4500 Server Edition are the cards for a separate retrieval server. The 4B and 8B models are large language models turned into encoders, and a full fine-tune of them is a job for a whole RTX PRO 6000 Server Edition or H200 NVL. With 20 GB or less left, the 8B model leaves little room for the batch that its loss depends on, so plan the cached loss or a smaller model.

Fine-tuning the reranker instead

A reranker, or cross-encoder, reads the query and one passage together and returns a score. It runs after retrieval on the top results, so a better reranker improves the answer without touching the index. Hugging Face's reranker training example of 26 March 2025 shows the scale. It fine-tuned ModernBERT-base on 99k query-answer pairs from the GooAQ dataset, mined five hard negatives per query with a static embedding model to reach 578k labelled pairs, and trained with BinaryCrossEntropyLoss at a pos_weight of 5 and a batch size of 16. Training took "just 30 minutes" on one RTX 3090, a 24 GB card of an older consumer generation; the large model took less than an hour.

Hugging Face blog "Training and Finetuning Reranker Models with Sentence Transformers v4", 26 March 2025, reranking the top 30 results of the static embedding model; read 24 October 2026.

These figures are on a public question-answering set, not on company documents. They show the order of magnitude: a reranker trained for the task beat the general-purpose rerankers in the comparison, after training on a 24 GB card, the size of the smallest option in this article. The author also writes that rerankers are "particularly receptive" to strong hard negatives, so the mining step above applies here as well.

Evaluate before you re-embed

Test the fine-tuned model against the model in production on the same evaluation set before you change anything in the index. sentence-transformers' InformationRetrievalEvaluator takes three inputs: queries, the corpus and the relevant documents per query. Keep the evaluation questions out of the training data, and include real user questions, not only generated ones, because the test should reflect how users ask. The documentation warns that "the evaluator only runs on the first device", which matters only when you train on several cards.

Run the same evaluation for every candidate and for the current model, and compare recall at the cut-off your RAG pipeline uses. Our guide to LLM evaluation before model upgrades describes how to keep such a test set for the language model too.

The re-index after a new embedding model

A fine-tuned embedding model produces new vectors for every passage. Vectors from the old and the new model do not belong in one index, so the whole corpus is embedded again. On our embedding page, 3 million chunks of 512 tokens take about 17 hours on one L4 and about 5.8 hours on one L40S with NVIDIA's 1B embedding model in FP8, by our arithmetic from NVIDIA's published throughput, and an 8B model takes several times as long. A million 1,024-dimensional float32 vectors take 4.10 GB before any index, as our vector database comparison shows.

Plan the re-index like a migration: build the new index next to the old one, run the evaluation set against both and switch over when the new one is better. A fine-tuned reranker avoids all of this, which is one reason to try it first.

Which card for retrieval training

The retriever is small and the server already has an RTX PRO 6000 Server Edition. Train Qwen3-Embedding-0.6B, bge-m3 or a cross-encoder in a 24 GB MIG instance, at a time when the instance is free, and leave the other instances to the language model.

A separate retrieval card for serving and training. One L40S with 48 GB, or an RTX PRO 4500 Server Edition with 32 GB, trains the small models with room for large batches and serves them afterwards. Our page on the L40S server lists the configurations.

4B or 8B embedding models. A whole RTX PRO 6000 Server Edition or an H200 NVL, for training and for the re-index that follows.

The data side, from generating questions to evaluation and re-indexing, is part of our Private AI/ML service, built by our engineering partner Vixen.UNO; we supply the cards and the servers, assembled and tested before delivery, on one EU contract and invoice. Describe your corpus and the retrieval failures you see, and we reply within one business day.

FAQ