RAG on company data: what it takes, and where it usually breaks

How a RAG pipeline works: four steps

RAG on company data (retrieval-augmented generation) has four steps: find the passages in the company’s documents that answer the question, put them in the prompt, have the model answer using only those passages, and show the user where the answer came from.

Most of the work is in the first step. If the retrieved passages contain the answer, a current instruction-tuned model will usually produce it, although an answer placed in the middle of a long context is found less reliably. If the passages do not contain it, no prompt or model change will help, so measure retrieval before you tune prompts or swap models.

For scoping, this means that a RAG project is mostly a data engineering project and should be planned and staffed as one. Whether to fine-tune a model instead is a separate question, which our comparison of fine-tuning and RAG answers.

Access permissions: enforce them at retrieval

A leak of restricted documents is the failure with legal consequences, so it comes first.

A tempting shortcut is to index everything into one collection and instruct the model not to reveal documents the user should not see. This does not control access. A prompt instruction is a request to a probabilistic system, and by the time the model processes it, the salary spreadsheet is already in the context window.

The permissions have to be applied at retrieval: the search itself never returns a passage the requesting user cannot open in the source system. That requires three measures.

Carry the access control lists (ACLs) into the index. Every chunk stores the users and directory memberships permitted to read its source document, and the query filters on the requesting user’s identity and memberships. With an approximate vector index, the filter has to act before or during the search: applied only to the candidates afterwards, it leaves a user who may read a small share of the corpus with too few passages. Our vector database comparison shows how pgvector, Qdrant, Milvus and OpenSearch handle this.

Re-sync when permissions change, not only when documents change. A person’s directory membership changes and the documents did not change at all, but their access did. Indexes that watch only file modification times miss this, and the old permission stays in the index until the document itself changes.

Log what was retrieved for whom. When someone asks whether the assistant could have exposed a document, the log answers it.

Filtering at retrieval keeps other people’s documents out of the context. It does not stop a document the user may read from carrying instructions to the model, for example to send data out through a link or a tool call. This is indirect prompt injection. OWASP lists prompt injection first, as LLM01, in its Top 10 for LLM Applications 2026, and our guide to prompt injection describes the controls that limit it.

Where the source systems have complicated inherited permissions (a document-management system that has grown over many years is the usual case), this is the step that determines the project timeline. It is worth discovering that at the start.

Assistants built in our Private AI/ML service answer from your documents, cite the source and respect each user’s access rights. Tell us which systems hold the documents and how access to them is granted, and we reply within one business day.

Chunking and metadata

Documents get split before they are indexed, and how they are split decides what can be found.

Metadata is used twice: at retrieval, where you can restrict the search to a document type or a date range, and in the answer, where a citation with a real document name and date is what makes the system trustworthy. With a citation the user can open the source and check the answer in a few seconds.

Hybrid retrieval (combining keyword matching with vector similarity) is worth the small extra effort in almost every corporate corpus. Embedding models capture meaning but can miss exact matches, so part numbers, contract references, error codes and project names need keyword matching as well.

Building an evaluation set from real questions

Write down fifty to a hundred real questions from the people who will use the system, with the correct answer and the document it should come from. It needs a subject-matter expert, and it is the most useful work in the project.

Without it, every change is a matter of opinion: someone adjusts the chunking, a few colleagues try it, and nobody can say whether it is better. With it, you can measure the two things that matter:

Retrieval hit rate: how often the correct passage appears in what was retrieved. If this is low, nothing downstream can save the answer, and you know exactly where to work.

Answer correctness: given the right passage, how often the answer is right. Only when retrieval is good and this is still low does the problem lie in the prompt or the model. Metrics and tooling for both measurements are in our guide to LLM evaluation.

The two figures show where to work: a low hit rate points to chunking, metadata or search, and low correctness with a good hit rate points to the prompt or the model. A single score for the final answer hides which of the two failed.

Keeping the index current

The failure that does the most damage to the system’s reputation is a confident, well-written answer from a document that was superseded four months ago.

Index on change, not on schedule where the source system supports it, so a corrected policy is searchable the same day. Show the document date in every citation, so a human notices when the answer is grounded in something old. And remove superseded documents from the index rather than relying on ranking: if the old version is still in there, it will eventually be retrieved.

The related discipline is teaching the system to decline. If nothing relevant is retrieved above a confidence threshold, the correct output is “I could not find this in the available documents”, not a plausible paragraph assembled from adjacent material. An answer that says nothing was found sends the user to the source system; a confident wrong answer may be acted on.

Hardware for RAG: embedding, serving and the vector index

The load has two parts: embedding the corpus and serving the questions.

Embedding the corpus is a one-off peak: every document is processed once and afterwards only what changes, until you change the embedding model and have to embed everything again. By our estimate from NVIDIA’s NIM figures for its 1B embedding model in FP8 (144.3 texts of 512 tokens per second on an L40S, 49.2 on an L4; NIM 1.14.0 documentation of 30 September 2026), 300,000 documents of ten 512-token chunks each take about 5.8 hours on one L40S and about 17 hours on one L4 (large, LLM-derived embedding models take days, and parsing the PDFs often takes longer than the embedding). After that, the load drops to almost nothing. The calculation behind these figures, and the GPU load of a reranker, are in our article on embedding and reranker servers for RAG.

Serving is the steady load, and it depends on concurrent conversations rather than on corpus size. The vector index is measured in GB, roughly 4 bytes times dimensions times chunks plus index overhead, or less if the vectors are stored quantised, and is usually held in RAM unless the database keeps it on disk; our vector database comparison gives the RAM per million vectors for four databases. GPU memory goes to the KV cache for simultaneous users, and RAG prompts are long because every one of them carries retrieved passages. The formulas for this sizing are in how much VRAM an LLM needs, and the cost comparison with a hosted API is in private LLM or cloud API.

For most internal deployments this is one server, not a cluster, and a multi-node cluster should follow from measured query load. Two of the reference configurations on our AI servers page fit this: two RTX PRO 6000 Server Edition cards for pilots and RAG assistants with models up to about 70B parameters, and four for production inference for several teams and several models.

Sizing that server, with the model and GPU selection, is part of the technical assessment in our Private AI/ML service, and we supply the hardware under the same contract. Write to us with the number of documents and of concurrent users.

Start with a narrow pilot

Begin with one department, one document set and one clearly defined type of question, and involve the people who will use the system from the beginning. Get the retrieval hit rate up on that before adding a second corpus.

Projects stall when they do the opposite: index everything the company has, demonstrate it to everyone, collect a long list of unrelated complaints and never find out which retrieval problem to fix first. A narrow pilot gives one team measurable answers, and its hit rate and answer correctness are the basis for deciding on the next document set.

What we do

Our private AI/ML service follows the order above. The first call with our engineering partner Vixen.UNO is free of charge and starts with your documents and how permissions work today; the paid technical assessment then covers the solution architecture, the model and GPU selection and a pilot plan with metrics, at a price fixed before work begins. The pilot runs on one document set, and only what proves its value is scaled. Eurokommerz holds the contract and supplies the hardware; Vixen.UNO builds and supports the platform, on your infrastructure or on dedicated hardware in a Tier III data centre in Lithuania, and public APIs are used only if you enable them for a task.

FAQ