RAG on company data: what it takes, and where it usually breaks

What the pipeline actually is

Retrieval-augmented generation has a reputation for complexity it does not deserve. Stripped down, it is four steps: find the relevant passages, put them in the prompt, have the model answer using only those passages, and show the user where the answer came from.

Everything difficult is in step one. The model is the part that works. If the retrieved passages contain the answer, almost any competent model will produce it; if they do not, no model will. This is why teams that spend their effort on prompt wording and model selection tend to plateau, and teams that spend it on retrieval quality tend to ship.

The corollary is useful when scoping: a RAG project is mostly a data engineering project. Budget it that way.

The permissions problem, which comes first

This is the failure with real consequences, so it goes before the quality discussion.

The tempting shortcut is to index everything into one collection and instruct the model not to reveal documents the user should not see. That is not a control. A prompt instruction is a request to a probabilistic system, and the salary spreadsheet is already in the context window by the time the model is deciding whether to be discreet about it.

The permissions have to be applied at retrieval: the search itself never returns a passage the requesting user cannot open in the source system. In practice that means three things.

Carry the ACLs into the index. Every chunk stores the identities and groups permitted to read its source document, and the query filters on the requesting user’s identity before ranking.

Re-sync when permissions change, not only when documents change. A person leaves a group and the documents did not change at all, but their access did. Indexes that watch only file modification times miss this entirely, and it is the most common way a stale ACL survives.

Log what was retrieved for whom. When someone asks whether the assistant could have exposed a document, you want to answer from a log rather than from reasoning about probabilities.

Where the source systems have complicated inherited permissions (a long-lived SharePoint estate is the usual case), this is the step that determines the project timeline. It is worth discovering that in week one.

Chunking and metadata: more important than the model

Documents get split before they are indexed, and how they are split decides what can be found.

Metadata pays for itself twice: once at retrieval, where you can restrict to a document type or a date range, and once in the answer, where a citation with a real document name and date is what makes the system trustworthy. An assistant that cites its sources gets checked and corrected by its users; one that does not gets quietly abandoned after the first wrong answer.

Hybrid retrieval (combining keyword matching with vector similarity) is worth the small extra effort in almost every corporate corpus. Semantic search alone reliably fails on the things businesses search for most: part numbers, contract references, error codes, project names. Those need exact matching.

The evaluation set nobody builds

Here is the step that separates the projects that improve from the ones that oscillate.

Write down fifty to a hundred real questions from the people who will use the system, with the correct answer and the document it should come from. It takes a couple of days with a subject-matter expert and it is the highest-value work in the entire project.

Without it, every change is a matter of opinion. Someone adjusts the chunking, three people try it, two say it feels better, and nobody knows. With it, you can measure the only two things that matter:

Retrieval hit rate: how often the correct passage appears in what was retrieved. If this is low, nothing downstream can save the answer, and you know exactly where to work.

Answer correctness: given the right passage, how often the answer is right. If retrieval is good and this is bad, then it is a prompt or model problem, and only then.

Splitting the measurement this way is what makes debugging possible. Most teams measure only the end result, see it is disappointing, and change the model, which is the one component that was probably fine.

Freshness, and the quiet failure

The failure mode that does the most reputational damage is not a wrong answer. It is a confident, well-written answer from a document that was superseded four months ago.

Three habits prevent it. Index on change, not on schedule where the source system supports it, so a corrected policy is searchable the same day. Show the document date in every citation, so a human notices when the answer is grounded in something old. And remove superseded documents from the index rather than relying on ranking: if the old version is still in there, it will eventually be retrieved.

The related discipline is teaching the system to decline. If nothing relevant is retrieved above a confidence threshold, the correct output is “I could not find this in the available documents”, not a plausible paragraph assembled from adjacent material. Users forgive “I don’t know”. They do not forgive being confidently misled, and they only need it to happen once.

What it runs on

The infrastructure question is usually smaller than people expect, and it splits in two.

Embedding the corpus is a one-off burst: every document is processed once, and thereafter only what changes, until you change the embedding model, which means embedding everything again. A few hundred thousand documents is an afternoon on a single professional GPU with a small embedding model (large, LLM-derived embedders take days, and parsing the PDFs often takes longer than the embedding), and then the load drops to near nothing.

Serving is the steady load, and it is driven by concurrent conversations rather than by corpus size; the vector index is measured in gigabytes, roughly 4 bytes times dimensions times chunks plus index overhead, and lives in RAM. What consumes GPU memory is the KV cache for simultaneous users, and RAG prompts are long, because you are putting retrieved passages into every one of them. Size for that: the formulas are in how much VRAM an LLM needs, and the cost trade against a hosted API is in private LLM or cloud API.

For most internal deployments this is one server, not a cluster. If a proposal starts with a multi-node cluster before anyone has measured a query, ask what the measurement was.

The order that works

Narrow first. One department, one document set, one clearly defined question type, with the people who will use it involved from the start. Get the retrieval hit rate up on that before adding a second corpus.

The failure pattern is the opposite: index everything the company has, demonstrate it to everyone, collect a long list of unrelated complaints and never find out which retrieval problem to fix first. A narrow pilot that genuinely answers one team’s questions is worth more than a broad one that half-answers everyone’s, and it is the thing that gets funded for the next stage.

What we do

Private AI/ML is an engagement, not a product: a fixed-price technical assessment first, what your documents actually look like, what the permissions model is, what hardware the measured load needs, then a pilot on one corpus, then production. Eurokommerz holds the contract and supplies the hardware; our engineering partner Vixen.UNO builds and runs the platform. Everything stays inside your perimeter or in EU-hosted capacity, whichever you need.

The first consultation is free and is mostly us asking about your documents and your permissions, because that is where the project will be decided.

FAQ