What is retrieval-augmented generation?

The problem RAG solves

A large language model knows a great deal about the world and nothing about your company. It has never read your refund policy, your leave rules or last month's release notes. Ask it anyway and it will often produce a fluent, confident answer that sounds right for a company like yours - which is worse than no answer, because nobody can tell it was invented.

There are two ways to give a model your knowledge: change the model (fine-tuning), or hand it the relevant pages each time it answers (retrieval). RAG is the second. It is the approach almost every serious company-knowledge product now uses, for reasons that become obvious once you see how it works.

How RAG works, step by step

There are two halves: preparing the knowledge before anyone asks, and answering when someone does.

Preparation happens whenever a document is added or changes. Each file is read and its text extracted, then split into passages (usually called chunks) of a few hundred words, ideally along the document's own headings. Each chunk is turned into an embedding - a long list of numbers that represents its meaning - and stored in a vector database alongside the text, the document's title and section.

Answering happens per question. The question is turned into an embedding with the same model, the closest chunks are retrieved, and the best handful are placed into a prompt along the lines of: "Answer the question using only these numbered passages. Cite the passage numbers. If they do not contain the answer, say so." The model's reply, with its citations, is what the user sees.

• Ingest: extract text, split into chunks, embed, store.
• Retrieve: embed the question, find the most relevant chunks.
• Augment: put those chunks into the prompt as numbered evidence.
• Generate: the model answers from the evidence and cites it.
• Check: confirm the citations exist and support what was said.

Retrieval is most of the job

If the right passage is not among the ones handed to the model, no model can produce the right answer. In practice, most bad RAG answers are retrieval failures, not writing failures.

Vector search is good at meaning: "how many days off do I get" finds a passage about annual leave entitlement even though the words differ. It is weak at exact terms: an error code like E101, a SKU, a form name or a clause number can be drowned out by passages that are merely about the same subject. Keyword (full-text) search has the opposite strengths.

Hybrid search runs both and merges the results. A common merge is Reciprocal Rank Fusion: each passage scores 1/(60 + its rank) in every list it appears in, and the scores are added. A passage near the top of both lists wins; one that only one method found still gets a fair chance. It needs no tuning of how vector scores compare with keyword scores, which is why it is so widely used.

Not every source deserves the same weight

Once you feed an assistant more than one kind of content, you need a view on which to believe. The published refund policy and a two-year-old support ticket may both mention refunds; they may not agree.

Good systems let you mark sources by how authoritative they are and use that in ranking - official documents first, informal or historical material only when nothing better exists. Without it, the assistant will occasionally quote an agent's workaround from 2023 over the policy you rewrote last month.

RAG versus fine-tuning

Fine-tuning retrains a model on your examples. It is useful for teaching a style, a format or a narrow task. It is a poor way to teach facts that change.

• Freshness: RAG reflects a changed document as soon as it is re-indexed; a fine-tuned model is as old as its last training run.
• Citations: RAG can show the passage behind every sentence; a fine-tuned model cannot say where a fact came from.
• Deletion: removing a document from a RAG index removes it from answers; removing a fact from a trained model is not practical.
• Scope: RAG can limit each question to one team's or one customer's documents; a trained model knows everything it was trained on, for everyone.
• Cost: re-indexing a document costs very little; retraining costs time and compute every time.

Where RAG goes wrong

RAG prototypes are easy to build and impressive in a demo. The failures show up in the second month, when real questions arrive. These are the usual ones.

• The right passage is not retrieved - vector-only search, poor chunking, or the user's words differ from the document's.
• The wrong passage is retrieved and confidently used - two documents cover similar ground and the older one ranks higher.
• Nothing relevant exists, and the model answers anyway from general knowledge.
• Citations that look right but do not support the sentence they sit on, or numbers in the answer that appear in no source.
• Stale content - the index was loaded once and never kept in step with the source.
• Leakage - internal or another customer's content is searchable by someone who should not see it.

The guardrails that make it trustworthy

Each of those failures has a fairly mechanical fix, and a production system should have all of them.

Before answering, a confidence gate: if the best passages are not actually about the question, do not call the model at all - say the answer was not found. After answering, a citation check: every cited number must exist, the answer must cite something, and figures in a sentence must appear in the passages it cites. A failed check means regenerate or refuse, never ship.

Around both, scope and sync: decide what each assistant may search, keep that content in step with its source automatically, and limit customer-specific content to that customer.

How SupaCortex does it

SupaCortex is a RAG system with those guardrails built in. Search is hybrid: a vector search and a full-text search run together and are fused with Reciprocal Rank Fusion, weighted by a trust tier you set on each document - official, standard, or community and past tickets. When enough official evidence exists, past-ticket passages are dropped altogether.

A confidence gate refuses questions the documents do not cover, without calling the model. Every answer carries numbered citations; markers are repaired, every citation and number is validated against the evidence, and an answer that fails is regenerated once and otherwise replaced with an honest "couldn't back that up". Content tagged for one customer is only searchable for that customer.

The language and embedding models are open models running on infrastructure we operate, so documents are not sent to a third-party AI API. We measure retrieval against fixed sets of questions with known answers - including a set of off-topic questions every project must refuse - before a change ships.