Retrieval-Augmented Generation

A large language model (LLM) is a program that has read an enormous amount of text and picks what word plausibly comes next. Your company's documents are not in there. Neither is anything that happened after its training cut-off. Ask about either and it doesn't say it doesn't know — it produces a convincing sentence.

At Pagedeck the problem surfaced in support. Mia tried having a model draft replies to customer questions, and one draft explained how to reach a menu that doesn't exist, complete with a navigation path. The prose was smooth and the format was right. The only wrong part was that no such item exists in settings. Producing false answers fluently is called hallucination.

Making the model larger doesn't solve it. What your help centre says is information the model doesn't hold, so it has to be fetched from outside and handed over. Adding a retrieval step before the answering step is what this approach is.

Split the work into fetching and answering. Four things get prepared in advance and three happen once a question arrives.

1. Gather the material. Help articles, policy documents, past support replies. Filter out anything stale or contradictory at this stage; feed in documents that disagree with each other and there's no way to tell which one an answer came from, or to argue with the answer afterwards.
2. Cut documents into pieces. Feeding a whole article means irrelevant passages ride along as noise. Cut at paragraph boundaries into roughly 300 to 600 words, and record which document and which section each piece came from.
3. Convert the pieces into embeddings — representations of meaning as lists of numbers, where text that shares no words still lands close if the meaning is close. It's why "when do I get my refund" and "payment cancellation processing time" end up in the same neighbourhood.
4. Store those numbers in a vector database, which finds the nearest ones quickly.
5. When a question arrives, convert it the same way and pull the five or so closest pieces. This step decides whether the whole thing works, and it is the step most teams spend the least time on.
6. Attach the retrieved pieces to the instruction and send it to the model, with a condition stated explicitly: answer only from the material below, and say you don't know when it isn't there.
7. Show the sources with the answer. Linking the document and passage behind each answer lets the user verify it and lets the team trace a wrong answer back to a specific piece.

Evaluate with a list that pairs questions to the location of their answers. Write 50 questions, note which document holds each answer, and you can measure whether retrieval surfaced the right document separately from whether the final answer was correct. Measuring the two separately is what tells you which half to fix.

It fits any feature where the answer has to come from your own material: internal document search, support replies, in-product help, or anything that has to cite a policy. It also suits material that changes often, since swapping the documents changes the answers from the next question onward with no work on the model. Where sources have to be shown, it's effectively the only option.

Five limits are worth knowing.

1. Bad retrieval means a bad answer. Most failures aren't the model composing badly, they're the wrong pieces being pulled. When an answer is wrong, look at the retrieved pieces before looking at the model.
2. The quality of the material sets the ceiling. If the help article describes a screen from three years ago, so does the answer, delivered with exactly as much confidence as a correct one.
3. Permissions have to be part of it. If retrieval can reach documents the user isn't allowed to see, the answer leaks their contents, and it leaks them as a paraphrase that no access log will flag. Filter by that user's access at search time.
4. It's the wrong tool for changing tone or format. That belongs to prompt engineering, or to additional training of the model itself.
5. More steps mean more latency and more cost. Three round trips — embed, search, answer — usually put the response somewhere between two and five seconds, which is slow enough that the interface needs something to show while it waits.

"The answer's wrong, but it isn't the model — it pulled the wrong document."

Diagnosing the failure as retrieval rather than generation.

"I put 'say you don't know if it isn't in the material' into the instruction."

Closing off the option of inventing content.

"The permission filter has to be in the retrieval step. Right now other people's documents come back."

Access control is missing where the search happens, so content leaks.

Theo and Mia's first version of the support draft feature answered correctly less than half the time. Inspecting the retrieved pieces showed the cause: the help centre was written as one long article per feature, so a single piece often contained three unrelated explanations. Re-cutting the documents per item and reducing pieces to about 400 words raised accuracy noticeably. The model and the instruction were unchanged; the only difference was how the material had been sliced.

The most common reading is that adding this removes hallucination. Supporting material is right there, so there's no reason to invent anything. But when the retrieved pieces don't relate to the question, the model still fills the gap. If none of the five pieces contains the answer, it assembles something plausible out of those five. The rate drops and it doesn't reach zero, so leave an explicit route to "I don't know" in the instruction and track the share of answers that carry no source. That share is the closest thing to a hallucination rate you can measure cheaply.

The other belief is that more material is always better. A bigger corpus sounds like broader coverage. In practice, several near-duplicate documents make retrieval fail to pick between them, and a stale document retrieved alongside a current one gets blended into one answer. Removing 30 percent of the material and watching accuracy rise is a common experience. Cleaning out duplicates and outdated pages before loading anything is worth more than changing models.