Hallucination

The first surprise for teams putting language models into products wasn't that answers came back wrong. It was that a wrong answer looked exactly like a right one. The sentences were smooth, the formatting was tidy, and only the facts inside were missing.

So the early instinct was to treat it as a bug: write down the steps to reproduce it, find the line of code behind it, fix that line, check again. But feeding the same question back in produced a different answer, and that one was wrong too, wrong in a different way. There was no line to fix.

Pagedeck ran into this directly. A feature that summarized meeting notes in three lines wrote down a decision nobody had made, and when Theo fed the same notes in again, the summary credited that decision to someone else. Nothing in the code had misbehaved. The model had done exactly what it was built to do, which is why this failure needed a name of its own instead of being filed as a defect.

A large language model (LLM) builds a sentence by picking, one at a time, the word most likely to follow what came before. It isn't looking facts up somewhere and copying them across.

There is no table of facts inside the model, only the patterns left behind by reading an enormous amount of text. It knows that a number follows "last year's revenue was," but it doesn't know which number, so it picks one that would plausibly go in that slot. There is no state for not knowing, so rather than leave the blank empty it fills it with whatever reads best. Tone is decided separately from truth as well: the documents it learned from are mostly written in flat declarative sentences, so the model writes that way regardless.

Hallucinations come from four distinct places.

1. The model was asked about something it never saw. Internal policies, or a pricing change from last week, get invented whole.
2. The source material got cut off. The context window is the maximum amount of input a model can look at in one pass, so anything longer has to be trimmed, and the model treats the trimmed part as nonexistent. Feed in a 200-page contract, ask about the penalty clause, and get told there isn't one.
3. The material attached to the question was the wrong material. Retrieval-augmented generation (RAG) pulls the relevant passage out of internal documents, attaches it to the prompt, and has the model answer from that passage alone. When the passage retrieved has nothing to do with the question, the model builds a confident answer on top of it anyway.
4. The question carried a false premise. Ask to have "the vacation carryover policy we introduced last year" explained when no such policy exists, and the model writes the explanation without questioning the premise.

The four call for different fixes. The first and third are about what the model gets shown; the last two are about what it is allowed to assume. Lumping them together as bad hallucinations leaves all four untouched.

The word comes up when a team is scoping or reviewing a feature built on a language model. In scoping it's used to ask what happens when the answer is wrong. In review it's used to count how often that happens.

The weight differs feature by feature inside one product. When a meeting summary invents a decision, people who weren't in the room learn it as fact. When a support reply invents a refund rule, that reply becomes the company's promise. A draft-writing feature is milder, since a person rewrites it anyway. So the first question isn't what the accuracy percentage is. It's who loses what when the answer is wrong.

Defenses stack in four layers.

1. Make it answer from evidence. Retrieving the relevant passage and putting it in front of the model means it reads rather than recalls.
2. Work the instructions. Prompt engineering is the craft of designing what you tell the model, and a line saying not to write anything absent from the attached evidence is the cheapest lever available.
3. Check the output. A guardrail is a limit written in code that the answer cannot cross, such as matching each sentence against the source documents and refusing to send anything that isn't there.
4. Put a person in the path. Human in the loop means placing a human confirmation step in front of consequential decisions, like having someone on the support team read a customer-facing reply before it sends.

Whether any of that helped is a question for an eval set, a fixed collection of questions paired with expected answers that gets rerun after every prompt change so error rates can be compared. The arithmetic turns urgent in multi-step features. In an agent setup that searches, checks, and searches again, a fact invented at step two becomes the input to step three, and everything after that stacks on top of it.

"This answer looks like a hallucination. There's no such clause in the source document."

What the reply says isn't in the material that was attached, so the model filled the gap itself.

"Did you measure the hallucination rate? I want evidence it improved before we ship."

Rerun the eval set after the prompt change and compare the numbers.

"If this one hallucinates it goes straight to the customer, so let's have someone read it before it sends."

The cost of a wrong answer is high enough here to justify a review step.

Before Pagedeck opened its meeting-summary feature, Mia and Theo read twenty summaries by hand. Three contained decisions nobody had made. Theo added a line to the prompt saying nothing outside the original notes should appear, and the count dropped to one.

They split on the remaining one. Theo thought more tuning would remove it. Mia wanted to surface it instead: if every line of the summary carries a marker showing which part of the notes it came from, the lines with nothing behind them are the first thing a reader notices. They shipped both.

The most common mistake is treating this as a problem that better models will eventually eliminate. Error rates really do fall with each new release, so the assumption is easy to reach. But falling and disappearing are different things. As long as the next word is chosen by probability, nothing appears that leaves an unknown slot empty. Teams that wait it out ship features with no source markers on screen and no set for counting wrong answers.

The other common one is believing a single line of instruction settles the matter. Write "say you don't know when you don't know" and the model agrees to do so, which reads like a fix. That line does lower the rate, but it doesn't take it to zero, because the model doesn't know what it doesn't know. Lean on it alone and a feature that admits ignorance nine times will deliver an invented answer on the tenth with exactly the same confidence.