Why a language model makes things up

A language model invents a citation, a date, a quote — confidently, in the same tone it uses for things that are true. People call this hallucination, as if the model normally reports facts and occasionally glitches. That framing gets it backwards, and the backwards version is why the problem is so persistent.

The model has one operation: given the text so far, predict the next word, then the next, over and over. It is not looking anything up. There is no database of facts inside it that it consults and occasionally misreads. There is a vast statistical sense of what word tends to follow what, learned from a lot of text.

That is the uncomfortable reframe: the model is generating plausible continuations every single time. When it tells you water boils at 100°C, it is not because it checked — it is because that continuation is overwhelmingly what the training text supports. The output is true, but the mechanism that produced it is the same mechanism that produces the fake citation.

So the right question is not “why does it sometimes hallucinate?” It is “why is it usually right?” And the answer is that for well-trodden facts, plausibility and truth line up. The failures happen exactly where they diverge: rare facts, specific numbers, names, anything where a confident-sounding guess is easy and the truth is thin in the training data.

Fluency is not confidence. The model writes a wrong answer in the same even, assured prose as a right one because it has no separate signal for “I actually know this.” It was trained to produce text that reads well, not to track its own certainty. The tone you read as confidence is just competence at writing.

It also has a bias toward answering. A model that shrugged and said “I don’t know” to everything would be useless, so it is tuned to produce something — which, at the edge of its knowledge, means producing something plausible rather than nothing.

The fixes all share a shape: give the model access to ground truth instead of asking it to recall. Retrieval — handing it the relevant documents to answer from. Tools — letting it run a search or a calculation rather than guessing the result. And verification — checking the claim against a source after the fact. None of these make the model “know” more; they change the task from remembering to reading.

Treat a language model as a fluent draft-writer, not an oracle. It is a plausibility engine that is right most of the time because plausible and true usually agree — and wrong, in the same calm voice, exactly where they do not. Knowing which situation you are in is the whole skill.