The AI confidently listed a phone model that never existed
I was planning to buy a new phone. I had my requirements ready: budget, camera priority, intended use. I opened Gemini, described what I needed, and asked it to find the best deal. The response came back confident and detailed. Model number, processor specs, camera comparison, a price breakdown against two competing handsets. It sounded like something a well-informed sales consultant would say.
It confidently listed a model that never existed.
The specs were specific. The price comparisons had numbers that looked real. But something felt off, so I went back to check. Only after checking the original source did I discover the phone had never been manufactured. That is what it means when they say AI makes up facts. The recommendation was complete, confident, and entirely invented.
The AI was not trying to deceive me. It was doing exactly what it was designed to do.
If the model produces a fabricated recommendation with the same confidence it uses for something real, how is anyone supposed to know when to trust it? The real question is why AI makes up information confidently rather than saying it does not know. That is a diagnostic question, not a scare story. The AI wasn't trying to deceive me. It was not malfunctioning. The AI wasn't lying. It was doing exactly what it was designed to do.
Understanding that mechanism is what changes how you use the tool. The failure is predictable once you see the pattern.
Consider AI to be just like your brain, which would use all the different references to derive a conclusion
Think about the last exam you sat without studying. Your brain did not go blank. It assembled the most plausible answer from whatever fragments it held: partial definitions, half-remembered examples, context clues in the question. It did not verify. It constructed. And the answer came out confident, regardless of whether it was right.
That is why AI hallucinates. An LLM does the same thing at a different scale. It predicts the most likely sequence of words based on patterns it learned during training. There is no lookup. There is no database query. The model is not checking a source when it returns an answer. It is generating text that follows statistically from whatever came before. That is why the fabricated phone recommendation sounded so authentic. It fit the pattern of what a phone recommendation looks like.
Five conditions make a hallucination more likely, and most of them are inside the prompt
A survey from Harbin Institute named the pattern precisely: LLMs are prone to hallucination, generating plausible yet nonfactual content. LLM hallucination is not random. Five causes come up repeatedly, and three of them sit on your side of the prompt.
Missing information is the first. When the model lacks reliable knowledge of a topic, either because the topic was under-represented in training data or because it postdates the training cutoff, the model does not stop. It fills the gap with what is statistically adjacent. The phone I asked about may never have appeared in any training corpus. The model built a description from parts of other phones it had seen.
Ambiguous questions push the risk higher. A research survey describes models as capable of extrapolating information from the biases in training data, misinterpreting ambiguous prompts, or modifying the information to align superficially with the input. When your prompt could mean two things, the model picks one interpretation and runs. It will not ask you to clarify.
Training limitations extend the same problem. The model inherits whatever gaps or outdated information existed in its training set. It cannot flag what it does not know with confidence, and it does not distinguish between topics it knows well and topics where its training was thin.
Long conversations introduce a structural failure mode. AI hallucination in long conversations is not just a fluency problem. A research analysis found that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Context volume alone does not protect you.
Overconfidence is the fifth. The model delivers every output with the same tone. The fabricated phone recommendation and an accurate one looked identical. There is no verbal hedge, no wavering. You get the same flat confidence either way.
Newer models do hallucinate less, but that improvement does not reach zero
Anthropic's Claude 3 announcement noted that Opus demonstrates a twofold improvement in accuracy on challenging open-ended questions, while also exhibiting reduced levels of incorrect answers. That is real progress. But improvement is not elimination.
But "less often" is not "reliably not." The five conditions above still operate. A vague prompt to a current model is still more likely to produce fabricated output than a scoped, source-anchored prompt to the same model. Waiting for the next model release is a different frame from changing what you put into the prompt.
This post is about that second frame. Two things you can do before the generation starts: scope the question precisely, and name the reference the model should work from. Both happen before you press send.
The second answer told me which paragraph to rewrite, because I named the source it should use
In August 2024, I came back to ChatGPT after a six-month gap. The second time, I stopped searching and started talking. I began supplying role, background, and both sides of the coin. The bland prompt: "Summarise this stakeholder interview transcript." The scoped prompt: "You are a consultant at a pharma IT company. I am about to write a requirements document for a new clinical data review feature. Below is a 40-minute interview transcript with the head of clinical operations. Pull out the three things she clearly wants, the two things she said no to, and the one thing she contradicted herself about."
Same transcript, same model. The second answer told me which paragraph I needed to rewrite before showing the document to anyone.
What changed between those two prompts was not just the role or the background. The transcript itself was the named reference. I was not asking the model to recall anything from training. I was pointing it at a specific document and giving it a precise extraction task.
The Role / Objective / Background / Data shape works because it does this systematically. Role sets the lens. Objective names the task precisely. Background gives the model the context it would otherwise guess at. Data is the named reference: not training memory, but the actual document you want the model to work from. A named source in the prompt reduces hallucination because it narrows what the model fills in on its own.
A well-scoped prompt to reduce hallucination is not a trick. It is better input for a pattern-matching machine.
Correcting a hallucination inside a long conversation can make the next answer worse
One more thing worth knowing before you close this tab.
If you catch a fabrication mid-session and try to correct it by arguing the model back to accuracy, the model may commit harder to the original error. A research paper named this: We refer to this phenomenon as hallucination snowballing: an LM over-commits to early mistakes, leading to more mistakes that it otherwise would not make.
The model is not getting defensive. It is doing what it always does, predicting the most plausible continuation of the context it has accumulated. The early mistake is now part of that context.
When you spot a clear fabrication in a long session, start a fresh conversation. A clean prompt in a new session removes the context the model was building on. You are not negotiating with memory. You are restarting the pattern.
Before the next prompt lands, ask yourself what source you would hand a new hire
You are behind the wheel here, not the model.
Before you send the next question, ask this: if I were onboarding a new employee, what source document would I give them before asking this question? If you can name it, put it in the prompt. If you cannot, rethink the question first.
Three seconds before you press send. That is the habit.
