A colleague flagged the wrong drug in my draft, and that is the only reason it never reached the customer.
I was creating documentation for a potential customer. The AI gave me a clean paragraph about the customer's research portfolio, and inside that paragraph it confidently attributed a specific drug to them. The detail sounded right. The therapeutic area was right. The register was right. I read it once, accepted it, and moved on to the next section.
A colleague proof-read the draft before it went out. He stopped on that one sentence. That drug was not being researched by the intended customer at all. It was being researched by a different pharma company. The AI had fabricated the attribution with full confidence, and I had not challenged it.
I sat there. The document was a day from leaving the building. Without that proof-read, the wrong document would have reached the customer.
If a confident sentence about a drug name slipped past me, what is slipping past you?
That is the part I want to sit with before anything else. The bland AI answer is the easy one to spot. You read it, you feel the genericness, you delete it and rewrite. The fluent AI answer that happens to be wrong is the dangerous one. It reads like the right answer. It carries the same calm tone, the same shape, the same authority as a sentence that is true. The interface gives you no surface tell.
This post is about why AI hallucination behaves that way, and what stops a fabricated sentence from leaving the building when the document is going to a customer.
AI is the new intern who will never say I do not know.
A new intern, given a question they cannot answer, will tell you. They hesitate. They say "I am not sure, let me check." That hesitation is information. It is the surface tell that protects the work.
AI does not hesitate. Ask it about a drug, a clause, a regulation, a person. It returns a smooth paragraph either way. Tone, grammar, sentence rhythm, paragraph length are identical for a real fact and an invented one. There is no tremor in the voice. The output that is right and the output that is fabricated arrive in the same envelope, and confident wrong answers are the failure mode that matters most because they are the ones you do not catch.
The research is blunt about this: The model is trained to guess, and it guesses wrong often enough to matter.
This is also where the question of why language models hallucinate gets its cleanest answer. Anthropic's interpretability write-up describes the mechanism without softening it. They say language model training incentivizes hallucination: models are always supposed to give a guess for the next word. That sentence, from Anthropic's interpretability research, is the part most readers skip past. The model is not adding a confident fabrication on top of an otherwise correct system. The confident fabrication is the system's default behaviour. Producing the next plausible word is the job. Whether that word is true is a downstream concern.
Now the rate. A Stanford RegLab paper, published in the Journal of Legal Analysis, profiled hallucinations across the largest publicly available models on real legal queries. They found that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2. The full study is worth reading, but the headline is enough on its own. This is not a niche failure inside a tiny corner of usage. This is the rate inside one of the most-tested professional domains on earth, on the kind of question lawyers ask every day.
If that is the band in law, AI hallucination in regulated industries sits inside the same band. Pharma documentation. Financial reports. Audit trails. Anything where one wrong noun sinks the document. The mechanism is the same and the work is at least as unforgiving.
The honest objection is that an expert can usually spot the bad sentence on sight.
Ethan Mollick has the cleanest version of this counterargument. In his One Useful Thing essay, he writes "Because you are expert, you will be able to quickly assess where the AI is wrong or right." I agree with the direction of that. On your own subject matter, you do have a strong filter.
But the drug-attribution sentence is the failure of that filter. The wrong drug was a plausible drug. It was in the right therapeutic area. It was written in the right register. The expert filter assumes the error looks wrong. Hallucinations are engineered, by training, to look right. So yes, expert review of AI output is part of the defense. But it works best when the expert is not the same person who generated the draft. The author already trusts the page. The fresh reader does not. The expert proof-read that catches the mistake is almost always the second reader, not the first.
What changed after that draft was the rule, not the prompt.
After the colleague's catch, the lesson I took home was not "write better prompts." Better prompts are useful. They were not the save here. The save was a second human, with the right expertise, reading the specific factual claims with a pen.
So the change was procedural. Nothing AI-generated leaves the building without a second pair of expert eyes on the named entities. I cross-verify any drug, company, person, or numeric claim against the source the customer would check. I ask AI to cite its sources, then I open them. I treat the draft as a draft, not as proofread output. That is what human in the loop means in practice. The human gatekeeper for AI generated documents closes the loop, not the prompt template above it. I still use AI for the same documentation work I always did. I just stopped treating fluent output as proofread output.
That is also how to catch AI hallucinations in pharma documentation, by routing every named entity through a reader who knows the domain. No tool replaces that reader.
The wrong takeaway is to add three more verification steps and call yourself safe.
The instinct after reading a post like this is to bolt a checklist onto your existing workflow and feel covered. Do not do that. Anthropic's own documentation, on the page where they tell developers how to reduce hallucinations, ends with this line. Remember, while these techniques significantly reduce hallucinations, they don't eliminate them entirely. Always validate critical information, especially for high-stakes decisions. That is from the developer documentation of the company that builds the model.
Reduction is not elimination. The only step that closes the loop is a human who knows the subject reading the specific claims before the document leaves. A longer prompt does not replace that step. A second model checking the first model does not replace that step. Human in the loop is not one of three options.
The question to sit with is which of your documents would survive a colleague reading them with a pen.
I still stay behind the wheel. I cross-verify, I ask AI for its sources, and nothing ships without expert review. The rule is the same on every document.
Pick one of yours. Hand it to someone who knows the subject. Watch where the pen stops.
