I don’t love the word “hallucination.” It suggests a glitch — a model briefly malfunctioning before snapping back to normal. What’s actually happening is closer to the opposite. The model is doing exactly what it was built to do: generate fluent, confident, plausible continuations from a learned distribution. Sometimes that distribution is anchored to something true. Sometimes it isn’t. The model has no internal alarm that goes off in the second case. Fluency doesn’t degrade. Confidence doesn’t drop. That’s not a malfunction. That’s the mechanism working as designed, pointed at the wrong target.
The mechanism is one thing. The outcome is two.
NIST doesn’t call it hallucination, for what it’s worth. Its Generative AI Profile — the operational companion to the AI Risk Management Framework I’ve written about before — names it “confabulation” instead: the confident production of false content. NIST treats it as one of twelve named risk categories, with more than two hundred suggested mitigating actions attached. That’s the right instinct. You don’t write two hundred suggested actions for a bug. You write them for a structural property of the system you have to manage around.
OpenAI’s own researchers published the clearest mechanistic account of why, in September 2025. Kalai, Nachum, Vempala, and Zhang trace hallucination to something almost mundane: training and evaluation procedures that reward confident guessing over admitting uncertainty. If a model can’t reliably distinguish a true statement from a false one during training, and the benchmarks it’s graded against give zero credit for “I don’t know” and partial credit for a plausible wrong answer, the model that guesses will outscore the model that hedges. That’s not a flaw introduced by accident. That’s the optimization target doing exactly what it was told to do.
Xu, Jain, and Kankanhalli went further in 2024, and proved something stronger: hallucination isn’t just likely under current training regimes, it’s unavoidable. Using computability theory, they show that a language model — modeled formally as a computable function — cannot learn every computable function it might be asked to reproduce. Some gap between what the model can compute and what’s actually true will always exist. You can shrink it. You can’t close it. Not with more data, not with more parameters, not with more RLHF. It’s a property of the object, not a bug in a particular build of it.
Inside the model, it’s a single circuit — and it protects you, right up until it misfires
Anthropic’s interpretability team traced the actual mechanism inside Claude, and the finding reframes the whole question. Refusal is the model’s default behavior — a circuit that’s “on” unless something turns it off. Ask about a well-known entity and a competing “known entity” feature activates, suppresses the refusal circuit, and the model answers. That’s the useful case: the same mechanism that lets a model answer fluently and confidently about things it actually knows. Researchers then tested what happens with a fabricated person’s name. Claude initially declined, correctly. But when they injected a subtle signal suggesting familiarity, the “known entity” feature misfired, suppressed the refusal circuit anyway, and the model started generating confident, detailed, entirely false biographical content.
Hallucination isn’t the absence of a safety mechanism. It’s the same safety mechanism, misfiring in the other direction.
You can’t remove that circuit without removing the thing that makes the model useful in the first place — its ability to answer confidently about what it does know, without hedging every sentence into uselessness.
The same mechanism, aimed on purpose
This is where “feature” stops being a rhetorical stretch. A 2026 paper puts the phrase directly in its title — “Hallucination as a Feature, not a Defect” — and evaluates a multi-agent architecture that does something specific with it: treats a model’s speculative, ungrounded output not as an error to suppress, but as raw material — a generator of testable scientific hypotheses, verified by a separate process before anyone trusts them. That’s not tolerance for wrong answers. That’s the same distinction I made in Guardrails Are Not Governance, applied one layer down: the generation doesn’t have to be true. The system around it has to be able to tell the difference before anyone acts on it.
There’s empirical weight behind the tradeoff, too. Research presented at AAAI 2026 investigated the relationship directly — asking whether less hallucination means less creativity — and found the two are entangled, not independent dials. Creativity often looks like an unconventional associative leap: connecting two things that don’t obviously belong together. That’s structurally close to what a model is doing when it hallucinates. Suppress the mechanism hard enough to kill the second, and you measurably dampen the first.
Guardrails channel it. They don’t remove it.
This is where “guardrails” earns its place from Guardrails Are Not Governance again, but pointed at a narrower job than governance: not deciding who’s accountable, just deciding where the generative mechanism is allowed to run free and where it has to stay on a leash.
Structured output is the clearest example. Constrained decoding forces a model’s output to conform to a schema — a fixed set of fields, a JSON shape — by masking out any token that would break the structure, at every step of generation. A 2025 benchmark of the technique is honest about what it actually buys you: it guarantees the shape is valid. It does not guarantee the values inside that shape are true. A perfectly-formed JSON object can still contain a fabricated field. That’s the guardrail doing its real job — constraining the space the model is allowed to hallucinate into, not whether it hallucinates.
Sampling parameters do something similar, more crudely. Temperature and top-p trade exploitation — safe, high-probability, grounded-sounding completions — against exploration: novel, lower-probability, more associative ones. Turn either up and you get more creative diversity at a real cost to factual reliability. A 2024 study asking directly whether temperature is “the creativity parameter” found the honest, less tidy version of that story: only a weak correlation between temperature and novelty, a moderately negative one between temperature and coherence. Not a clean dial. A real, measurable tilt between staying close to what’s grounded and wandering.
The most direct guardrail is also the simplest: telling the model, explicitly, that it’s allowed to not know something. Anthropic’s own published guidance for reducing hallucination in Claude leads with exactly this — give the model an explicit out, and it reaches for it more often instead of guessing. A separate 2025 paper on calibrating verbal uncertainty found the same effect from the inside: a model’s own stated confidence, elicited directly, is a workable signal for when to answer and when to abstain.
None of these guardrails touch the underlying mechanism Kalai, Nachum, Vempala, and Zhang described, or the inevitability Xu, Jain, and Kankanhalli proved. They don’t stop the model from generating a confident, fluent, ungrounded sentence when nothing stops it. What they do is narrower and more honest: shrink the space where that sentence is allowed to land unchecked, and make “I don’t know” a more reachable answer than a guess. That’s channeling the mechanism toward the outcome you actually want, not suppressing it — the same distinction, one more time, between constraining behavior and being naive about what the constraint actually buys you.
What happens without a system around it
None of this is an argument for shipping ungrounded output into anything that matters. Mata v. Avianca is the case everyone already half-remembers: in 2023, lawyers representing a plaintiff in a personal injury suit against Avianca Airlines filed a brief citing six court decisions as precedent. All six were fabricated by ChatGPT — the cases didn’t exist. Opposing counsel couldn’t find them. The court couldn’t find them either, and ordered the attorneys to produce copies. Instead of correcting course, the attorneys filed an affidavit attaching “excerpts” from the fake opinions — also fabricated. The judge sanctioned them $5,000 and ordered them to personally notify every real judge the model had falsely named as an author. Nothing about that outcome required the model to be broken. It required zero verification between generation and filing.
What the system has to do instead
Vectara has been measuring this continuously, model by model — a leaderboard scoring how often each major model hallucinates when summarizing real news articles it’s given, grounded, using a dedicated evaluation model against a dataset of thousands of articles. Every model on it hallucinates sometimes. The rate varies by model and by month, but no model has ever topped the chart with a zero. That’s the wrong thing to be surprised by, and the wrong thing to try to buy your way out of by switching vendors. Grounding is the standard system-level answer, and it’s not new: Lewis et al. published retrieval-augmented generation in 2020, pairing a language model with a retriever that hands it real source documents to condition on, so the model has something to copy or paraphrase instead of guess. It doesn’t eliminate the mechanism. It gives the mechanism something true to point at, and it gives a verification layer something to check the output against afterward.
What I’d actually ask
Not “does this model hallucinate.” Per Vectara’s own tracking, they all do, measurably, continuously. The better questions:
- What’s grounded, and what’s generated from the model’s own weights with nothing to check it against?
- What catches an ungrounded claim before a user or a downstream system acts on it — and is that a real verification step, or just a disclaimer at the bottom of the page?
- Where do we actually want the associative, ungrounded version of this mechanism — brainstorming, hypothesis generation, first drafts — where confident wrongness is cheap to catch and the alternative is a blank page?
- What’s actually constraining generation — a schema, a sampling setting, an explicit instruction to abstain — versus a prompt instruction nobody has tested against a model that will confidently ignore it?
The mechanism doesn’t need fixing. The system around it needs building — same argument, one layer down from where I made it about guardrails.
Sources
- National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile” (NIST AI 600-1) — twelve named risk categories including “Confabulation,” 200+ suggested mitigating actions. nvlpubs.nist.gov
- Kalai, Nachum, Vempala & Zhang, “Why Language Models Hallucinate,” OpenAI, September 2025 — training and evaluation procedures that reward confident guessing over admitting uncertainty. arxiv.org
- Xu, Jain & Kankanhalli, “Hallucination is Inevitable: An Innate Limitation of Large Language Models,” 2024 — a computability-theory proof that hallucination cannot be fully eliminated. arxiv.org
- Anthropic, “On the Biology of a Large Language Model” — the “known entity” circuit, refusal as default behavior, and the misfire mechanism behind hallucination. transformer-circuits.pub
- “Hallucination as a Feature, not a Defect: Evaluating a Multi-Agent Architecture to Transform Speculative Language-Model Outputs into Testable Scientific Hypotheses,” 2026. arxiv.org
- “Does Less Hallucination Mean Less Creativity? An Empirical Investigation in LLMs,” AAAI 2026 — the entangled relationship between hallucination suppression and creative/associative capability. papernotes.org
- Mata v. Avianca, Inc., 678 F.Supp.3d 443 (S.D.N.Y. 2023) — attorneys sanctioned $5,000 for filing a brief citing six fabricated ChatGPT-generated cases. law.justia.com
- Vectara, Hallucination Leaderboard — continuous, model-by-model measurement of hallucination rate on grounded summarization tasks. github.com
- Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 2020 — the original RAG paper; grounding a language model in retrieved source documents. neurips.cc
- “Generating Structured Outputs from Language Models: Benchmark and Studies” (JSONSchemaBench), 2025 — constrained decoding guarantees schema validity, not the truth of the values inside it. arxiv.org
- “Is Temperature the Creativity Parameter of Large Language Models?”, ICCC 2024 — a weak correlation between temperature and novelty, a moderately negative one between temperature and coherence. computationalcreativity.net
- Anthropic, “Reduce hallucinations,” Claude Platform Docs — first-party guidance on giving a model explicit permission to abstain instead of guess. platform.claude.com
- “Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations,” 2025 — a model’s own elicited confidence as a workable answer-or-abstain signal. arxiv.org