What Is AI Hallucination?

AI hallucination is the phenomenon where a large language model generates information that is factually incorrect, internally inconsistent, or entirely fabricated - while presenting it with the same confidence and fluency as accurate information. The model doesn't know it is wrong. It isn't lying. It is doing exactly what it was trained to do - predicting plausible tokens - but producing output that diverges from reality.

The term "hallucination" is borrowed from psychology, where it refers to perception without external stimulus. In AI, the analogy is apt: the model generates an "answer" that has no grounding in actual facts, sourced data, or verifiable reality. It sounds like something that could be true. It often looks authoritative. And it can be completely wrong.

What Is AI Hallucination?

Consider a user who asks an enterprise AI assistant: "What was our net revenue in Q3 2025?" The model, lacking access to internal financial data, generates a plausible-sounding figure - perhaps extrapolated from vague contextual signals in the conversation, perhaps simply a number that fits the expected distribution of answers. The user, trusting the confident response, includes it in a board deck.

This is a hallucination. The model didn't flag uncertainty. It didn't say "I don't have access to that data." It produced a wrong answer with the same tone and style as a right one. For low-stakes applications, this is an inconvenience. For enterprise use cases - financial reporting, compliance, clinical decisions, legal analysis - it can be a serious liability.

Why LLMs Hallucinate

Hallucination is a structural property of how LLMs are built and trained, so it has to be managed continuously rather than patched once. A 2025 paper by researchers at OpenAI and Georgia Tech traces it to both stages: pretraining leaves a model unsure about facts it saw rarely, and evaluations that grade answers only right or wrong reward guessing over admitting uncertainty (Kalai et al., 2025). Understanding the root causes is the first step to managing the risk.

The Statistical Nature of Language Modeling

LLMs are trained to predict the next token given a context. This means they are, fundamentally, probability machines over language - not truth machines over facts. A model trained on a corpus containing millions of texts learns the statistical patterns of how people write about things, not the facts themselves. When it generates text about a topic it hasn't encountered in its training data, it generates the most statistically probable continuation - which may be factually wrong. Kalai et al. show that facts appearing only once in the training data, such as a little-known person's birthday, are especially exposed: the share of such one-off facts sets a rough lower bound on how often a base model will get them wrong.

Confidence the Reader Cannot See

A model's next-token probabilities do carry information about confidence. The GPT-4 Technical Report (OpenAI, 2023) found the pre-trained model well calibrated on a subset of the MMLU benchmark, and found that post-training reduced that calibration. But a chat answer does not show those probabilities, and the prose reads equally sure whether the model is recalling something well covered in training or extrapolating. Techniques like calibration, chain-of-thought reasoning, and retrieval-augmented generation partially address this, but the baseline model generates fluently whether it has solid ground or none.

Training Data Gaps and Outdated Knowledge

LLMs are trained on a static snapshot of text. Anything that happened after the training cutoff, or that wasn't well-represented in the training data (internal company data, niche technical domains, recent regulatory changes), is unknown to the model. When asked about these topics, the model's options are to say "I don't know" or to generate plausible-sounding text. Kalai et al. argue that common benchmarks push models toward the second option: most of them give no credit for "I don't know", so a model that guesses scores better than one that abstains.

Types of Hallucination

Not all hallucinations are the same. Different types require different mitigation strategies. Research surveys such as Huang et al. (2023) separate factuality hallucinations, which conflict with verifiable real-world facts, from faithfulness hallucinations, which diverge from the user's input or the context provided. The four practical types below fall on either side of that split.

• Factual hallucination - The model states an incorrect fact. "The GDPR has applied since May 2018" (correct, per Article 99) vs. "The GDPR has applied since 2016" (wrong). These are often the easiest to catch if you have a verification layer.
• Contextual hallucination - The model contradicts information provided in the context window. The user provides a document and asks a question; the model answers with information not in the document, or contradicting it. This suggests retrieval is working but the model is still going off-context.
• Source hallucination - The model cites a source that doesn't exist: a paper with a real-sounding title, a regulation with a plausible citation number, a dataset name that isn't in any catalog. Particularly dangerous because the hallucinated citation looks authoritative. In Mata v. Avianca (S.D.N.Y., June 22, 2023), a US federal court sanctioned lawyers who had submitted non-existent judicial opinions with fake quotes and citations created by ChatGPT.
• Logical hallucination - The model's reasoning steps are internally inconsistent, or the conclusion doesn't follow from the stated premises. Common in multi-step reasoning tasks where the model loses track of intermediate results.

Business Risks

In consumer applications, hallucinations are embarrassing. In enterprise applications, they create measurable risk:

• Compliance violations - An AI assistant citing the wrong regulatory threshold, misquoting a policy, or generating a compliance report with fabricated metrics creates audit exposure.
• Financial error - Hallucinated numbers in financial analysis, cost modeling, or revenue reporting can propagate through downstream processes before anyone catches them.
• Legal liability - In customer-facing AI systems, a hallucinated claim about a product capability or a misquoted contract term can create contractual or regulatory liability.
• Trust erosion - Once users encounter a clear hallucination, trust in the system can drop sharply - even for queries where the model would have been correct. The cost of a single high-profile failure can outweigh the productivity gains from many correct answers.

How to Reduce Hallucination

The toolbox for reducing hallucination has matured considerably:

1. Retrieval-augmented generation (RAG) - Don't let the model answer from parametric memory for queries that require specific, current, or private data. Retrieve authoritative context and instruct the model to answer from it. RAG was named in a 2020 paper (Lewis et al.), and in human evaluations of knowledge-grounded dialogue, retrieval augmentation substantially reduced knowledge hallucination (Shuster et al., 2021). It does not eliminate it: a model can still contradict the context it was given. See the full entry on RAG in AI.
2. Citation enforcement - Require the model to cite the source for every factual claim. If the model can't cite a source from the provided context, it should flag uncertainty rather than generating a claim.
3. Output verification pipelines - For high-stakes use cases, build automated verification: extract factual claims from the model's output and check them against authoritative sources. This is most tractable when those sources are in a well-maintained data catalog.
4. Constrained generation - Limit the model's answer space. "Choose from the following five options" is harder to hallucinate than "tell me anything." For classification and structured extraction tasks, constraining outputs narrows the room for hallucination.
5. Human-in-the-loop for high-stakes decisions - For decisions where hallucination has severe consequences (compliance signoffs, financial approvals, medical guidance), maintain human review. AI reduces the burden; humans catch errors the model won't flag.

Data Governance as Hallucination Defense

Better models help, but for questions about an organization's own data, much of the defense comes from data governance. Three practices matter most:

A maintained business glossary gives the model a governed, version-controlled source of truth for business definitions. When the model retrieves the definition of "active customer" from a governed glossary rather than generating it from training data, the risk of a hallucinated definition is much lower.

A current data catalog tells the model which datasets exist, who owns them, what they contain, and when they were last updated. This prevents the model from inventing dataset names, citing outdated tables, or confusing two similarly-named assets.

Data lineage enables auditability - the ability to trace an AI-generated answer back through the retrieval chain to the underlying data. When a hallucination does occur, lineage makes it detectable and debuggable.

Conclusion

AI hallucination is a structural property of language models, and better models reduce it without removing it. Managing it in enterprise environments means treating it partly as a data governance problem: ensuring the model receives accurate, governed, current context via a well-maintained retrieval layer, and building verification infrastructure that catches errors before they propagate. Organizations that have invested in data catalogs, business glossaries, and knowledge graphs are better positioned to deploy trustworthy enterprise AI - because their retrieval layers are built on authoritative foundations.

Sources

• Kalai et al., Why Language Models Hallucinate, arXiv, September 2025.
• OpenAI, GPT-4 Technical Report, arXiv, March 2023.
• Huang et al., A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, arXiv, November 2023.
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv, May 2020.
• Shuster et al., Retrieval Augmentation Reduces Hallucination in Conversation, arXiv, April 2021.
• gdpr-info.eu, Art. 99 GDPR - Entry into force and application.
• U.S. District Court for the Southern District of New York, Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions, June 22, 2023.