What Is Explainable AI (XAI)?

Explainable AI (XAI) is the ability to reconstruct and justify how an AI system produced a particular outcome. It is not a property of a model so much as a property of a system plus its records: an explanation has to survive being questioned by someone who was not there, months later, without access to the person who built it.

The reason explainability has moved from a research topic to an operational requirement is that the audience changed. Early XAI work was aimed at data scientists debugging their own models. The audience now includes a customer who was declined, an internal auditor testing a control, a regulator assessing conformity, and increasingly a court. Each needs a different explanation of the same decision, and none of them will accept a feature-importance chart as an answer. That shift is why explainability today means auditability: tracing inputs, processing, and outputs in a way that supports review and accountability.

Explainable AI Defined

An explanation is a claim about causation that a specific audience can check. That framing makes two things obvious that get lost in technical discussions of XAI.

First, explanations are audience-relative. A developer needs to know which features moved the output and whether the model is relying on a proxy it should not be. A loan applicant needs to know what about their application drove the decision and what would change it. An auditor needs to know that the process was followed and that the same input would yield the same output. A regulator needs to know the system was assessed, documented, and overseen. Producing one artifact and calling it "the explanation" satisfies at most one of them.

Second, explanations decay unless they are recorded. The model version that produced last March's decision may not exist now. The reference data it consumed has changed. The prompt has been edited four times. If an explanation can only be regenerated by re-running the system as it exists today, it is not an explanation of what happened - it is a plausible reconstruction, and the difference matters precisely when someone is contesting the outcome.

This is why mature programs treat explainability as a lifecycle obligation rather than a modeling technique. The requirement is defined at intake, designed into the system, and evidenced continuously.

NIST's Four Principles

The most useful specification of what an explanation must satisfy comes from NIST's report Four Principles of Explainable Artificial Intelligence (NIST IR 8312, published September 29, 2021). Its four principles are that the system produces an explanation, that the explanation is meaningful to humans, that the explanation accurately reflects the system's processes, and that the system expresses its knowledge limits.

• Explanation. The system delivers accompanying evidence or reasons for its outputs. This is the floor, and it is the principle most often satisfied in isolation - a score with a chart attached.
• Meaningful. The explanation is understandable to its intended recipients. Since recipients differ, a system serving several audiences owes several explanations. An explanation that only a model developer can read fails this principle for every other audience.
• Explanation accuracy. The explanation correctly reflects the system's actual process for generating the output. This is the principle that post-hoc techniques strain, because an approximation of a model's reasoning can be plausible and wrong. NIST notes that misleading explanations are demonstrably constructible, which is a strong argument for treating an explanation as something to be validated rather than trusted because it looks reasonable.
• Knowledge limits. The system operates only under conditions for which it was designed, and signals when a case falls outside them. This is the principle that connects explainability to safety: a system that answers confidently outside its competence is producing explanations for outputs that should never have been issued.

The set is worth reading as a checklist because the four principles fail independently. Most real systems satisfy the first, struggle with the second, rarely test the third, and ignore the fourth entirely.

Explainability vs Interpretability

The two words are used interchangeably in marketing and mean different things in practice, and the difference drives a design decision worth making deliberately.

• Interpretability is a property of the model itself. A linear model, a small decision tree, or a scorecard can be read directly: the mechanism is the explanation. Nothing is approximated.
• Explainability is what you add around a model whose mechanism cannot be read. Post-hoc techniques produce an account of the model's behavior that is, by construction, an approximation.

The trade-off is real but frequently overstated. An interpretable model gives up some predictive performance and gains an explanation that cannot be wrong about itself, which is often the better trade for high-stakes decisions about people. This is a tiering question as much as a modeling one: for a top-tier use case where individual outputs must be justified to the affected person, choosing an inherently interpretable model can be cheaper than building, validating, and maintaining an explanation layer over an opaque one. See AI risk tiering for where that decision belongs in the process.

Generative systems complicate the picture further, because a fluent natural-language rationale is not evidence. A language model asked why it produced an answer will produce a plausible account, and that account is generated the same way the answer was - it is not a readout of the computation. For LLM applications, the auditable explanation usually lives outside the model: which documents were retrieved, which governed definitions were in context, which tools were called, and what the system was permitted to do. That is grounding and traceability rather than introspection.

How Explanations Are Produced

The technique families are worth knowing at the level of what question each answers, and where each one breaks.

• Feature attribution. Which inputs moved this output, and by how much. Useful and widely deployed. It breaks when features are correlated - attribution then distributes credit in ways that read as causal and are not - and it says nothing about what would have changed the outcome.
• Counterfactuals. The smallest change to the input that would have flipped the output. This is usually the most meaningful explanation for an affected person, because it is actionable. It breaks when the suggested change is not something the person can act on, or is a proxy for a protected characteristic.
• Surrogate models. Fit a simple, readable model to the complex model's behavior, locally or globally. Cheap and intuitive, and the technique most exposed to the accuracy principle: a surrogate can match the model's outputs on observed data and diverge exactly where it matters.
• Example-based explanation. Show the most similar training cases, or the influential ones. Concrete and intuitive, with an obvious privacy hazard when the examples are real people's records.
• Documentation artifacts. Model cards, data sheets, and intended-use statements. Not per-decision explanations, but they carry the knowledge-limits principle better than any per-decision technique, and they are what a conformity assessment actually reads.

None of these is sufficient alone, and the failure to test them is the systemic weakness. An explanation is a claim, and claims should be validated - by checking whether the explanation is stable under small input changes, whether it agrees across techniques, and whether the audience it targets can actually act on it.

Where the Law Requires It

Explainability has moved from good practice to obligation in several overlapping regimes. The details matter, because the scope is narrower than the headlines suggest and the practical burden is wider.

• EU AI Act, Article 86. A person affected by a decision a deployer took on the basis of a high-risk AI system's output has the right to obtain clear and meaningful explanations of the role of the AI system in the decision-making procedure, where the decision produces legal effects or significantly affects their health, safety, or fundamental rights. The right applies to the high-risk systems listed in Annex III, excluding point 2, and does not apply where Union or national law provides an exception. Note what it asks for: the AI system's role in the procedure, which is a process explanation as much as a technical one.
• EU AI Act transparency and documentation duties. High-risk systems must be designed so deployers can interpret output and use it appropriately, with technical documentation and record-keeping to match. This is where the Act puts most of the actual engineering burden.
• GDPR, Article 22. Decisions based solely on automated processing, including profiling, that produce legal effects or similarly significantly affect a person are restricted, and where they are permitted the controller must implement safeguards including the right to obtain human intervention, to express a point of view, and to contest the decision. Contesting a decision you cannot see the basis of is not meaningful, which is why Article 22 has functioned as an explainability requirement in practice.
• Sector rules. Credit, insurance, employment, and healthcare each carry their own adverse-action and justification duties that predate AI and apply to it unchanged. In banking, model risk management expectations already require documented understanding of a model's approach and limitations.

The pattern across all of them: the law asks for an explanation a specific person can use, delivered at the time it matters, with a record that survives afterward. That is a governance capability, not a library.

Explainability Needs Lineage

Here is the gap that sinks most explainability programs. A team invests in model-level explanation, produces good attributions, and then cannot answer the first question an auditor asks - what data was this computed from, and what did those fields mean at the time?

An explanation that omits the data layer is unfalsifiable. If nobody can establish which source a feature came from, whether it was the authoritative one, what transformations it passed through, or whether the definition of the underlying term changed six months ago, then no amount of model interpretability produces an auditable account. Two systems can compute "customer tenure" differently and both be explainable in isolation while contradicting each other, and the explanation for the contradiction lives entirely in the data layer.

This is where Dawiso fits, and the boundary is worth stating: Dawiso does not generate model explanations. It supplies the two layers that make a model explanation auditable. Interactive lineage is the provenance record - what fed the system, through which transformations, from which authoritative source - and it is the same artifact that answers impact questions when something upstream changes. The business glossary holds the governed definition of every term, so an explanation refers to a meaning the organization has agreed on rather than a column name. The catalog establishes what was authoritative and who owns it, and classification records which inputs were sensitive. Together with AI governance for the process layer, that turns "we can explain the model" into "we can reconstruct and defend the decision."

Conclusion

Explainable AI is best understood as auditability rather than transparency for its own sake: the ability to reconstruct an outcome and justify it to whoever is asking. NIST's four principles set a useful bar because they fail independently - producing an explanation is easy, making it meaningful is harder, proving it accurate is rarely attempted, and stating knowledge limits is usually skipped. Regulation has now attached rights to the first two, so the bar is no longer optional for high-stakes systems. What separates programs that can meet it from programs that cannot is not the choice of technique. It is whether the data underneath every AI decision was defined, traced, and owned before anyone needed to explain it.

Sources

• NIST - Four Principles of Explainable Artificial Intelligence (NIST IR 8312)
• NIST - NIST IR 8312, full report (DOI)
• EU Artificial Intelligence Act - Article 86: Right to Explanation of Individual Decision-Making
• GDPR - Article 22: Automated individual decision-making, including profiling
• NIST - AI Risk Management Framework (AI RMF 1.0)