Explainable AI (XAI)

Explainable AI (XAI) is the set of methods that make the outputs of artificial intelligence (AI) models understandable to people: which inputs drove a prediction, how a model behaves overall, and when its answers deserve trust. Deep learning (DL) models with millions or billions of parameters act as black boxes, accurate but opaque, and XAI gives developers, users, auditors, and regulators a view inside. Explanations are used to debug models, detect algorithmic bias, justify decisions to the people they affect, and meet legal requirements. DARPA's Explainable AI program, launched in 2017, popularized the term.

Explainability vs. Interpretability#

The two words are often used as synonyms. When researchers distinguish them, interpretable models are understandable by design, such as linear models, small decision trees, and concept bottleneck models, while explainability methods produce after-the-fact explanations of models that are not. Cynthia Rudin argued in a 2019 Nature Machine Intelligence paper that high-stakes decisions should use interpretable models, since a post-hoc explanation can misrepresent what the model actually computes.

Two neighboring terms are broader or deeper. Transparency in AI covers openness about a whole system, including its data sources, design, and intended use. Mechanistic interpretability reverse-engineers the internal circuits of a network, an approach used mainly on large language models.

Types of Explanations#

Explanations differ in scope and in how much access they need. A local explanation accounts for one prediction, such as why one loan was denied, and a global explanation describes behavior across all predictions, such as which features matter most overall. Model-agnostic methods treat the model as a black box and only query its inputs and outputs, and model-specific methods use its internals, such as gradients or attention weights.

Common XAI Techniques#

• SHAP (SHapley Additive exPlanations): Assigns each feature a share of a prediction using Shapley values from cooperative game theory, so the contributions add up exactly to the difference between the prediction and a baseline.
• LIME (Local Interpretable Model-agnostic Explanations): Perturbs an input many times, records how the prediction changes, and fits a simple surrogate model, such as a sparse linear model, that mimics the black box near that one input.
• Permutation feature importance: Shuffles one feature at a time and measures how much the model's score drops, a global explanation that works with any model.
• Counterfactual explanations: Report the smallest change to the input that would flip the decision, such as the income at which a loan would have been approved.
• Saliency maps: Highlight the pixels that most influenced a vision model's output. Class activation map (CAM) methods such as Grad-CAM weight a network's feature maps by the gradients of the predicted class score to produce a coarse heatmap of the evidence.

Ultralytics YOLO26 builds a CAM method into prediction: setting visualize=True saves a LayerCAM heatmap, which weights each position of the feature maps entering the detection head by its own positive gradient toward the predicted class:

from ultralytics import YOLO

# Load a COCO-pretrained YOLO26 detector
model = YOLO("yolo26n.pt")

# Save a LayerCAM heatmap explaining only the "person" detections (class 0)
model.predict("https://ultralytics.com/images/bus.jpg", visualize=True, classes=[0])

The run folder gains bus_cam.jpg, a heatmap overlaid on the image in which warm colors mark the pixels that raised the person scores.

When Explanations Mislead#

An explanation can look convincing and still be wrong about the model. In "Sanity Checks for Saliency Maps" (2018), researchers randomized a network's trained weights and found that several popular saliency methods produced nearly the same maps as before, which showed the maps reflected edges in the input image more than anything the model had learned. A 2020 study showed that a deliberately biased classifier could be wrapped so that LIME and SHAP reported innocuous features while the underlying decisions stayed discriminatory. NIST's four principles of explainable AI address this directly: explanations must exist, be meaningful to their audience, and accurately reflect the system's process, and the system must operate only within the conditions it was designed for.

Applications and Regulation#

• Medical image analysis: Heatmaps let a radiologist confirm that a model flagged an X-ray for the lesion itself and catch models that key on scanner markers or text burned into the image.
• Shortcut detection: In the original LIME paper, explanations exposed a deliberately flawed husky-versus-wolf classifier that relied on snow in the background.
• Credit and lending: Under the Equal Credit Opportunity Act, US lenders must give applicants the specific principal reasons for a denial, with no exception for decisions made by complex algorithms.

The GDPR entitles people subject to automated decisions to meaningful information about the logic involved, Article 13 of the EU AI Act requires high-risk systems to be transparent enough for deployers to interpret their output, and Article 86 gives a person affected by a decision based on a high-risk system the right to clear and meaningful explanations of the AI system's role and the main elements of the decision.