Mechanistic Interpretability

Mechanistic interpretability is the research field that reverse-engineers trained neural networks into human-understandable mechanisms: the features a model represents in its activations and the circuits, chains of weights and components, that turn those features into an output. Where most evaluation treats a model as a black box judged by its predictions, mechanistic interpretability treats the weights like a compiled program to be decompiled. It has grown from studies of small vision models into a core part of AI safety research on large language models (LLMs), and MIT Technology Review named it one of its 10 Breakthrough Technologies of 2026.

Features, Circuits, and Superposition#

The field rests on a few working claims, laid out for image classifiers in the Distill article Zoom In: An Introduction to Circuits:

• Features: Directions in a layer's activation space that correspond to meaningful properties of the input. Early layers of a convolutional neural network (CNN) contain curve and edge detectors, and later layers contain features for object parts such as wheels or dog snouts.
• Circuits: Groups of features connected by weights that implement an algorithm. In transformers, induction heads are pairs of attention heads that find an earlier occurrence of the current token and copy what followed it, a mechanism behind in-context learning. A study of GPT-2 small mapped the circuit that completes sentences such as "When Mary and John went to the store, John gave a drink to" with "Mary" to 26 attention heads in 7 functional groups.
• Superposition: Models represent more features than they have neurons by overlapping them, so a single neuron often responds to several unrelated concepts. This polysemanticity is why individual neurons make poor units of analysis and why the field turned to dictionary learning.

Key Methods#

Researchers combine observation, which finds candidate features, with intervention, which tests whether they matter:

• Linear probing: Training a simple classifier on activations to check whether a concept is linearly decodable at a given layer. A probe shows that information is present, and an intervention is needed to show that the model uses it.
• Ablation and activation patching: Zeroing an activation, or swapping in its value from a different input, then measuring how the output changes. These causal tests separate components that compute a behavior from ones that merely correlate with it.
• Sparse autoencoders (SAEs): Networks that decompose a layer's activations into a much larger set of sparsely active features, undoing superposition. Anthropic's Scaling Monosemanticity extracted millions of features from Claude 3 Sonnet, including one for the Golden Gate Bridge; clamping it to a high value produced a model that described itself as the bridge.
• Attribution graphs: Anthropic's 2025 circuit tracing work on Claude 3.5 Haiku replaced the model's MLP layers with an interpretable approximation and traced feature-to-feature paths, showing the model reaching "Austin" from "the capital of the state containing Dallas" through an intermediate "Texas" step, and choosing a rhyming word before writing the line of a poem that ends with it.
• Steering: Adding or subtracting a feature direction during inference to change behavior, the basis of steering vectors and representation engineering.

A typical investigation alternates between these methods:

Ablating Channels in a Vision Model#

Ablation is the simplest causal test to run. The snippet below loads a pretrained ResNet-18 from torchvision, uses a PyTorch forward hook to zero each of the 256 channels of an intermediate layer's feature maps in turn, and records how much the logit of the model's top class falls:

import urllib.request

import torch
from PIL import Image
from torchvision.models import ResNet18_Weights, resnet18

weights = ResNet18_Weights.DEFAULT
model = resnet18(weights=weights).eval()
image = Image.open(urllib.request.urlopen("https://ultralytics.com/images/bus.jpg"))
x = weights.transforms()(image).unsqueeze(0)

with torch.no_grad():
    logits = model(x)
    top = logits.argmax().item()
    drops = torch.zeros(256)
    for c in range(256):  # zero one layer3 channel at a time and record how far the top-class logit falls
        handle = model.layer3.register_forward_hook(lambda m, i, out: out.index_fill(1, torch.tensor([c]), 0))
        drops[c] = logits[0, top] - model(x)[0, top]
        handle.remove()

values, channels = drops.topk(3)
print(f"{weights.meta['categories'][top]}: logit {logits[0, top]:.2f}, median drop {drops.median():.3f}")
print(f"largest drops: channels {channels.tolist()} -> {values.round(decimals=2).tolist()}")

The model predicts "minibus" with a logit of 13.42. Removing a typical channel changes that logit by almost nothing (median drop -0.002), while removing channel 62 alone cuts it by 1.80 and channels 70 and 155 by about 1.15 each, so a handful of channels carry much of the evidence. Those channels become the starting point for the next steps: visualizing what each one detects and tracing which earlier features feed it.

Mechanistic Interpretability vs. Explainable AI#

Explainable AI (XAI) is the broader field of making model decisions understandable, and much of it is attribution: saliency maps and class activation maps highlight which input pixels drove a prediction. Setting visualize=True in Ultralytics YOLO26 predict mode saves such a heatmap next to each prediction, showing which pixels raised the predicted class scores. Attribution answers where the model looked; mechanistic interpretability asks which internal features and circuits turned those pixels into the score. Concept bottleneck models take a third route, building interpretability into the architecture by forcing predictions through human-named concepts.

Applications and Limits#

Mechanistic tools are used to audit LLMs for unwanted behavior before release, such as monitoring activation directions linked to deception or sycophancy, and to debug vision models that rely on shortcuts, as when a skin lesion classifier learns that rulers, which dermatologists place next to suspicious lesions, signal malignancy. That kind of check matters in medical image analysis, where a model can score well on a test set for the wrong reason. Open tools such as TransformerLens make hooks, caching, and patching on transformer models routine.

The field's main limit is coverage. SAEs reconstruct activations imperfectly, so part of every computation stays unexplained, and the replacement models used for attribution graphs capture only some of a frontier model's behavior on a given prompt. Each circuit still takes substantial human effort to verify, which keeps complete explanations of large models out of reach.