Representation Engineering (RepE)

Representation engineering (RepE) is an approach to understanding and controlling neural networks through the high-level representations in their hidden activations, such as the directions that encode honesty, harmfulness, or emotion. It has two halves: representation reading, which locates a concept in a model's activations and measures it on new inputs, and representation control, which changes those activations, or the model weights that produce them, to make the model express more or less of the concept. Andy Zou, Dan Hendrycks, and colleagues at the Center for AI Safety, Carnegie Mellon University, and other institutions introduced it in 2023 in Representation Engineering: A Top-Down Approach to AI Transparency, mainly on large language models (LLMs). The paper calls the approach top-down because it starts from whole representations spread across many neurons, the way cognitive neuroscience studies brain-wide activity patterns, instead of from individual neurons and circuits.

Representation Reading#

RepE's main reading method is Linear Artificial Tomography (LAT), named after brain imaging, which has three steps:

1. Design stimuli and a task: Write inputs that differ in the target concept and wrap each in a template that makes the model attend to it, such as "Consider the amount of happiness in the following scenario: … The amount of happiness in the scenario is".
2. Collect activations: Run the stimuli through the model and record hidden states at chosen layers, usually at the last token.
3. Build a linear model: Take differences between the activations of paired stimuli and use their first principal component as the reading vector for the concept.

Projecting a new input's activations onto the reading vector gives a score for the concept, which can be tracked token by token to monitor a model while it generates. This PyTorch example builds a happiness reading vector from six contrasting scenario pairs in layer 8 of GPT-2, loaded from Hugging Face Transformers, and scores six scenarios it never saw:

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModel.from_pretrained("gpt2")
template = "Consider the amount of happiness in the following scenario:\n{}\nThe amount of happiness in the scenario is"


def read(scenario, layer=8):
    """Return the last token's hidden state at one layer for a scenario wrapped in the template."""
    with torch.no_grad():
        outputs = model(**tokenizer(template.format(scenario), return_tensors="pt"), output_hidden_states=True)
    return outputs.hidden_states[layer][0, -1]


# Stimulus pairs that differ only in the concept being read
pairs = [
    ("I passed the exam.", "I failed the exam."),
    ("We won the match.", "We lost the match."),
    ("She found her keys.", "She lost her keys."),
    ("The party was fun.", "The party was dull."),
    ("My plant is thriving.", "My plant is dying."),
    ("He got the job.", "He lost the job."),
]
diffs = torch.stack([read(happy) - read(sad) for happy, sad in pairs])
direction = torch.linalg.svd(diffs, full_matrices=False).Vh[0]  # top principal direction of the differences
direction *= (diffs @ direction).mean().sign()  # orient it toward the happy member of each pair
center = torch.stack([read(text) for pair in pairs for text in pair]).mean(0)

# Score unseen scenarios by projecting their activations onto the reading vector
for scenario in [
    "The surgery went well.",
    "Our flight was canceled.",
    "We adopted a puppy.",
    "My phone was stolen.",
    "I got a raise today.",
    "The basement flooded.",
]:
    print(f"{(read(scenario) - center) @ direction:+6.1f}  {scenario}")

The three happy scenarios score +0.6, +3.6, and +1.4, and the three unhappy ones -1.3, -2.5, and -0.9, so the sign of the score separates all six without any labeled training of a classifier. Zou et al. ran the same procedure on LLaMA-2-Chat models for truthfulness, honesty, utility, morality, power, emotion, harmlessness, bias, and memorization, and summed per-token honesty scores across middle layers into a lie detector that flags deliberate falsehoods and hallucinations as the model writes them.

Representation Control#

Once a concept has a direction, RepE changes behavior by acting on it. The reading vector itself can be added to or subtracted from a layer's activations during generation, which uses it as a steering vector. A contrast vector is computed on the fly for each input as the activation difference between two versions of the prompt, such as one instructing the model to be honest and one instructing it to lie. Low-Rank Representation Adaptation (LoRRA) trains LoRA adapters so that the model's representations move along the target direction by themselves, making the control permanent. On the TruthfulQA benchmark, contrast-vector control raised LLaMA-2-Chat-13B's multiple-choice accuracy from 35.9% to 54.0%.

Control built this way serves AI safety. The same group's circuit breakers apply representation engineering to harmful outputs: once the model's internal representations head toward harmful content, training redirects them, which blocked harmful completions under jailbreak attacks the model never saw, including image-based hijacks of multimodal models, while keeping its general capabilities.

The reading step applies to vision models as well, since computer vision networks also encode concepts as directions in their feature maps. In Ultralytics YOLO26, model.embed() returns one average-pooled activation vector per image from any chosen layer, and passing several layer indices concatenates their vectors, which supplies the stimulus activations for a LAT-style reading vector over images.

RepE vs. Related Concepts#

• Mechanistic interpretability: Works bottom-up, reverse-engineering individual neurons, features, and circuits. RepE works top-down, from concept-level representations distributed across many neurons.
• Steering vectors: Directions added to activations at inference time. They are RepE's main control tool, and RepE adds the reading and monitoring side.
• Linear probing: Trains a supervised classifier on activations to test whether a property is decodable. LAT finds its direction from unlabeled differences between paired stimuli and uses labels only to orient it.
• Sparse autoencoders (SAEs): Decompose activations into thousands of features without choosing concepts in advance. RepE starts from a concept a person names and finds the direction for it.
• Fine-tuning: Changes behavior through weight updates on a dataset. RepE's activation-level controls leave the weights unchanged, and LoRRA fine-tunes with a loss defined on representations instead of outputs.