Superalignment
Superalignment is the problem of aligning AI systems smarter than their human supervisors, studied through weak-to-strong generalization and oversight.
Superalignment is the problem of aligning AI systems that are far more capable than the humans supervising them, and the name of the research program OpenAI launched in 2023 to work on it. Current alignment methods, such as reinforcement learning from human feedback (RLHF), rely on people judging which model outputs are better. Once a model writes code, proofs, or plans that its supervisors cannot reliably check, those judgments stop being a trustworthy training signal, and a model could learn to produce answers that look right to its evaluators instead of answers that are right. Superalignment research looks for ways to supervise, test, and correct such systems anyway.
Origin of the Term#
OpenAI introduced superalignment on July 5, 2023, with a team co-led by chief scientist Ilya Sutskever and head of alignment Jan Leike. The company committed 20% of the compute it had secured to date and set a goal of solving the core technical challenges of aligning superintelligence within four years. Its plan was to build a roughly human-level automated alignment researcher and then scale that research with more compute. OpenAI disbanded the team in May 2024 after both leaders left the company, and the term remains in use for the general problem.
Superalignment is a subset of AI alignment. Alignment covers any system pursuing its designers' intent, through methods such as RLHF, constitutional AI, and direct preference optimization that assume a human, or rules a human wrote, can judge the outputs. Superalignment covers the case where that assumption fails, which arises for systems beyond human level, such as future artificial general intelligence (AGI) and superintelligence. It sits within AI safety alongside work on misuse and accidents.
Weak-to-Strong Generalization#
Because superhuman models do not exist yet, the team's first paper, Weak-to-Strong Generalization by Collin Burns and colleagues in December 2023, studied an analogy: a small model plays the human supervisor and a large foundation model plays the superhuman student. When GPT-4 was fine-tuned only on labels produced by a GPT-2-level supervisor, it consistently outperformed that supervisor while falling well short of its own full capability. Adding an auxiliary confidence loss, which rewards the student for confident predictions even where they contradict the weak labels, recovered close to GPT-3.5-level performance on natural language processing (NLP) tasks.
The paper measures success as performance gap recovered (PGR): the fraction of the accuracy gap between the weak supervisor and a strong model trained on true labels that the weakly supervised student closes. The same experiment runs at small scale with scikit-learn, using a shallow decision tree as the weak supervisor and a support vector machine (SVM) as the strong student:
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
x, y = load_digits(return_X_y=True)
x_weak, x_rest, y_weak, y_rest = train_test_split(x, y, train_size=0.3, random_state=0)
x_strong, x_test, y_strong, y_test = train_test_split(x_rest, y_rest, train_size=0.6, random_state=0)
# Weak supervisor: a shallow decision tree trained on ground-truth labels
weak = DecisionTreeClassifier(max_depth=5, random_state=0).fit(x_weak, y_weak)
weak_labels = weak.predict(x_strong)
# Strong student learns only from the weak labels; the ceiling learns from the true labels
student = SVC().fit(x_strong, weak_labels)
ceiling = SVC().fit(x_strong, y_strong)
weak_acc, student_acc, ceiling_acc = (m.score(x_test, y_test) for m in (weak, student, ceiling))
pgr = (student_acc - weak_acc) / (ceiling_acc - weak_acc)
print(f"weak {weak_acc:.3f}, weak-to-strong {student_acc:.3f}, ceiling {ceiling_acc:.3f}, PGR {pgr:.2f}")The tree classifies 69.0% of test digits correctly, and the SVM trained only on its labels reaches 81.9%, recovering 45% of the gap to the 97.4% of an SVM trained on true labels. The student beats its supervisor because the tree's errors follow axis-aligned splits that the regularized SVM fits poorly, so many of them do not carry over into its predictions. Ultralytics Platform poses the same question at small scale: Batch Annotation labels a whole dataset with one model, and a YOLO26 model later trained on those labels can outperform the labeling model only by not copying its mistakes.
Other Approaches#
Weak-to-strong training is one of several lines of work. Scalable oversight gives human supervisors AI help: in AI debate, proposed by Geoffrey Irving, Paul Christiano, and Dario Amodei in 2018, two models argue for opposing answers before a human judge, on the theory that it is harder to lie than to refute a lie. Sandwiching experiments test oversight methods on tasks where non-experts assisted by a model must match expert judgments, which stand in for ground truth.
Other work tries to check what a model computes instead of only what it outputs. Mechanistic interpretability and sparse autoencoders look for internal features tied to deception, and researchers deliberately build misaligned models, such as backdoored sleeper agents, to test whether detection methods catch them. Behaviors that a superhuman model could exploit at scale have already been observed in current large language models (LLMs), including alignment faking during training, sandbagging on evaluations, and reward hacking.
Burns and colleagues name imitation as a major gap in their analogy. GPT-4 was never pretrained to imitate GPT-2, and larger students had more difficulty imitating weak supervisors, while a superhuman model pretrained on human data is expected to imitate human supervisors very well, reproducing their mistakes and outputting only human-level answers.










