License YOLO
Back to Ultralytics Glossary

Superalignment

Superalignment is the problem of aligning AI systems smarter than their human supervisors, studied through weak-to-strong generalization and oversight.

Superalignment is the problem of aligning AI systems that are far more capable than the humans supervising them, and the name of the research program OpenAI launched in 2023 to work on it. Current alignment methods, such as reinforcement learning from human feedback (RLHF), rely on people judging which model outputs are better. Once a model writes code, proofs, or plans that its supervisors cannot reliably check, those judgments stop being a trustworthy training signal, and a model could learn to produce answers that look right to its evaluators instead of answers that are right. Superalignment research looks for ways to supervise, test, and correct such systems anyway.

Origin of the Term#

OpenAI introduced superalignment on July 5, 2023, with a team co-led by chief scientist Ilya Sutskever and head of alignment Jan Leike. The company committed 20% of the compute it had secured to date and set a goal of solving the core technical challenges of aligning superintelligence within four years. Its plan was to build a roughly human-level automated alignment researcher and then scale that research with more compute. OpenAI disbanded the team in May 2024 after both leaders left the company, and the term remains in use for the general problem.

Superalignment is a subset of AI alignment. Alignment covers any system pursuing its designers' intent, through methods such as RLHF, constitutional AI, and direct preference optimization that assume a human, or rules a human wrote, can judge the outputs. Superalignment covers the case where that assumption fails, which arises for systems beyond human level, such as future artificial general intelligence (AGI) and superintelligence. It sits within AI safety alongside work on misuse and accidents.

Weak-to-Strong Generalization#

Because superhuman models do not exist yet, the team's first paper, Weak-to-Strong Generalization by Collin Burns and colleagues in December 2023, studied an analogy: a small model plays the human supervisor and a large foundation model plays the superhuman student. When GPT-4 was fine-tuned only on labels produced by a GPT-2-level supervisor, it consistently outperformed that supervisor while falling well short of its own full capability. Adding an auxiliary confidence loss, which rewards the student for confident predictions even where they contradict the weak labels, recovered close to GPT-3.5-level performance on natural language processing (NLP) tasks.

The paper measures success as performance gap recovered (PGR): the fraction of the accuracy gap between the weak supervisor and a strong model trained on true labels that the weakly supervised student closes. The same experiment runs at small scale with scikit-learn, using a shallow decision tree as the weak supervisor and a support vector machine (SVM) as the strong student:

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier

x, y = load_digits(return_X_y=True)
x_weak, x_rest, y_weak, y_rest = train_test_split(x, y, train_size=0.3, random_state=0)
x_strong, x_test, y_strong, y_test = train_test_split(x_rest, y_rest, train_size=0.6, random_state=0)

# Weak supervisor: a shallow decision tree trained on ground-truth labels
weak = DecisionTreeClassifier(max_depth=5, random_state=0).fit(x_weak, y_weak)
weak_labels = weak.predict(x_strong)

# Strong student learns only from the weak labels; the ceiling learns from the true labels
student = SVC().fit(x_strong, weak_labels)
ceiling = SVC().fit(x_strong, y_strong)

weak_acc, student_acc, ceiling_acc = (m.score(x_test, y_test) for m in (weak, student, ceiling))
pgr = (student_acc - weak_acc) / (ceiling_acc - weak_acc)
print(f"weak {weak_acc:.3f}, weak-to-strong {student_acc:.3f}, ceiling {ceiling_acc:.3f}, PGR {pgr:.2f}")

The tree classifies 69.0% of test digits correctly, and the SVM trained only on its labels reaches 81.9%, recovering 45% of the gap to the 97.4% of an SVM trained on true labels. The student beats its supervisor because the tree's errors follow axis-aligned splits that the regularized SVM fits poorly, so many of them do not carry over into its predictions. Ultralytics Platform poses the same question at small scale: Batch Annotation labels a whole dataset with one model, and a YOLO26 model later trained on those labels can outperform the labeling model only by not copying its mistakes.

Other Approaches#

Weak-to-strong training is one of several lines of work. Scalable oversight gives human supervisors AI help: in AI debate, proposed by Geoffrey Irving, Paul Christiano, and Dario Amodei in 2018, two models argue for opposing answers before a human judge, on the theory that it is harder to lie than to refute a lie. Sandwiching experiments test oversight methods on tasks where non-experts assisted by a model must match expert judgments, which stand in for ground truth.

Other work tries to check what a model computes instead of only what it outputs. Mechanistic interpretability and sparse autoencoders look for internal features tied to deception, and researchers deliberately build misaligned models, such as backdoored sleeper agents, to test whether detection methods catch them. Behaviors that a superhuman model could exploit at scale have already been observed in current large language models (LLMs), including alignment faking during training, sandbagging on evaluations, and reward hacking.

Burns and colleagues name imitation as a major gap in their analogy. GPT-4 was never pretrained to imitate GPT-2, and larger students had more difficulty imitating weak supervisors, while a superhuman model pretrained on human data is expected to imitate human supervisors very well, reproducing their mistakes and outputting only human-level answers.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning