Jagged Intelligence

Jagged intelligence describes the uneven capability profile of modern AI systems: the same model can perform at expert level on hard tasks and fail simple ones that most people find trivial. Andrej Karpathy coined the term in 2024 for large language models (LLMs) that could solve difficult math problems yet insisted that 9.11 is larger than 9.9. Miscounting the letter "r" in "strawberry" became another famous example. Plotted across tasks, such a model's competence forms a jagged line with sharp peaks and deep valleys, and the location of the valleys is hard to predict from the peaks.

Why Capabilities Are Uneven#

A model is strongest where its training data is dense and its internal representation fits the task, and both conditions vary sharply from one task to the next. Representation explains many of the famous text failures. Tokenization splits text into subword tokens before the model sees it, as OpenAI's tokenizer for GPT-4o shows:

import tiktoken

# o200k_base is the tokenizer OpenAI's GPT-4o uses
encoding = tiktoken.get_encoding("o200k_base")

for question in ["How many r letters are in strawberry?", "Which is bigger, 9.11 or 9.9?"]:
    tokens = [encoding.decode([t]) for t in encoding.encode(question)]
    print(tokens)

The word " strawberry" arrives as a single token, so the model never sees its three r's and has to recall the spelling from training text. The numbers arrive as "9", ".", "11" and "9", ".", "9", and in much of that text such pairs are version numbers, dates, or book chapters, where 9.11 does come after 9.9 and 11 is larger than 9. The model answers from those patterns instead of performing a numeric comparison.

Data coverage explains much of the rest. Skills that appear constantly in training data become reliable, while rare formats, unusual phrasings, and edge cases stay weak, however simple they look to a person. Vision models show the same pattern. An object detection model can be excellent on common, large objects in daylight and much weaker on small objects, rare classes, or unusual viewpoints, even when a person would find all of them equally easy.

Jaggedness differs from Moravec's paradox, Hans Moravec's 1988 observation that tasks hard for people, such as chess, are easy for computers while effortless human skills such as perception and movement are hard. Jagged capabilities do not follow human difficulty in either direction, so intuition about what is "easy" is a poor guide to where a model will fail.

The Jagged Frontier at Work#

The same unevenness shapes how people work with AI. In a 2023 field experiment with 758 Boston Consulting Group consultants, researchers from Harvard Business School and other institutions described a jagged technological frontier. On 18 tasks inside the frontier, consultants using GPT-4 completed 12.2% more tasks, worked 25.1% faster, and produced work rated more than 40% higher in quality. On a task just outside it, consultants with AI were 19 percentage points less likely to reach the correct answer than those working without it. Jagged intelligence names the model's uneven capability profile, and the jagged frontier names the resulting boundary between work AI helps with and work it quietly makes worse.

Jaggedness also complicates claims about artificial general intelligence (AGI). A score averaged over a broad benchmark dataset can rise steadily while specific valleys stay in place, so a headline average says little about whether a model is dependable on any one task.

Measuring a Jagged Profile#

Uneven capability becomes visible once evaluation is split into slices instead of reported as one number. For a detector, the first split is by class. Validation mode for Ultralytics YOLO26 reports mean average precision (mAP) overall and for every class in the validation labels, and sorting the per-class metrics puts the valleys first:

from ultralytics import YOLO

# Validate a pretrained detector on labeled images
model = YOLO("yolo26n.pt")
metrics = model.val(data="coco8.yaml")

# One row per class in the validation labels, weakest first
rows = sorted(metrics.summary(), key=lambda row: row["mAP50-95"])
print(f"Overall mAP50-95: {metrics.box.map:.3f}")
for row in rows[:3]:
    print(f"{row['Class']}: {row['mAP50-95']} mAP50-95 over {row['Instances']} instances")

On a full validation set such as the 5,000 images of COCO, the same breakdown points to the classes that need more data or a closer look at their labels. Classes are only the first slice. Lighting, camera angle, motion, and occlusion can each hide a valley of their own, and model testing on new images taken under those conditions brings them out:

After deployment, the profile keeps moving as inputs change, and tracking data drift shows when production images start falling into regions the validation set never covered.