Humanity's Last Exam (HLE)

Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across more than 100 academic subjects, built to measure AI systems at the frontier of human knowledge. The Center for AI Safety and Scale AI released it in January 2025, finalized the public question set in April 2025, and published it in Nature in January 2026. The name reflects its goal: a closed-ended academic exam hard enough to stay informative after earlier tests stopped separating the best large language models (LLMs).

That problem is benchmark saturation. By 2024, frontier models scored above 90% on MMLU, a widely used knowledge test, so further gains were hard to read. HLE was built so that the same models would score in the single digits, and at launch they did.

How the Questions Were Collected#

Nearly 1,000 subject experts, mostly professors, researchers, and graduate-degree holders from more than 500 institutions in 50 countries, wrote the questions. Each candidate was first tested against several frontier LLMs and kept only if the models answered it wrong, or, for multiple-choice items, did worse than random guessing. More than 70,000 such attempts left about 13,000 questions for human review. In the first round, reviewers with graduate degrees gave each question one to three reviews so the writer could make it closed-ended and unambiguous; in the second, organizers and reviewers approved the best questions for the final set. The filter resembles the one behind GPQA, whose Diamond set keeps science questions only when most skilled non-experts with web access failed them. A $500,000 prize pool paid $5,000 for each of the top 50 questions and $500 for each of the next 500.

Mathematics makes up 41% of the set, followed by biology and medicine at 11%, computer science and AI at 10%, physics and the humanities at 9% each, chemistry at 7%, and engineering at 4%. One published sample asks how many paired tendons are supported by a sesamoid bone found only in hummingbirds, the kind of detail a specialist knows and a search engine does not surface quickly. About 24% of questions are multiple choice. The other 76% need an exact short answer, which a judge model, o3-mini on the official leaderboard, compares with the reference while accepting equivalent forms such as a fraction in place of a decimal. The private held-out set detects models that have overfit to the public questions, a risk that grows once benchmark text leaks into training data, a form of data leakage that LiveCodeBench counters by date-stamping its problems.

Roughly 14% of questions are multimodal, pairing text with a diagram, chart, or photographed inscription, which makes HLE an expert-level test of visual question answering. These questions grade the reasoning of vision-language models (VLMs), the models that the LLM block in Ultralytics Platform Agents calls after a YOLO26 detector has screened an image, sending the image and the detector's results to whichever vision-language model the user selects.

Accuracy and Calibration Error#

HLE reports two numbers for every model:

• Accuracy: The share of questions answered correctly. In the original paper, GPT-4o scored 2.7%, Claude 3.5 Sonnet 4.1%, Gemini 1.5 Pro 4.6%, and OpenAI's o1 reasoning model 8.0%, while o3-mini reached 13.4% on the text-only questions.
• Calibration error: Each model states a confidence from 0% to 100% with every answer, and the root-mean-square gap between stated confidence and actual accuracy measures how well it knows what it knows. Launch-era models had calibration errors between 73% and 89%, meaning they gave confident answers to questions they got wrong, a measurable form of hallucination.

Accuracy rose quickly with reasoning models while calibration lagged. On the official leaderboard, Gemini 3 Pro reached 38.3% accuracy with a 57.2% calibration error, and GPT-5 scored 25.3% with a 50.0% calibration error. Calibration error is a direct application of uncertainty quantification to LLMs, and the same gap between confidence and correctness drives detection methods such as semantic entropy.

Revisions: HLE-Rolling and HLE-Diamond#

An investigation by the AI research lab FutureHouse, reported in July 2025, estimated that nearly 30% of the text-only chemistry and biology answers conflicted with published literature, and other reviewers flagged ambiguous wording. The organizers partly replicated the findings and launched HLE-Rolling, a continuously updated fork, in October 2025. After a year of cleaning, they released HLE-Diamond on September 22, 2026: a refined 1,000-question subset split into 500 reasoning and 500 knowledge questions, designed to be answered closed-book. Tool access changes results sharply, so leaderboards and agent evaluations report tool settings with every score. At release, GPT-6 Astra led HLE-Diamond with 59.9% without tools and 82.9% with web search and code execution.