License YOLO
Back to Ultralytics Glossary

GPQA

GPQA is a graduate-level science benchmark that PhD experts answer 65% correctly and web-searching non-experts 34%. Its Diamond set and how to read scores.

GPQA, short for Graduate-Level Google-Proof Q&A, is a benchmark of 448 multiple-choice questions in biology, chemistry, and physics, written by domain experts so that specialists can answer them but skilled non-experts cannot, even with unrestricted web access. David Rein and colleagues at New York University, Cohere, and Anthropic released it in November 2023 in the paper GPQA: A Graduate-Level Google-Proof Q&A Benchmark to measure scientific reasoning in large language models (LLMs). Its harder 198-question Diamond subset has become one of the most widely reported scores in model announcements, from OpenAI's GPT-5 to Anthropic's Claude 4 models.

The benchmark was also built for research on scalable oversight, a branch of AI safety and superalignment: if AI systems eventually answer questions their human supervisors cannot check, people need ways to verify those answers. GPQA supplies questions where that gap between expert and non-expert already exists.

How the Questions Were Written and Validated#

The authors hired 61 contractors through Upwork, each holding or pursuing a PhD. A writer drafted a question with one correct answer and three plausible distractors. A second expert in the same field answered it and gave feedback, the writer revised it, and a third expert answered the revised version. Three non-expert validators, PhD-level specialists in other fields, then tried each question with full internet access but no AI assistants, spending 37 minutes on average. Writers earned bonuses when the experts answered correctly and the non-experts did not, which rewarded questions that are objectively answerable but hard to look up.

The organic chemistry sample published with the paper has methylcyclopentadiene react with methyl isoamyl ketone to form fulvene derivatives, which then react with ethyl acrylate, and asks how many chemically distinct isomers make up the final product. Answering requires working through each reaction step, the kind of multistep problem that chain-of-thought prompting and reasoning models target.

That process produced the benchmark's headline baselines. Experts answered 65% of questions in their own field correctly, rising to 74% when mistakes they recognized in hindsight are excluded. Non-experts reached only 34%, nine points above the 25% expected from random guessing, which is what "Google-proof" means in practice. At release, the strongest GPT-4 baseline, using few-shot chain-of-thought prompting, scored about 39%.

Extended, Main, and Diamond Sets#

GPQA ships in three nested subsets, each filtered more strictly than the last:

  • Extended (546 questions): Every released question, before filtering. The authors held back 18 more as an unreleased set.
  • Main (448 questions): At least one of the two expert validators agreed with the answer, and at most two of the three non-experts got it right.
  • Diamond (198 questions): Both experts agreed with the answer, and at most one non-expert got it right. An expert who answered wrong still counts as agreeing if they afterward clearly identified their own mistake.

Diamond is the standard choice for comparing models because it keeps the questions with the clearest answers and the widest gap between experts and non-experts.

Reading GPQA Scores#

In September 2024, OpenAI reported that its o1 reasoning model exceeded the accuracy of PhD experts on Diamond, and by 2026 the best scores reported for frontier models were above 90%. At that level small gaps carry little information. Each Diamond question is worth about half a percentage point, and at 90% accuracy the standard error over 198 questions is about 2.1 points, so a 95% confidence interval spans roughly plus or minus 4 points and two models a few points apart are statistically indistinguishable.

Protocol details also move scores. Zero-shot or few-shot prompting, whether chain-of-thought is allowed, how many samples are averaged, whether answer options are shuffled, and whether the model can use tools all change results, so fair comparisons hold them fixed, as Humanity's Last Exam does by reporting tool settings separately. Contamination is a further risk for any public benchmark. The GPQA data carries a canary string so training pipelines can filter it out, and the authors ask users not to post questions online in plain text, because once examples leak into web text a high score can reflect memorization, a form of data leakage that LiveCodeBench avoids by date-stamping its problems.

Vision benchmarks follow the same reporting discipline. The Ultralytics object detection docs list YOLO26 mean average precision (mAP) on the 5,000-image COCO val2017 split next to the exact validation command that reproduces each number, with nms=False added to reproduce the end-to-end column. The Ultralytics VLM Leaderboard evaluates vision language models with the same validation code and documents how it samples images and scores each box.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning