License YOLO
Back to Ultralytics Glossary

LiveCodeBench

LiveCodeBench tests LLMs on contest problems released after their training cutoff, scoring code generation, self-repair, execution, and test prediction.

LiveCodeBench is a continuously updated benchmark for the coding ability of large language models (LLMs), built from competitive programming problems on LeetCode, AtCoder, and Codeforces. Every problem carries the date its contest was published, so a model can be scored only on problems released after its training cutoff, problems it cannot have seen during training. Naman Jain and colleagues at UC Berkeley, MIT, and Cornell introduced it in March 2024 in the paper LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, and its leaderboard lets evaluators choose the date window they compare on.

Why Release Dates Matter#

Older code benchmarks such as HumanEval, OpenAI's 164 hand-written Python problems from 2021, have circulated online for years, and their solutions sit in the web text that training data is scraped from. A model can then score well by recalling answers, a form of data leakage known as benchmark contamination. The LiveCodeBench authors measured it directly: DeepSeek's coding models and GPT-4o performed far worse on LeetCode problems published after their release or stated training cutoff than on older ones. The paper also found a cluster of fine-tuned open models that scored well on HumanEval+ but poorly on new problems, such as DeepSeek-Coder-Instruct 1.3B, which solved 59.8% of HumanEval+ but only 26.3% of LiveCodeBench's easy problems, a sign of overfitting to the older benchmark.

The collection grows with each dataset release on Hugging Face. The first release, release_v1, held 400 problems published from May 2023 to March 2024, and release_v6 held 1,055 problems published through April 2025, so a reproducible score names its release and date window.

The Four Scenarios#

LiveCodeBench tests more than writing code, because a useful coding model also has to read, run, and debug it:

  • Code generation: Write a program from a problem statement and example inputs and outputs. The evaluator runs it against hidden tests, and a solution counts only if it passes every one.
  • Self-repair: Fix an incorrect solution after seeing the error message or the failing test, the same feedback loop that agentic coding tools run on their own.
  • Code execution: Predict what a given program prints for a given input, which tests whether the model can trace code without running it.
  • Test output prediction: Predict the correct output for an input from the problem statement alone, the skill used when writing tests before an implementation exists.

Code execution asks what an existing program does, and test output prediction asks what a correct program should do. The paper found that the relative order of models stays mostly consistent across the four scenarios, while the gaps between models vary from one scenario to another.

Reading LiveCodeBench Scores#

The headline metric is pass@1, the probability that a single generated solution passes all hidden tests, averaged over problems. The paper estimates it from 10 samples per problem with nucleus sampling at a temperature of 0.2, part of the pass@k family of metrics described under agent evaluation. Scores are also broken down by easy, medium, and hard problems. Prompt engineering, output length limits, and the date window all shift results, so leaderboard numbers are compared only under matching settings.

LiveCodeBench measures self-contained algorithmic problems. SWE-bench grades repository-level bug fixes, and LiveBench, despite the similar name, is a separate general benchmark that also refreshes its questions to limit contamination. Humanity's Last Exam and GPQA handle the same leakage risk with private held-out questions and canary strings instead of release dates.

A training cutoff also limits what a coding model knows about library versions released after it. Ultralytics Agent Skills close that gap for AI agents writing code for YOLO26: they ship catalogs of weight names, training arguments, and export formats grounded against a pinned ultralytics release, and tell the agent to trust the installed package when the two differ.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning