LiveCodeBench
LiveCodeBench tests LLMs on contest problems released after their training cutoff, scoring code generation, self-repair, execution, and test prediction.
LiveCodeBench is a continuously updated benchmark for the coding ability of large language models (LLMs), built from competitive programming problems on LeetCode, AtCoder, and Codeforces. Every problem carries the date its contest was published, so a model can be scored only on problems released after its training cutoff, problems it cannot have seen during training. Naman Jain and colleagues at UC Berkeley, MIT, and Cornell introduced it in March 2024 in the paper LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, and its leaderboard lets evaluators choose the date window they compare on.
Why Release Dates Matter#
Older code benchmarks such as HumanEval, OpenAI's 164 hand-written Python problems from 2021, have circulated online for years, and their solutions sit in the web text that training data is scraped from. A model can then score well by recalling answers, a form of data leakage known as benchmark contamination. The LiveCodeBench authors measured it directly: DeepSeek's coding models and GPT-4o performed far worse on LeetCode problems published after their release or stated training cutoff than on older ones. The paper also found a cluster of fine-tuned open models that scored well on HumanEval+ but poorly on new problems, such as DeepSeek-Coder-Instruct 1.3B, which solved 59.8% of HumanEval+ but only 26.3% of LiveCodeBench's easy problems, a sign of overfitting to the older benchmark.
The collection grows with each dataset release on Hugging Face. The first release, release_v1, held 400 problems published from May 2023 to March 2024, and release_v6 held 1,055 problems published through April 2025, so a reproducible score names its release and date window.
The Four Scenarios#
LiveCodeBench tests more than writing code, because a useful coding model also has to read, run, and debug it:
- Code generation: Write a program from a problem statement and example inputs and outputs. The evaluator runs it against hidden tests, and a solution counts only if it passes every one.
- Self-repair: Fix an incorrect solution after seeing the error message or the failing test, the same feedback loop that agentic coding tools run on their own.
- Code execution: Predict what a given program prints for a given input, which tests whether the model can trace code without running it.
- Test output prediction: Predict the correct output for an input from the problem statement alone, the skill used when writing tests before an implementation exists.
Code execution asks what an existing program does, and test output prediction asks what a correct program should do. The paper found that the relative order of models stays mostly consistent across the four scenarios, while the gaps between models vary from one scenario to another.
Reading LiveCodeBench Scores#
The headline metric is pass@1, the probability that a single generated solution passes all hidden tests, averaged over problems. The paper estimates it from 10 samples per problem with nucleus sampling at a temperature of 0.2, part of the pass@k family of metrics described under agent evaluation. Scores are also broken down by easy, medium, and hard problems. Prompt engineering, output length limits, and the date window all shift results, so leaderboard numbers are compared only under matching settings.
LiveCodeBench measures self-contained algorithmic problems. SWE-bench grades repository-level bug fixes, and LiveBench, despite the similar name, is a separate general benchmark that also refreshes its questions to limit contamination. Humanity's Last Exam and GPQA handle the same leakage risk with private held-out questions and canary strings instead of release dates.
A training cutoff also limits what a coding model knows about library versions released after it. Ultralytics Agent Skills close that gap for AI agents writing code for YOLO26: they ship catalogs of weight names, training arguments, and export formats grounded against a pinned ultralytics release, and tell the agent to trust the installed package when the two differ.










