Needle In A Haystack (NIAH)

Needle in a haystack (NIAH) is an evaluation that hides a single fact, the needle, at a chosen depth inside a long stretch of unrelated text, the haystack, and asks a large language model (LLM) to retrieve it. Repeating the test across many context lengths and needle positions produces a heatmap that shows where in its context window a model stops finding information. Greg Kamradt introduced the test in November 2023 with an open-source NIAH repository: he hid the sentence "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day" inside a collection of Paul Graham's essays and asked each model for the best thing to do in San Francisco. The first runs tested OpenAI's GPT-4 Turbo, with a 128K-token window, and Anthropic's Claude 2.1, with 200K.

How the Test Works#

The haystack is trimmed to a target length in tokens, the needle is inserted at a fraction of that length, from 0% at the start to 100% at the end, and the model's answer is scored against the needle. A full run sweeps both axes:

The needle has nothing to do with the essays around it, so the test isolates one ability of the attention mechanism: locating and copying a specific passage anywhere in a long prompt.

Headline Results#

Kamradt's first heatmaps showed GPT-4 Turbo missing the needle above about 73K tokens, mostly when it sat between about 7% and 50% depth, close to the mid-prompt weakness that the Lost in the Middle study measured the same year. Claude 2.1 scored 27% on the original prompt, and Anthropic raised that to 98% with a prompt engineering change: starting Claude's answer with "Here is the most relevant sentence in the context:". NIAH became a standard chart in model launches. Google's Gemini 1.5 report claimed over 99% retrieval up to at least 10 million tokens and extended the test to audio and video haystacks, and Anthropic reported that Claude 3 Opus passed 99% recall and sometimes remarked that the needle looked artificially inserted into the text.

Variants and Limits#

Plain NIAH is close to a lexical search, since the question shares its key words with the needle, and frontier models now saturate it. Harder variants change what the needle demands:

• Passkey retrieval: A random number hidden in repeated filler text, used in 2023 to test Landmark Attention before NIAH existed.
• Multi-needle and aggregation: NVIDIA's RULER benchmark hides several keys and values, adds variable tracking and common-word aggregation, and found that only half of 17 models claiming 32K tokens or more performed satisfactorily at 32K.
• No literal match: NoLiMa words each question so it barely overlaps with the needle. At 32K tokens, 11 of 13 models fell below half of their short-context scores.

The accuracy loss these harder tests expose as inputs grow is called context rot. NIAH also differs from retrieval-augmented generation (RAG), where an external vector search picks passages before the model reads them, and from anomaly detection, which looks for unknown deviations instead of a known target.

Needles in Images#

Computer vision uses the same phrase for small object detection: a person in aerial imagery, a ship in a satellite scene, or a tumor cell in a whole-slide pathology scan for medical image analysis. The obstacle is resolution: Ultralytics YOLO26 resizes each image to 640 pixels on its longest side by default, so a 4,000-pixel drone frame shrinks more than sixfold and a 10-pixel pedestrian ends up under 2 pixels wide. Sliced inference with SAHI runs YOLO26 on overlapping slices, such as 256 × 256 tiles, at native resolution and merges the bounding boxes. In the SAHI paper's tests on the VisDrone and xView aerial datasets, slicing raised object detection AP by 5.1% to 6.8% for three different detectors.