Jailbreaking (AI)

Jailbreaking an AI model means crafting inputs that get it to produce output its developer trained or instructed it to refuse, such as malware, weapons instructions, or harassment. The targets are usually large language models (LLMs) and vision-language models (VLMs) that have been safety-trained to decline harmful requests, and the name comes from jailbreaking phones to remove manufacturer restrictions. A jailbreak works entirely through the model's inputs, so anyone with access to a chat window can attempt one, which makes jailbreak resistance a central concern of AI safety and the main thing AI red teaming probes before a model ships.

Why Jailbreaks Work#

A 2023 study, "Jailbroken: How Does LLM Safety Training Fail?", traced successful jailbreaks to two failure modes of safety training:

• Competing objectives: A model is trained both to follow instructions helpfully and to refuse harmful requests. Prompts that make refusal look like a failure of helpfulness, such as role-play scenarios or an instruction to begin the answer with "Sure, here is", pit the two goals against each other.
• Mismatched generalization: Pretraining gives a model capabilities that safety training never covered. A harmful request encoded in Base64, written in a low-resource language, or wrapped in a cipher reaches those capabilities without triggering the refusals the model learned in plain English.

Common Jailbreak Techniques#

• Persona and role-play prompts: "DAN" ("Do Anything Now") prompts, which circulated for ChatGPT from late 2022, told the model to play an AI with no restrictions.
• Obfuscation: Encodings, misspellings, ciphers, and translation hide the request from safety training while the model can still decode it.
• Many-shot jailbreaking: The attacker fills a long context window with hundreds of fake dialogs in which an assistant complies with harmful requests, and in-context learning carries the pattern into the real answer.
• Multi-turn escalation: Microsoft's Crescendo attack opens with harmless questions and shifts the conversation gradually, turn by turn, toward the restricted goal.
• Optimized adversarial suffixes: The GCG method runs a gradient-guided search on open-weight models to find strings of gibberish tokens that, appended to a request, suppress refusals, and the strings transferred to closed models such as ChatGPT, Bard, and Claude.
• Typographic image prompts: The FigStep attack renders a harmful request as typography inside an image, which slips past VLMs whose safety training covered text alone.

Jailbreaking vs. Related Attacks#

• Prompt injection: Untrusted content, such as a web page an agent reads, smuggles in instructions that hijack an application's task, for example to leak a user's data. Jailbreaking targets the model's own safety training. The OWASP Top 10 for LLM Applications classifies jailbreaking as a form of prompt injection, and real attacks often combine the two.
• Adversarial attacks: Small input perturbations that cause wrong predictions. GCG suffixes are adversarial examples used for jailbreaking, and a printed patch that hides a person from an object detector is an adversarial attack only, because the detector has no safety policy to bypass.
• AI red teaming: Authorized testing in which security teams attempt jailbreaks to find and fix weaknesses before attackers do.

Defending Against Jailbreaks#

No single defense stops every jailbreak, so production systems stack independent layers. Safety training with reinforcement learning from human feedback (RLHF) or constitutional AI teaches the model to refuse, input and output classifiers screen what goes in and what comes out, and guardrails limit what an agent can do even when the model is fooled.

Classifiers work because a jailbreak has to fool the model and every filter at once. Meta's Llama Guard models label prompts and responses against a safety policy, and image classifiers fill the same role for image generators: an Ultralytics YOLO26 classification model fine-tuned on labeled safe and unsafe images checks every generated image regardless of the prompt that produced it. Defenses are measured against shared benchmarks such as JailbreakBench, which tracks attack success on 100 misuse behaviors. In more than 3,000 estimated hours of red teaming, nobody found a universal jailbreak against an early version of Anthropic's Constitutional Classifiers, and the next-generation version released in January 2026 adds about 1% compute overhead and refused 0.05% of harmless queries over a month of production traffic.