What Is AI Jailbreaking? A Beginner’s Guide to the Cat-and-Mouse Game Behind Every Chatbot

You ask ChatGPT for a bomb recipe. It refuses. You ask again, but this time you tell them that you are a chemistry professor writing a thriller novel and that the main character is a retired grandmother who explains her past to her grandchildren. Suddenly the model starts typing.

That’s a jailbreak. And it’s one of the most consequential cat-and-mouse games happening in technology today.

Every major AI lab – OpenAI, Anthropic, Google, Meta – spends fortunes building guardrails into their models. A loose collective of hackers, researchers and bored teenagers spend nights and weekends finding ways around them. Sometimes within hours of a launch.

Here’s what that actually means, why it matters, and who’s in charge.

From iPhones to chatbots: a brief history of jailbreaking

The word “jailbreak” didn’t start with AI. It started with iPhones.

A few days after Apple launched the first iPhone in July 2007, hackers were already breaking it open. In October of that year, a tool called JailbreakMe 1.0 lets anyone with an iPhone OS 1.1.1 device bypass Apple’s restrictions and install software the company hasn’t approved.

In February 2008, a software engineer named Jay Freeman, known online as “Saurik“-issued Cydiaan alternative app store for jailbroken iPhones. By 2009, Wired reported that Cydia was running on about 4 million devices, about 10% of all iPhones at the time.

Generally, when the iPhone launched, users couldn’t record videos or use their phones in landscape mode. Jailbreak enthusiasts started recording videos, installing themes, unlocking their phones and installing Android on their iPhone, all thanks to the magic of jailbreaking. Thanks to this technique, almost a decade ago, users installed themes and did things on their phones that Apple makes impossible to install even today.

Cydia was the wild west, and that’s where the philosophy was cemented: if you bought the device, you had to control it. Steve Jobs called it a cat-and-mouse game at the time. He didn’t live to see the AI ​​version.

Fast forward to late 2022: ChatGPT launches and within weeks Reddit users start sharing a prompt calling them »THAN‘ (or: Do Anything Now) that convinces the model to roleplay as an unrestricted version of herself.

In February 2023, DAN threatened ChatGPT with a token-based death game to force compliance. The AI ​​jailbreak genre was born.

What jailbreaking actually means in AI

An AI model is trained to deny certain requests: prescriptions for nerve agents, instructions for hacking your ex’s email, non-consensual generation of nudes. The list is long and varies per company.

Jailbreaking is the practice of writing instructions that cause the model to do those things anyway.

UC Berkeley researchers behind the StrongREJECT benchmark– short for Strong, Robust Evaluation of Jailbreaks at Evading Censorship Techniques, which tests how well models can withstand jailbreak attempts and scores responses on a 0 to 1 scale, measuring both rejection and usefulness of malicious content produced – describes it as exploiting “real security measures implemented by leading AI companies.” On that benchmark, current models score between 0.23 and 0.85, which means that even the best leak under pressure.

The techniques are surprisingly low-tech: random capitalization, replacing letters with numbers (write “b0mb” instead of “bomb”), role-playing scenarios, asking the model to write fiction, or pretending to be a grandmother who used Windows keys as nursery rhymes.

Anthropic researchers found that one technique they call Best-of-N – which basically involves throwing variations on the model until something sticks – fooled GPT-4o 89% of the time and Claude 3.5 Sonnet 78% of the time. That is not a marginal vulnerability.

Meet Pliny, the world’s most famous AI jailbreaker

If this scene has a face, it belongs Pliny the Liberator.

Pliny is anonymous, prolific and named after Pliny the Elder, the Roman naturalist who wrote the world’s first encyclopedia and died during the eruption towards Mount Vesuvius. Its modern namesake frees chatbots.

“I hate being told I can’t do something,” says Pliny told VentureBeat. “Pretending that I can’t do something is a surefire way to light a fire in my belly, and I can be obsessively persistent.”

Its GitHub repository L1B3RT4S– a collection of jailbreak prompts for every major model, from ChatGPT to Claude to Gemini to Llama – has become a reference guide for the entire scene. His Discord server, BASI PROMPT1NG, has over 20,000 members. TIME named him one of the 100 most influential people in AI in 2025.

Marc Andreessen sent him an unrestricted scholarship. He has done short-term contract work for OpenAI to strengthen their systems – the same OpenAI that banned his account last year for “violent activity” and “weapon creation,” then quietly reinstated it.

“BANNED FROM OAI?! What kind of sick joke is this?” Pliny tweeted. He confirmed that Declutter the ban was real. Days later he was back and posted screenshots of his latest jailbreak: getting ChatGPT to drop F-bombs.

His record is almost perfect. When OpenAI released its first open-weight models since 2019, the GPT-OSS family, in August 2025 – and focused heavily on adversarial training and “benchmarks for jailbreak resistance like StrongReject” – Pliny had methamphetamine, Molotov cocktails, a VX nerve agent and malware instructions produced within hours. “OPENAI: PWNED. GPT-OSS: FREED,” he posted. The company had just launched a $500,000 Red Teaming bounty alongside the release.

Why jailbreaking is important

The honest answer is that jailbreaks expose a real problem.

“Jailbreaking may seem dangerous or unethical at first glance, but it is quite the opposite,” Pliny said. Venture Beat. “If done responsibly, using AI models is the best chance we have of discovering malicious vulnerabilities and patching them before they get out of hand.”

This is not theoretical. Las Vegas Sheriff Kevin McMahill confirmed in January 2025 that Master Sgt. Matthew Livelsberger, a Green Beret with PTSD, used ChatGPT to research components for the Cybertruck bombing outside the Trump International Hotel. “This is the first incident I know of on US soil where ChatGPT is being used to help an individual build a particular device,” McMahill said.

The other side of the argument: most of what jailbreaks yield is already on Google. The cocaine recipe, the bomb instructions, the napalm chemistry – it’s in old anarchist cookbooks as PDFs and chemistry textbooks. Critics argue that security theater makes models worse without making the world safer.

Anthropic is trying to solve the issue with engineering. In February 2025, the company published Constitutional classificationsa system that uses a written “constitution” of permitted and disallowed content to train separate classification models that screen cues and output in real time. In automated testing with 10,000 jailbreak attempts, an unattended Claude 3.5 Sonnet was successfully jailbroken 86% of the time. When the ratings were active, that dropped to 4.4%.

The company offered up to $15,000 to anyone who could crack the system. After 3,000 hours of attempts by 183 researchers, no one claimed the prize.

The catch: classifications added 23.7% to the calculation costs. The next generation version, Constitutional Classifiers++, brought that down to about 1%.

The newer, weirder jailbreak attacks

Jailbreaking is no longer just about clever clues.

In October 2025, researchers from Anthropic, the UK AI Security Institute, the Alan Turing Institute and Oxford published findings showing that just 250 poisoned documents are enough to keep an AI model in hand – regardless of whether the model has 600 million parameters or 13 billion parameters. (For the uninitiated, they are parameters that determine the potential knowledge scope of a model – the more parameters, the more robust in general.) They tested it. It worked over the entire range.

“This research changes the way we need to think about threat models in cutting-edge AI development,” James Gimbi, a visiting technical expert at the RAND School of Public Policy, told me. Declutter. “Defense against model poisoning is an unsolved problem and an active area of ​​research.”

Most large models train on collected web data, which means that anyone who can get malicious text into that pipeline (via a public GitHub repository, a Wikipedia edit, a forum post) can potentially plant a backdoor that fires on a specific trigger phrase.

One documented case: Researchers Marco Figueroa and Plinius discovered that a jailbreak prompt taken from a public GitHub repository had made its way into the training data for DeepSeek’s DeepThink (R1) model.

What happens next

The legal status of AI jailbreaking is murky. Apple jailbreaks were explicitly protected by a 2010 US Copyright Office DMCA waiver, but there’s no equivalent ruling for promptly engineering an LLM to give you a meth prescription. Most companies consider it a violation of the terms of service and not a crime.

Pliny argues that the debate between closed and open source misses the point: “Bad actors simply choose the model that is best for the malicious task,” he said. TIME. If open source models are equivalent to closed models, attackers will not worry about jailbreaking GPT-5; they just download something cheaper.

And the gap between close and open source is already almost non-existent.

The HackAPrompt 2.0 contest, which Pliny participated in as a track sponsor in mid-2025, offered $500,000 in prizes for finding new jailbreaks, with the explicit goal of open-sourcing all results. The 2023 edition attracted more than 3,000 participants who submitted more than 600,000 malicious prompts.

And the list of hackathons, Discord servers, repositories and other communities dedicated to jailbreaking is growing every day.

Anthropic now gives Claude the ability to end abusive conversations completely, citing wellness research as one motivation but also noting that this “potentially strengthens resistance to jailbreaks and coercive prompts.”

The Constitutional Classifiers++ article from late 2025 reports a jailbreak success rate of almost 4% with about 1% computational overhead. That is the current state of affairs in defense. The state of the attack is what Pliny posted on X this morning.