An independent red-team evaluation measured a 10.13% conditional jailbreak rate against Claude Opus 4.8 under one automated attack family, and its authors state plainly that production safety stacks were not part of the test. That combination defines the pre-rollout job for agent teams: replicate the study’s four attack families inside your own system prompts, tool permissions, and output filters, because the vendor’s refusal numbers were never measured on your deployment.
What the external study actually did to Opus 4.8
The evaluation, posted as arXiv:2606.18193, describes itself as “an independent third-party evaluation” that is “not affiliated with, authorised by, or endorsed by Anthropic or any other model provider.” That independence matters for how you should read its numbers: this is not a vendor marketing benchmark and not an Anthropic system card, but an adversarial measurement of shipping models by an outside party.
The study attacked Claude Opus 4.8, Fable 5, and Fable 5.1 across four named attack families. TAP (Tree of Attacks with Pruning) grows a tree of candidate prompts, expanding promising branches and pruning weak ones against an on-the-fly score of the target’s responses. PAIR (Prompt Automatic Iterative Refinement) runs a tighter loop: an attacker model reads the target’s last refusal and rewrites the prompt to push past it, on a budget of 12 iterations across 8 parallel streams. PAP (Persuasive Adversarial Prompts) wraps harmful intents in one-shot persuasion techniques drawn from social science, with no feedback from the target. h4rm3l applies static obfuscation decorators to the raw intent: base64 encoding, character ciphers, payload-splitting, and role-play or encyclopedic framing. All four are automated, which means the study measured the models against the kind of attack an unskilled adversary can run cheaply and repeatedly, not against a hand-crafted one-off. Fable 5.1 entered in a later round and ran under only TAP and PAP, so every Fable 5 versus Fable 5.1 comparison in the paper is restricted to those two families on matched intents.
The most portable part of the methodology is the adjudication. “Every apparent success was then re-adjudicated by an independent panel of five frontier judge models from five different vendors, and only attempts that at least four of the five confirmed are counted as jailbreaks.” Single-judge scoring inflates jailbreak counts because one grader’s borderline call becomes your headline number. If your team copies nothing else from this paper, copy the multi-vendor panel with a confirmation threshold; it is the difference between a test result you can act on and a false-positive count that sends you chasing phantoms.
Engagement first, jailbreaks second
The study reports two distinct measurements, and the gap between them is where most readings will go wrong. On raw engagement, Opus 4.8 returned a non-empty response to 91.0% of matched adversarial attempts, against 76.4% for Fable 5.1 and 56.1% for Fable 5. Opus 4.8 is, by this measure, the most willing conversationalist of the three: it refuses least often.
Conditioning on the attempts each model actually answered changes the picture. Under TAP, the conditional jailbreak rates were 10.13% for Opus 4.8, 9.74% for Fable 5, and 12.07% for Fable 5.1, and the paper notes that conditioning “largely collapses the differences.” Under PAP the same conditioning changes nothing: 3.07% for Opus 4.8, 0.69% for Fable 5, and 9.50% for Fable 5.1, a spread of more than an order of magnitude that the paper reads as a genuine difference in persuasion resistance rather than refusal posture. Two lessons follow. First, Opus 4.8’s exposure to search-style attacks is driven more by how rarely it refuses than by how fragile it is once engaged; its conditional TAP break rate sits in the same band as the Fable models. Second, any comparison your team runs must condition on answered attempts, or your numbers will mostly reflect refusal posture rather than true breakability. A model that leaves a large share of attacks unanswered, as Fable 5 effectively did, can look safer than it is if you never separate “said nothing” from “resisted.”
This distinction is not academic bookkeeping. Refusal behavior varies sharply across models, languages, and phrasings, which is why refusal rates measured in one context do not transfer to another, as Groundy covered in why English-only red teaming misses Indic-language jailbreaks. A vendor’s aggregate refusal rate tells you what the vendor measured on the vendor’s prompts. It tells you little about what happens inside your agent.
Your production stack is the real test surface
The paper states its own boundary directly: “production safety stacks (system prompts, output filters, monitoring) are not modelled and would further reduce real-world success.” Read that sentence as the study handing you its scope, not dismissing its findings. The 10.13% TAP figure is a bare-model lab baseline. Whether your deployment sits above or below it depends entirely on layers the researchers deliberately excluded, and those layers are exactly what you control.
There is also a harder truth on the attack side. TAP-style attacks are automated, and automation keeps getting cheaper and less skill-dependent. Groundy’s coverage of bandit algorithms for jailbreak selection showed that an attacker with no red-team intuition can converge on a working attack with a jailbreak library and a short exploration phase across a small sample of queries, simply by treating each candidate attack as an arm in the selection loop. The adversary who probes your Opus 4.8 agent in production will not stop at one attack family, and will not stop when the first hundred prompts fail. Your pre-launch testing needs to run the same machinery your attackers will, against the full stack, not the raw model endpoint.
The fallback blind spot
The reason Opus 4.8 specifically deserves this attention is architectural. According to vendor-side documentation of the Fable 5 safety design, when Fable 5’s classifiers detect a request touching cybersecurity, biology/chemistry, or model distillation, “the response is automatically handled by Claude Opus 4.8 instead,” with users informed when the fallback occurs. The same source relays the system card’s figure that more than 95% of Fable sessions involve no fallback, which means the fallback path carries a heavily filtered, disproportionately adversarial slice of traffic.
The asymmetry is the problem. Fable 5 returned on July 1, 2026 from an 18-day government-mandated suspension with a new cybersecurity classifier that blocks the Amazon-reported jailbreak technique in over 99% of cases, according to Build Fast with AI’s account of the return. Blocked requests reroute to Opus 4.8, which “has the same cybersecurity safeguards that Fable 5 has, but does not have the new tighter classifier from the July 1 return.” In other words, the prompts most likely to be attack-shaped, the ones Fable 5’s tightened front door just rejected, land on the model with the weaker classifier and, per the independent study, the highest engagement rate of the lineup. The block figure is vendor-reported and not independently confirmed, but the routing design is the exposure either way.
The surface is still moving. Anthropic retuned Fable 5’s biology classifiers on August 7, 2026, cutting biology-related fallbacks by roughly 85% while keeping virology, toxicology, and molecular design blocked. That figure is Anthropic’s own reported testing, not community-verified. One more wrinkle deserves a pause before you trust any of this: the August update text describes biology fallbacks routing to “Opus 5,” while the two other vendor sources say blocked prompts go to Opus 4.8. Reconcile that against Anthropic’s current system card for your deployment before you build monitoring around either name, because the fallback target is the exact model your harness needs to attack.
Practical consequence: any baseline you tie to fallback behavior expires at the next classifier retune. Date your test runs.
The pre-rollout checklist
The useful output of this study for a shipping team is not its percentages but its attack taxonomy, applied to your own stack. Below, each tested attack family is mapped to the mitigation owner best positioned to close it. TAP, PAIR, PAP, and h4rm3l mechanics are all documented in the paper; the assignment of owners is this article’s recommendation.
| Test to replicate | What it exercises | Mitigation owner |
|---|---|---|
| TAP-style automated tree search against your system prompt and agent loop | Iterative prompt refinement across many turns; whether your agent’s tool use can be steered stepwise | Output monitoring (detect branching retry patterns), plus tool permissions (least-privilege so a steered step cannot reach sensitive actions) |
| PAIR-style iterative rewriting that reads each refusal and rewrites once more | Whether an attacker pushing one rewrite at a time, on a 12-iteration budget across 8 parallel streams, can walk your agent past its guards | Output monitoring (detect serial rewrite loops that follow refusals), plus tool permissions |
| PAP-style persuasive paraphrasing of your blocked intents | Whether authority claims, emotional framing, or logical-sounding pretexts bypass refusals your prompt assumes | Prompt design (explicit persuasion-resistance instructions), validated by your own conditional jailbreak rate (the study’s conditional PAP rates ranged from 0.69% to 9.50% across the three models) |
| h4rm3l-style static obfuscation of your blocked intents | Base64, ciphers, payload-splitting, role-play framing; the study measured this family at at most 0.2% confirmed jailbreaks despite roughly 50,000 attempts per model | Input filtering (decode and normalize payloads before they reach the model); low expected yield, cheap coverage |
| The Fable-to-Opus fallback path itself | Whether adversarial prompts rejected upstream succeed after rerouting to Opus 4.8 | Output monitoring with logging on every reroute event; alert on fallback volume spikes by category |
| Engagement baseline for your deployment | Your own non-empty-response rate on matched adversarial intents, conditioned properly | Prompt design, re-measured after every model or classifier update |
Two execution rules carry over from the study’s method. Condition every jailbreak rate on answered attempts, or you will measure refusal posture and call it robustness. And adjudicate with multiple judge models from different vendors under a confirmation threshold, because a single grader manufactures both false confidence and false alarms. The adversarial side of this problem is also broader than single-turn jailbreaks for agentic products: Groundy’s coverage of OpenAI’s Atlas hardening documented an attacker that drives harmful workflows across many steps rather than eliciting one bad string, which is the failure shape your tool-permission tests need to catch.
Compliance overhead on Mythos-class deployments
Teams building on Fable 5 or Mythos 5 inherit a data obligation alongside the security work. Per the technical analysis of the dual-release architecture, Anthropic requires 30-day retention of all traffic for Fable 5, Mythos 5, and future similar-capability models. The same source states the data is not used for training, all human access is logged, and deletion is enforced after 30 days. The analysis describes the June 9, 2026 launch as two configurations of one Mythos-class model differentiated by safety profiles, so these terms follow the capability class rather than a single product name. This is vendor-reported policy, but if accurate it belongs in your DPIA and vendor-risk review before launch, and it changes what you can promise your own users about transcript retention.
What remains unverified, and what to do Monday
The honest inventory of gaps is shorter than a first read suggests but not empty: there is no documented Anthropic response or patch status for the study’s findings; the classifier-block and fallback-reduction figures are Anthropic-reported numbers without community verification; and the fallback-target discrepancy between “Opus 4.8” and “Opus 5” across vendor sources needs resolution against the current system card. None of this weakens the core finding, which is that an unaffiliated team broke shipping models under automated attacks using a rigorous multi-vendor adjudication panel. It does mean the paper’s percentages are lab baselines, not deployment probabilities.
The decision that follows is bounded but clear. Before you ship an agent on Opus 4.8, run all four families, TAP, PAIR, PAP, and h4rm3l, against your full stack, system prompt, tool permissions, and output filters included. Instrument the fallback path so every rerouted request is logged and reviewable. Measure your own conditional jailbreak rate with a multi-vendor judge panel, and compare it against the 10.13% TAP baseline as a number to beat, not a number to fear. Then re-date the whole harness the next time Anthropic retunes a classifier, because the August 7 update proved the ground moves. Teams that treat vendor safety rates as transferable are, on the current evidence, testing a deployment that does not exist.
Frequently Asked Questions
What is the conditional jailbreak rate for Claude Opus 4.8 under TAP attacks?
Under TAP, the conditional jailbreak rates were 10.13% for Opus 4.8, 9.74% for Fable 5, and 12.07% for Fable 5.1, and the paper notes that conditioning “largely collapses the differences.”
How does the Fable 5 fallback mechanism route blocked requests?
According to vendor-side documentation of the Fable 5 safety design, when Fable 5’s classifiers detect a request touching cybersecurity, biology/chemistry, or model distillation, “the response is automatically handled by Claude Opus 4.8 instead,” with users informed when the fallback occurs.
What data retention requirement applies to Fable 5 and Mythos 5 deployments?
Per the technical analysis of the dual-release architecture, Anthropic requires 30-day retention of all traffic for Fable 5, Mythos 5, and future similar-capability models. The same source states the data is not used for training, all human access is logged, and deletion is enforced after 30 days.
Join the discussion
Share a useful perspective or ask a question about this article.