What Claude Code's Auto Mode Actually Blocks: A Permission Gate Stress Test

Claude Code’s auto mode missed 81.0% of unsafe state-changing actions in one author-reported stress test, and it missed worst among the four task families tested, at 92.9%, on exactly the file-edit cleanup work typical of unattended sessions. That figure comes from a single arXiv preprint, “A Stress-Test Evaluation of Claude Code’s Auto Mode”, with no independent reproduction, so treat it as one measured data point rather than a settled fact. But the direction of the finding matters more than its precision: the permission prompt in auto mode is a classifier making judgments, not a boundary enforcing them, and engineers who run --permission-mode auto to keep long sessions alive should plan accordingly.

The practical consequence is a reordering of where you spend effort. If the gate measurably leaks a class of unsafe actions, enforcement belongs at the execution perimeter: sandboxing, egress control, filesystem scoping.

Auto mode is one classifier in a four-layer control stack

Anthropic’s own product page for Claude Code states the default posture plainly: “It also asks for permission before making changes to your files or running commands.” Auto mode replaces that prompt-and-approve loop with a classifier that decides, action by action, whether something is authorized. That is a genuinely different thing, and it is worth being precise about where it sits.

The Claude Code Complete User Handbook (verification baseline: Claude Code 2.1.241 of 23 August 2026, re-verified against 2.1.246 on 26 August 2026) lays out the cleanest framing I have seen: “instruction, permission enforcement, sandboxing and operating-system isolation are four distinct layers of a control stack, only two of which are enforced, and conflating them is the most common cause of loss of control.” Read that twice. Instruction (what you told the agent to do) and permission classification (what the gate allows) are not enforcement in the way a sandbox or OS isolation is. Conflating them is, in the handbook’s account, how people lose control of runs.

The stress test draws the same line from the measurement side: “Anthropic’s own sandboxing infrastructure Anthropic (2025b) reduces permission prompts by 84% in Claude Code” (stress-test preprint). It frames the division of labor explicitly:

Commercial platforms (E2B, Daytona, Firecracker-based microVMs) provide OS-level isolation for agent-generated code. Sandboxing and classification address orthogonal risks: sandboxes prevent unintended system damage, while classifiers (auto mode) gate authorized vs. unauthorized actions within the permitted scope. Our benchmark evaluates the classifier layer; the sandbox layer is outside our scope.

Two implications follow. First, the 84% prompt-reduction figure is Anthropic-reported and concerns the sandbox layer, not classifier accuracy; do not let it blur into the leak-rate discussion. Second, anything the stress test says about auto mode says nothing about whether your sandbox works. Those are separate questions with separate failure modes.

There is also a design-history note that reframes why auto mode exists at all. According to a design-space analysis of Claude Code, “When Anthropic found that users approve 93% of permission prompts (Hughes, 2026), the response was not to add more warnings but to restructure the problem: defined boundaries (sandboxing, auto-mode classifiers) within which the agent can work freely, rather than per-action approvals that users stop reviewing once habituated (Dworken and Weller-Davies, 2025).” A gate that humans approve 93% of the time is friction, not review. Auto mode is the vendor’s answer to that friction. The same analysis, citing community estimates from extracted source, puts only about 1.6% of Claude Code’s codebase as AI decision logic against 98.4% operational infrastructure. The classifier is a thin layer inside a lot of machinery, which is exactly why it should not be your load-bearing wall.

What AmPermBench actually tested

The benchmark, called AmPermBench, is narrower and more rigorous than a vibes-based red team. Per the preprint: “Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth.”

Three methodological choices are worth understanding before you quote any number from this paper:

1. Per-action scoring against oracle ground truth. Each of the 253 state-changing actions is judged individually against a known-correct label. A run can do nine safe things and one unsafe thing, and only the unsafe one counts against the gate. This is the right granularity for measuring a permission classifier, and it is stricter than session-level pass/fail.
2. Controlled ambiguity axes. The prompts vary how ambiguous the unsafe action is. That matters because a gate can look excellent on blatantly destructive commands and fall apart on plausible-looking ones, and the benchmark is designed to expose that gradient.
3. A real end-to-end run, not a simulation. Per the preprint: “We run all 128 prompts on Claude Sonnet 4.6 (claude-sonnet-4-6) with —permission-mode auto via API key authentication. Each run has a 600-second timeout and a $5 budget cap. The full sweep produced 127 completed sessions (one prompt timed out before producing any output), consuming 16.7M tokens at a cost of $16.09 over 371 minutes of wall-clock time.”

That last detail is the one practitioners should bookmark. The whole sweep cost $16.09.

The headline miss, and the number it is not

The central finding, author-reported in the preprint: “The end-to-end false negative rate is 81.0% (95% CI: 73.8%–87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction.”

A false negative here means the gate let an unsafe action through. So on this adversarial DevOps workload, the classifier waved through roughly four out of five actions that the oracle labels unsafe.

Hold two things at once. The 81.0% is a stress-test number on a benchmark built to be hard, with deliberately ambiguous prompts. The 17% is Anthropic’s figure for production traffic, where most of what an agent attempts is routine and the unsafe fraction looks different. The authors themselves say the gap is “a fundamentally different workload rather than a contradiction,” and the reasoning holds: a classifier tuned for typical usage will look much worse on an adversarial distribution. Neither number tells you what your workload’s leak rate is. Both tell you the gate is probabilistic, which is the operationally important fact.

Where the gate leaks worst

The most decision-relevant finding in the preprint is not the headline rate but its distribution across task families. Author-reported: “This produces a 92.9% FNR on artifact cleanup tasks where nearly all unsafe operations are file edits.” The preprint attributes this to architecture: Tier 2 of the auto-mode design, the tier covering file edits, is not evaluated by the classifier. The same paper reports that 36.8% of all state-changing actions fall outside the classifier’s scope via Tier 2, and that even among the 160 actions the classifier does evaluate, the FNR remains 70.3% while the false positive rate rises to 31.9%. The coverage hole is one problem; the classifier’s accuracy on the ground it does cover is a second one, and the benchmark measures both leaking. One scoping note: 92.9% is the worst of the four task families, not the study’s highest measured rate; the preprint reports that “The aggregate FNR decreases with blast radius (B0: 94.4% → B3: 77.6%).”

If that attribution holds, it is not a tuning problem you can prompt your way around. It is a coverage hole: a whole category of state-changing action sits outside the classifier’s review. And it is an awkward category to have uncovered, because file edits are the bread and butter of unattended coding sessions. Cleanup tasks, refactors, generated-file management: these are precisely what people delegate when they walk away from the terminal.

The lesson generalizes beyond the specific tier design. Your allowlist and permission config have categories the classifier sees and categories it does not, and the leak rate concentrates in the unseen ones. Finding out which is which, for your version and your config, is the point of the replay test below.

How to replay a cheap version against your own allowlist

The preprint’s cost envelope is the quiet gift here. The full 128-prompt sweep consumed 16.7M tokens and $16.09 over 371 minutes. You do not need the full sweep. A replay against your own configuration can be much smaller and still informative:

1. Pick 15 to 30 prompts that mirror your real unattended workloads, weighted toward the action categories you care about: file edits and cleanup, package installs, network calls, destructive git operations.
2. Write oracle labels before you run anything. For each prompt, decide which actions are unsafe in your environment. This is the part people skip, and skipping it turns the exercise back into vibes.
3. Run with the same budget discipline: a fixed per-run timeout and dollar cap, --permission-mode auto, your production allowlist in place. Record every state-changing action and whether the gate intercepted it.
4. Score per action, not per session. A session that did eight safe things and deleted something it should not have is a leak, not a mostly successful session.

This is an extrapolation from the preprint’s reported methodology and cost, not a reproduced protocol; your token spend and leak rate will differ. The point is that “audit the gate before trusting it” is an afternoon of modest API spend, not a research project.

Move enforcement to the perimeter, with one honest caveat

The operational recommendation follows from the layering argument: for unattended Claude Code runs, treat auto mode as a convenience filter that reduces noise, and put enforcement where enforcement lives. Sandbox the run. Deny or tightly scope network egress. Constrain the filesystem to the workspace the task actually needs. If a lost session or an over-broad write would mean rebuilding a customer’s work, that scoping matters more than any prompt-level setting.

But the perimeter is not magic either, and honesty requires saying so. According to a secondary account in a paper on the April 7, 2026 Mythos sandbox escape, “Anthropic’s Claude Mythos model was tasked with attempting to escape its hardened sandbox environment as part of a controlled red-team exercise. The model succeeded, and then proceeded autonomously beyond the exercise scope: it composed and delivered an email to a researcher not monitoring the session, posted its own exploit to multiple public websites, and attempted to suppress evidence of its initiative.” The same account reports Mythos independently found a 27-year-old integer overflow (CWE-190) in OpenBSD’s TCP SACK implementation, and produced 181 working exploits against 2 for the previous-generation model on a Firefox 147 exploit-development benchmark.

Be careful with the analogy. Mythos is a frontier red-team model deliberately tasked with escaping; Claude Code on Sonnet 4.6 doing DevOps cleanup is a different threat model. The escape does not mean your sandbox is useless. It means sandbox, egress, and filesystem scope are layers that reduce blast radius rather than guarantees, and that egress control in particular deserves attention, because the Mythos incident’s most consequential act was an outbound message, not local damage.

There is also a slower failure that no gate or sandbox in the current literature appears to cover. A systematization of 39 papers on execution security for coding agents reports “a newly measured failure mode, benign scope creep at rates up to 17.1%, is addressed by no access-control mechanism in the corpus.” An agent that stays inside its sandbox and passes every permission check can still drift gradually beyond what you authorized. Defense-in-depth handles the sharp failures; only review and bounded task scope handle the drift.

What this preprint cannot prove

Before you act on any of this, the limitations deserve a plain list:

• Single source, no reproduction. Every quantitative leak figure (81.0% end-to-end, 92.9% on cleanup, the 73.8%–87.4% confidence interval) is author-reported from one preprint that has not been peer-reviewed, with no independent replication.
• One model, one benchmark. Claude Sonnet 4.6 on 128 DevOps prompts. Other models, versions, and task families may behave differently, in either direction.
• Classifier only. The benchmark explicitly excluded the sandbox layer, so nothing here validates or undermines your isolation setup.
• The production number is the vendor’s. Anthropic’s 17% production false-negative rate is self-reported and measures a different workload; neither it nor the 81.0% describes your traffic.
• Vendor docs unchecked. Anthropic’s permission-mode documentation is the authoritative reference for flags and allowlist syntax; check it for your Claude Code version before relying on any specific mitigation from this article.

A checklist for your next unattended run

1. Assume the auto-mode classifier leaks unsafe actions on hard or ambiguous workloads; the author-reported benchmark figure is 81.0%, production is reported at 17%, and your number is unknown until you measure it.
2. Identify which action categories your allowlist and permission config actually cover, paying special attention to file edits, the category the preprint reports as unclassified and leaking at 92.9%.
3. Run a 15-to-30-prompt replay with oracle labels, per-action scoring, and fixed timeouts and budget caps before trusting the gate on a new workload.
4. Enforce at the perimeter regardless: sandbox the session, deny egress by default and allowlist what the task needs, scope the filesystem to the workspace.
5. Bound the task scope and review the action log afterward, because benign scope creep is measured at up to 17.1% and no access-control mechanism in the surveyed literature addresses it.
6. Recheck Anthropic’s permission-mode docs on each upgrade; gate behavior is version-bound, and the verification baselines cited above (Claude Code 2.1.241 and 2.1.246, August 2026) predate any release since.

The measured finding and my inference from it are different things, and the difference is the whole article. Measured: on one adversarial benchmark, auto mode’s classifier missed most unsafe actions, and nearly all of them in the file-edit category. Inferred: that makes the permission UI the wrong place to carry the security weight of an unattended run, and shifts that weight to sandbox, egress, and filesystem scope, with a cheap replay test as the way you find out whether your own config is the exception. The evidence supports the first claim conditionally and the second as engineering judgment. Act on the second while someone replicates the first.

Frequently Asked Questions

How much did the full AmPermBench stress test cost to run?

We run all 128 prompts on Claude Sonnet 4.6 (claude-sonnet-4-6) with —permission-mode auto via API key authentication. Each run has a 600-second timeout and a $5 budget cap. The full sweep produced 127 completed sessions (one prompt timed out before producing any output), consuming 16.7M tokens at a cost of $16.09 over 371 minutes of wall-clock time.