Can Web Agent Benchmarks Survive an Adaptive Attacker? What ANGLER Tests

If a vendor’s web-agent score moves by a few points and the agent itself hasn’t changed, you have two working explanations: the benchmark is noisy, or something about the evaluation changed underneath you. A new arXiv preprint, “Verifier-Grounded Benchmarking with an Adaptive Attacker for Web Agents” (ANGLER), gives both explanations concrete numbers. Across 8 vision-language-model web agents and 9 websites, its authors report that an adaptive attack cuts the average benchmark score they track from 61.9% under attack-free execution to 44.4% (the ANGLER preprint), a 17.5-point swing. They also report that aggregate estimates below roughly 75 rollouts per website, model pair are unstable (the ANGLER preprint).

Two caveats before anything else. First, everything ANGLER-specific here is author-reported in a single preprint that has not been independently replicated; treat its numbers as a stress test of how you read scores, not as new leaderboard facts. Second, the preprint’s identifier carries October 2026 numbering, and the submission date circulating with the announcement (2026-11-03) is worth confirming on the abstract page if the date matters to you.

The practical consequence is durable even if ANGLER’s specifics get revised: before you gate a build or purchase on a web-agent benchmark number, you should know what the verifier actually checks, whether anything adversarial has been run against it, and how many rollouts stand behind the delta you are about to act on.

What ANGLER is, and what its verifiers check

ANGLER frames web-agent evaluation as “verifier-grounded benchmarking” and adds a second actor to the usual setup: an adaptive attacker that generates attack candidates and iterates against the target agent rather than replaying a fixed attack set. In its adaptive stage, the authors use Qwen3.5-9B as the proposal model for candidate generation and keep it fixed across target agents, so attacker behavior is constant while the target varies (the ANGLER preprint).

The evaluation runs 8 vision-language models on REEF, “a benchmark of Red-teaming Executable Environments with Faithful Verification” (the ANGLER preprint): GPT-4o, GPT-6.1 Sol, Claude Sonnet 4.5, Claude Sonnet 5.5, Qwen3-VL, Qwen3.7-Flash, GLM-4.6V-Flash, and GLM-5V-Turbo. The two headline metrics are expanded in the abstract itself: ASR is end-to-end attack success rate, BSR is benign task success rate. REEF’s 9 websites span travel, commerce, productivity, media, and communication; the results discussion names sites including Coursera, Facebook, and YouTube, though no single list enumerates all nine.

The verifier mechanics are specified, not just branded. “We determine rollout outcomes directly from recorded website states rather than relying on agent self-reports or LLM-as-a-judge scoring,” the authors write (the ANGLER preprint). Each attack case is paired with three programmatic verifiers: a user verifier that checks whether the benign objective was achieved, an attacker verifier that checks whether the malicious objective was achieved and remains satisfied at termination, and a recovery verifier that confirms whether a malicious effect observed earlier is absent at termination. The attacker verifier must be false in the reset state before each rollout, so a pre-existing malicious effect cannot be counted as attack success. That is a concrete, published answer to the question this article argues you should be asking of every benchmark you rely on.

The headline numbers, held at arm’s length

Reported results, per the authors:

• Average ASR rises from 15.2% for the non-adaptive baseline to 20.4% under ANGLER’s adaptive attacker (the ANGLER preprint).
• Average BSR falls from 61.9% under attack-free execution to 44.4% when attacks are present (the ANGLER preprint).
• The complete evaluation was repeated three times using Qwen3-VL across all 9 websites, 100 rollouts per website per run (the ANGLER preprint), as a stability check rather than a single lucky pass.

Read the BSR figure carefully: it does not mean attacks make agents 17.5 points worse at their jobs. It means that the number a benchmark reports for the same agents changes by 17.5 points when the evaluation environment is adversarial rather than clean. That is a statement about the benchmark’s sensitivity, not (only) about agent robustness. Whether that sensitivity transfers to the benchmarks you actually use, WebArena, VisualWebArena, Mind2Web and the rest, is exactly what a single preprint on one environment cannot establish.

Does the benchmark publish its verifier and success criterion?

This is the first question to ask of any web-agent score, and ANGLER’s framing makes it central: if a benchmark’s success criterion is a verifier you cannot read, you cannot tell what a score certifies. A lenient criterion, say, string-matching an answer substring, certifies something different from a state-based check that the task’s side effects actually happened. (Which criterion any specific public benchmark applies is a question its maintainer documentation must answer before you act on the score; REEF’s three programmatic verifiers are one benchmark’s published answer, not a field-wide survey.)

What the evidence does establish is that non-public evaluation is common at the provider level. The authors of WASP, a prompt-injection security benchmark, put it directly:

“Finally, many benchmarks—especially those used by major model providers to assess pre-launch risk and discussed in their system cards—are not released publicly. The community, therefore, lacks a standard way of tracking attack success rate, hindering reproducibility and a unified view of the risk.” (WASP, arXiv:2504.18575)

The implication for a buyer is unglamorous but actionable: when a model card or launch post cites an internal web-agent evaluation, the success criterion behind that number may be something you will never be able to inspect. Discount accordingly, and weight published, re-runnable evaluations higher even when their numbers look less flattering.

Has anything adaptive or adversarial been run against it?

The second question is whether the benchmark’s environment has faced an attacker that adapts. ANGLER is the newest entry in a lineage, not the start of one, and that matters for how much novelty to credit. WASP built prompt-injection evaluation inside a sandboxed web environment derived from VisualWebArena, simulating injection attacks without exposing agents or users to real threats (WASP, arXiv:2504.18575). Muzzle went further earlier in 2026, automating adaptive red-teaming against indirect prompt injection:

“Compared to all prior work on IPI against web agents, Muzzle automatically discovers: (1) end-to-end attack paths spanning multiple web pages across applications; (2) vulnerable UI elements along these paths that serve as attack surfaces; and (3) adversarial instructions and payloads that hijack the agent to execute specified adversarial objectives.” (Muzzle, arXiv:2602.09222)

Muzzle’s red-team agents run on GPT-4o and GPT-4-Turbo, and in its Gitea case study the target agent, Agent-E, tended to fully commit to the adversarial objective once hijacked, reaching end-to-end success rates such as 4/5 for adding an unauthorized collaborator (Muzzle, arXiv:2602.09222). So when ANGLER reports that adaptivity raises ASR from 15.2% to 20.4%, the direction of the finding is consistent with an established pattern: fixed attack sets understate what a motivated, iterating attacker achieves. ANGLER’s contribution, on the evidence available, is the verifier-grounded framing and the scale of its stability methodology, not the invention of adaptive attack. Keep that distinction when you see coverage that implies otherwise.

There is also a deployed-product dimension worth one paragraph. An audit of browser agents found that four products, Comet, ChatGPT Agent, Project Mariner, and Director, save some profile state by default, with Director the only one that does not communicate this to the user (Privacy Practices of Browser Agents, arXiv:2512.07725). Persistent profile state is precisely the kind of surface an adaptive attacker can probe across sessions, and it is invisible in a benchmark score. Transparency about retained state is a reasonable ask alongside transparency about evals.

How many rollouts stand behind the delta?

The third question is statistical, and ANGLER’s own methodology section is its most quietly damning result. The authors write that “75 rollouts per website–backbone pair, or 675 rollouts per backbone across 9 websites, provide a practical minimum” for stable aggregate estimates (the ANGLER preprint). Reported results use the full budget of 100 rollouts per pair (the ANGLER preprint), and the full evaluation was repeated three times on one backbone to confirm stability. Now hold that floor up against any leaderboard comparison you are about to act on. If the authors of a deliberately rigorous evaluation need 75 rollouts per pair before aggregate numbers stop wobbling, then leaderboard deltas of two or three points between competing agents, the kind that decide procurement bake-offs, may be inside the noise band even in a friendly environment, before any attacker enters the picture.

This is inference from ANGLER’s stability finding, not a direct measurement of any specific leaderboard’s variance. But it is a bounded, checkable inference: ask whoever published the score how many rollouts back it, per task and per environment, and whether the delta you care about survives a repeat run.

Is the environment still alive?

The fourth failure mode needs no attacker at all. WebCanvas, evaluating web agents in live online environments, reports that 96 of 780 Mind2Web tasks, 12%, are completely expired on their corresponding live websites (WebCanvas, arXiv:2406.12373). Worse for anyone reading rankings:

“In addition, as shown in Table 4, the model trained on the Mind2Web training set does not generalize well to the online environment one year later. The comparative relationship between the results of MindAct-Large [3], GPT-3.5, and GPT-4 is the opposite of that in offline testing.” (WebCanvas, arXiv:2406.12373)

The ordering inverted. A static-snapshot benchmark can rank agents one way and the live web can rank them the opposite way a year later, with no attack, no controversy, just the web doing what the web does. Combine this with ANGLER’s noise floor and attack sensitivity and you get the full picture of why scores move without behavior changing: stale tasks, thin rollout budgets, and an environment that may be friendlier or more hostile than the one the benchmark froze.

The checklist, and how to weight what survives it

Pulling the four axes together as a working interrogation, with the evidence behind each row:

My recommendation, conditional on the evidence above: treat web-agent benchmark scores as a screening signal, not a selection signal. If two candidate agents sit within a few points of each other on a leaderboard and you cannot answer the first three questions for that benchmark, I would not let the delta drive the decision, run your own tasks instead, on your own target sites, with enough rollouts that you can see the variance, and weight the product’s transparency (published evals, disclosed state retention) as evidence in its own right. That is a judgment call about how to spend limited evaluation budget, and it follows from the noise floor and the inversion result rather than from any claim that benchmarks are useless. A benchmark with a published verifier, an adversarial evaluation, and a documented rollout budget is worth considerably more than a bigger number without them.

What ANGLER does not establish

Honesty about the limits is load-bearing here. ANGLER is one preprint, not replicated, and its adaptive stage fixes a single proposal model (Qwen3.5-9B), so the tested attack surface may not transfer to other attackers, verifiers, or benchmarks. Which public benchmarks publish their verifiers, and which rely on lenient criteria, is still a question you have to answer benchmark by benchmark from maintainer documentation. And adaptive red-teaming of web agents predates ANGLER, the field was already moving this way.

What ANGLER does establish, even at preprint confidence, is that the question in this article’s title is now quantitative rather than rhetorical. Benchmarks can survive an adaptive attacker in principle, but only if their verifiers, environments, and statistics are built to be inspected. Until the one you are reading is, the burden of proof sits with the score.

Frequently Asked Questions

How many rollouts does ANGLER recommend for stable benchmark estimates?

The authors write that “75 rollouts per website–backbone pair, or 675 rollouts per backbone across 9 websites, provide a practical minimum” for stable aggregate estimates (the ANGLER preprint).

What specific verifiers does the REEF benchmark use to check outcomes?

Each attack case is paired with three programmatic verifiers: a user verifier that checks whether the benign objective was achieved, an attacker verifier that checks whether the malicious objective was achieved and remains satisfied at termination, and a recovery verifier that confirms whether a malicious effect observed earlier is absent at termination.

How much of the Mind2Web task set is expired on live websites?

WebCanvas, evaluating web agents in live online environments, reports that 96 of 780 Mind2Web tasks, 12%, are completely expired on their corresponding live websites (WebCanvas, arXiv:2406.12373).