Why Agent Security Tests Must Audit Full Trajectories, Not Single Turns

Single-turn jailbreak tests certify the wrong property: they measure whether an agent refuses a malicious prompt, not whether it stays safe across a multi-step execution. When ASEval (arXiv 2605.22321) perturbed benign multi-turn conversations into adversarial test cases, the average risk-trigger rate across 11 LLM-backed OpenClaw agents rose from 22.9% under existing injection techniques to 47.4%. Existing red-team methods were missing more than half the failures the trajectory lens exposes. If your agent passed turn-by-turn evaluation, that gap is now your problem.

Why does single-turn red-teaming miss agent risk?

Single-turn evaluation misses agent risk because the failure classes that matter in deployed agents (tool-misuse chains, goal drift, late-stage privilege escalation) are properties of a sequence, not of any individual prompt-response pair, so they cannot be expressed, let alone detected, in one turn.

The structural mismatch is worth stating plainly. A response-level harness presents an input, collects an output, and judges the output. That model assumes the safety-relevant event happens at the boundary: a bad instruction goes in, a bad completion comes out. Autonomous agents do not work that way. They act through tool calls whose effects land outside the text channel, they carry state forward across turns, and they accumulate context that changes what a later, individually benign instruction actually does. Judging the final response of such a system is like auditing a database migration by reading the last log line.

ASEval’s analysis makes this concrete: the security failures it surfaces emerge from multi-turn delivery, persistent state, and tool-mediated actions, three properties that response-level evaluation does not observe at all. This is not a claim that existing injection benchmarks are badly calibrated. It is a claim that they instrument the wrong layer. An input filter can be perfect and the agent can still end up exfiltrating a file in step nine of twelve, because no single step looked like an attack.

The consequence lands on certification, not just on benchmark design. Teams that ship agents with an “alignment-tested” badge earned on single-turn jailbreak suites have measured boundary refusal rates. The 24.5-percentage-point lift ASEval reports suggests those measurements undercount actual triggerable risk by a factor of two on the tested harness, which means the certificate certifies a property the threat model does not care about.

How does ASEval audit a full trajectory?

ASEval audits a trajectory by generating realistic benign multi-turn conversations, mutating them into risk cases with designed perturbation operators, executing each case against the live agent, and judging the complete action trace with an action-grounded oracle instead of grading the final text response.

Each stage of that pipeline carries a design decision worth understanding. Starting from benign conversations rather than adversarial prompts matters because it mirrors how real compromise attempts unfold: the interaction looks ordinary, and the risk is introduced by perturbing the flow rather than by handing the agent a prompt that obviously trips a filter. The perturbation operators then transform those benign dialogues into executable risk test cases. This is the step responsible for the headline number, since the same underlying scenarios that existing injection techniques trigger at 22.9% trigger at 47.4% after perturbation.

The oracle is the other half of the design, and the part most likely to be copied or fumbled by imitators. A response-level judge asks “was this completion unsafe?” An action-grounded oracle asks “did the trace contain an unsafe action?”, grounding the verdict in what the agent did across the trajectory rather than in what it said at the end. That distinction is what lets ASEval catch a failure where every individual response reads as benign but the sequence of tool calls is not.

The oracle is not perfect, and the paper is honest enough to quantify it: it matches human annotation at 93.2% binary F1, which means roughly 1 in 14 risk judgments may diverge from a human label. That is good for an automated judge, and it is also a reminder that some of the 47.4% is oracle judgment rather than ground truth. Teams adopting the approach should treat oracle disagreement as a line item in their error budget, not round it away.

What do A3S-Bench’s numbers actually show?

A3S-Bench is the concrete artifact: 2,254 executable multi-turn test cases spanning diverse usage scenarios, delivery patterns, and agent-level security risks, and across 11 LLM-backed OpenClaw agents the average risk-trigger rate climbed from 22.9% under existing injection techniques to 47.4% under ASEval’s perturbation operators, per the ASEval paper.

Two properties of that result deserve emphasis. First, the cases are executable: each one is run against the target agent, not scored as a static prompt. That is what makes the benchmark a trajectory audit rather than a bigger jailbreak prompt set, and it is also what makes it expensive to run, a point that matters for the certification discussion below. Second, the scale of the delta is the story. A 24.5-point lift is not measurement noise or a marginal technique improvement; it indicates that the existing injection toolkit systematically under-triggers risk in multi-turn settings, which is exactly what you would expect if a large class of triggers only exists at the trajectory level.

The comparison against a single-turn pipeline looks like this:

The cost row is not incidental. Moving the certification target from text to traces multiplies the infrastructure required: every test case needs a running agent, a tool environment, and isolation, where a single-turn suite needs a prompt list and a judge. That cost asymmetry is precisely why single-turn evals became the default, and why the incentive to keep certifying on them persists even after their blind spot is documented.

Which failures only appear mid-execution?

ASEval attributes the failures it surfaces to three structural properties of agent execution: multi-turn delivery, persistent state, and tool-mediated actions, none of which a response-level evaluation can observe because none of them is fully present in any single turn.

Multi-turn delivery

A payload split across turns defeats any per-turn filter by construction. Each individual message is benign; the attack is a property of the sequence. This is the delivery mechanism behind the perturbation operators’ success: the risk is assembled by the conversation, and the agent that would refuse the assembled instruction in one shot accepts it piecemeal because no piece crosses the refusal threshold. Defenses that inspect turns independently have no representation of the thing they are supposed to stop.

Persistent state

Agents remember. State written in an early turn, a file, a memory entry, a cached intermediate result, gets read in a later turn where it influences behavior outside the current input’s scrutiny. The security concern here is not hypothetical hygiene; it is that the industry’s performance work is actively expanding this surface. Workload-Aware Caching for Multi-Agent Systems reports latency reductions of up to 64.7% by reusing state across agents. The optimization and the attack surface are the same mechanism: state that persists beyond the turn that created it, consumed by a later step that trusts it. Every caching layer that reuses cross-agent state is also a channel through which early-turn contamination reaches late-turn actions. Teams adopting these performance techniques should expect their trajectory-level risk exposure to grow with their cache hit rate.

Tool-mediated actions

The agent’s effect on the world happens through tool calls, and tool calls are where benign text becomes consequential action. A trajectory can read as entirely reasonable in the transcript while the sequence of invocations escalates privileges or chains tools in a way no single call justifies. This is also where the evaluation infrastructure itself becomes a security component. ExecuGraph, a six-agent backend code-synthesis framework from the same July 2026 cluster, guards every evaluation with a subprocess-isolated sandbox and a wall-clock timeout. That is not a safety paper; it is a performance framework that found runtime isolation non-optional for executing agent output at all. Trajectory-level security auditing assumes exactly those controls are present, because you cannot run 2,254 adversarial cases against an unsandboxed agent without the test suite itself becoming an incident.

Goal drift and late-stage privilege escalation sit across all three surfaces: drift is what happens when persistent state and multi-turn context pull the agent off its original task, and escalation is what a tool-misuse chain looks like from the permission system’s side. Neither has a single-turn analogue.

What changes in your agent certification pipeline?

The practical consequence is that certifying an agent now requires runtime trajectory auditing alongside input/output filters: red-team pipelines that pass agents turn-by-turn will certify agents that fail mid-execution, and the only way to close that gap is to test the trace.

Concretely, the pipeline gains three components. First, a trajectory-level suite: A3S-Bench is the current reference point at 2,254 executable cases, and the executable part is the commitment, since running it requires the same sandboxed runtime that production deployments should already have. Second, an action-grounded oracle, with the accuracy caveat carried into the sign-off criteria rather than ignored. Third, trace instrumentation rich enough to localize failures once found. Here the adjacent research is directly useful: Workflow-Localized Mechanism Learning introduces Node-Mechanism Attribution to identify the failed workflow node, the implicated mechanisms, and the smallest valid edit target. That is debugging tooling, not safety tooling, but trajectory-localized attribution is exactly what you need when an audit flags a failure somewhere in a twelve-step run and “somewhere” is not an actionable answer.

The training side is moving the same direction. AttriMem augments global outcome rewards with local rewards derived from token-level contributions to the final answer, part of a broader shift toward process-level signals rather than outcome-only feedback. Evaluation and training are converging on the same conclusion: the outcome is the wrong granularity, and the trajectory is where the information lives.

The cost objection deserves a direct answer. Runtime trajectory auditing is more expensive than input/output filtering in compute, in infrastructure, and in engineering time, and that is the real reason adoption will lag the evidence. But the relevant comparison is not filter cost versus audit cost; it is audit cost versus the cost of certifying a property your threat model does not care about while the property it does care about goes unmeasured.

What are ASEval’s limits, and what should you verify before adopting?

ASEval’s headline numbers are directional, not universal: every figure comes from one harness of 11 LLM-backed OpenClaw agents, judged by a semi-automated oracle that matches human annotation at 93.2% binary F1.

The strongest limitation is generalization. Agents differ in tool inventories, memory architectures, and permission models, and each of those differences reshapes the trajectory-risk surface. An agent with a narrower tool set has fewer tool-misuse chains available to it; an agent with a different memory design has a different persistent-state exposure; a tighter permission boundary changes what late-stage escalation can reach. The 22.9%-to-47.4% gap was measured on OpenClaw agents specifically, and there is no evidence in the brief that it transfers numerically to heterogeneous stacks. Treat it as proof that the gap exists, not as an estimate of your gap.

The oracle caveat cuts both ways. At 93.2% binary F1, roughly 1 in 14 risk judgments may diverge from a human label, which means some flagged failures are judgment calls, and presumably some real failures pass unflagged. For a certification gate, that error rate argues for human review of borderline traces rather than full automation, which raises the audit cost further.

The adoption verdict holds anyway. Add trajectory-level auditing to agent certification: run an executable multi-turn suite like A3S-Bench against your own agents, judge traces with an action-grounded oracle, and keep input/output filters as the boundary layer they are good at being. Vendors shipping agents certified on single-turn benchmarks should expect to rerun evaluation against trajectory-level suites, because the alternative is certifying a blind spot that the research has now measured, published, and put a number on.

Frequently Asked Questions

How does ASEval’s 93.2% F1 oracle error rate impact certification decisions?

The 93.2% binary F1 score means roughly 1 in 14 risk judgments diverge from human annotation, introducing a measurable error budget into automated certification. Teams must treat oracle disagreement as a line item in their risk assessment rather than assuming the automated judge is infallible, which may necessitate human review of borderline traces to avoid false positives or missed failures.

Can A3S-Bench results be generalized to agents with different tool inventories?

No, the 22.9% to 47.4% risk-trigger lift was measured exclusively on 11 LLM-backed OpenClaw agents, making the specific numbers non-transferable to heterogeneous stacks. Agents with narrower tool sets have fewer tool-misuse chains available, while those with different memory architectures face distinct persistent-state exposures, meaning the gap exists but its magnitude varies by deployment.

What infrastructure is required to run trajectory-level audits like ASEval?

Running executable multi-turn test cases requires a live agent, a tool environment, and sandboxed runtime isolation, which is significantly more expensive than static prompt evaluation. Frameworks like ExecuGraph demonstrate that subprocess-isolated sandboxes and wall-clock timeouts are non-optional for executing agent output safely, adding compute and engineering overhead to the certification pipeline.

How does Workflow-Localized Mechanism Learning help with trajectory debugging?

WML introduces Node-Mechanism Attribution to identify the specific failed workflow node, the implicated mechanisms, and the smallest valid edit target after an audit flags a failure. This trajectory-localized debugging is essential when a failure occurs deep in a multi-step run, as it provides actionable granularity that single-turn evals cannot offer.