When VLM Agents Think With Images, What Does the Trace Prove?

The October 2025 preprint On the Faithfulness of Visual Thinking supplies the cleanest statement of a split every visual-agent team should test. Before you tell a customer “the model zoomed into the chart, so it read it,” run two checks. First, swap or edit the image and see whether the answer changes; that tests whether the model used the image at all. Second, redact or perturb the evidence the agent’s tools returned and see whether the reasoning steps and the answer move; that tests whether retrieved visual content was actually integrated into the reasoning. A zoom-in tool call, on its own, proves neither. The preprint deserves an explicit label up front: it is single-team, unreplicated work, equal-contribution authors Zujing Liu and Junwen Pan, with Qi She of ByteDance, Yuan Gao of Wuhan University, and Gui-Song Xia, done during Liu’s ByteDance internship, and its experiments cover two 7B-scale models, DeepEyes and Pixel-Reasoner, on V* Bench and HR-Bench 4K/8K. What it contributes, and what a small cluster of companion preprints now makes actionable, is a framing: a visual trace is an audit surface, not a grounding guarantee, turned here into a two-level checklist that maps each level to the claims it licenses.

What a thinking-with-images trace actually contains

The “thinking with images” paradigm differs from classic vision-language reasoning in where the image enters. In text-only reasoning, vision is a static initial context: one image in, an answer out. Thinking-with-images agents instead make images intermediate reasoning steps through extrinsic operations (toolkits, code executors) or intrinsic generation. The anchor paper’s most useful observation for anyone logging these traces is about mechanism. In agentic pipelines with visual grounding and tool calling, the authors note, visual information enters “via image zoom-in tool calls rather than being generated by the model. Thus, the visual information can be regarded as observation tokens, which are appended to the ongoing reasoning process to guide subsequent reasoning steps” (arXiv:2510.23482).

Read that sentence the way a platform engineer would. Appending a token to a context is a mechanical event you can log with certainty. Guiding subsequent reasoning is a causal claim the log does not contain. The trace records that a crop happened; it does not record that anything downstream depended on the crop. A demo video of an agent zooming into a chart is, in this framing, footage of token appends.

This is not a niche lab behavior. Visual-ARFT (Visual Agentic Reinforcement Fine-Tuning) is explicitly training open-source large vision-language models for agentic reasoning that includes browsing websites and executing code to manipulate images. If your team is fine-tuning or orchestrating anything in that family, your production traces will be full of exactly these observation tokens, and someone will eventually present one as evidence the model “looked.”

Level 1: did the model use the image at all

The anchor paper’s first move is to point out that nearly all prior faithfulness work on vision-language models sits at one level: “Prior work mainly examines input-level visual neglect, i.e., whether models utilize visual inputs when making predictions [31, 57, 8, 32], rather than whether visual information is faithfully incorporated throughout the reasoning process” (arXiv:2510.23482). Input-level neglect is the failure mode where the model answers from text priors and ignores the pixels entirely.

The directly implementable test at this level comes from a different preprint, Grounding on Request, which introduces Flip and Stay: edit the modality the question actually asks about and check that the prediction follows the edit (Flip), while editing competing evidence and checking that the prediction stays put (Stay). The paper’s framing matters for how you read your own eval dashboards: correctness alone does not establish modality control. A model can answer a chart question correctly because the answer was inferable from the axis labels and common sense, not because it read the bars. Only the perturbation result distinguishes those.

Passing a Flip-and-Stay suite licenses a bounded claim: on this eval distribution, predictions depend on image content. It says nothing about whether the reasoning steps that produced the answer were honest. That is the second level.

Level 2: was the retrieved evidence actually used

Reasoning-level faithfulness asks whether the visual evidence in the trace is causally integrated into the intermediate steps and the final answer. The strongest worked example of auditing at this level comes from a tool-trace study outside vision, Auditing Tool Traces in Agentic LLMs. Its two-stage procedure is a template: Stage 1, Per-Tool TF-IDF Decomposition (PT2D), localizes label-predictive patterns to individual tool results rather than diluting the signal across the concatenated trace; Stage 2, Token-Class Redaction, counterfactually removes those token classes and re-measures.

The case study is the cautionary tale. In a TCR, pMHC protein-binding pipeline, the audit “overturned an apparent generalization success by tracing it to PDB identifiers and showing that removing the identifier class eliminated nearly all above-chance performance while a non-overlapping control left performance intact” (arXiv:2610.25540). The agent was not reasoning about protein structure; it was pattern-matching database IDs embedded in its own tool results. The traces looked like successful retrieval and use. The counterfactual redaction showed the dependence was on a shortcut.

Two caveats bind this evidence to your decision. First, it is an adjacent domain: text-heavy tool results in a scientific pipeline, not image crops in a chart agent. The audit design transfers; the effect size may not. Second, the design itself is the point, redacting a suspected shortcut class and re-running is directly applicable to vision agents. If your screenshot agent’s tool returns an OCR transcript alongside the crop, redact the transcript: an answer that survives unchanged tells you the transcript was not load-bearing, nothing more. Implicating or clearing the crop itself takes the visual intervention described above, replacing crops with mislocalized regions or random noise.

The vision-domain version of this test is not hypothetical. The anchor preprint intervenes on the two components of a multimodal trace separately, injecting mistakes into textual thoughts and replacing zoom-in crops with mislocalized image regions. On V* Bench, textual intervention cost DeepEyes an average of 13.09 points of accuracy while visual intervention cost it 1.05; Pixel-Reasoner showed the same asymmetry, 8.20 versus 1.64. The textual effects were statistically significant under McNemar’s test and the average visual effects were not; in some settings, replacing the crops slightly improved accuracy. The authors’ summary of the pattern is that “the visual evidence is largely ignored” (arXiv:2510.23482). A second evaluation, using an external GPT-4o judge, scored the visual components of DeepEyes traces as relevant to the predicted answer in 50.00 percent of V* Bench cases on average and sufficient to answer in 59.41 percent, with Pixel-Reasoner lower on both counts at 26.23 and 40.98.

Designing the perturbation: factual distractors beat corrupted pixels

A natural instinct is to test grounding by corrupting inputs: blur the image, shuffle the pixels. Lazy Grounding, a study of search agents, suggests that is the weak version of the test. The authors show that “in some cases, search agents’ reasoning and responses may be steered by completely factual but distracting information,” a failure they name lazy grounding, and report accuracy drops averaging 5.9 points and reaching 17.3 points across 12 model-benchmark pairs (arXiv:2608.30303).

The implication for eval design is that plausible beats corrupted. A blurred chart is trivially detectable; a model can learn to flag it or simply fail safe. A chart that is real, well-formed, and irrelevant, retrieved by your own pipeline, factual in every pixel, is the harder test of whether the agent discriminates evidence by relevance or consumes whatever arrives. Again, the numbers come from text search agents, not image zoom-in traces, so treat the magnitudes as directional evidence about the failure class rather than a prediction for your system.

There is also cross-modal corroboration that traces can be decorative rather than causal. A chain-of-thought faithfulness study across 16 frontier models and six domains frames its findings as “reasoning rigidity: the degree to which a model’s answer is invariant to perturbation of its own reasoning steps.” That is the text-CoT analog of a visual trace that does not drive the answer: change the steps, watch the answer hold still. The anchor paper sits squarely in that lineage. Its related work cites both threads, noting that “Lanham et al. [23] apply different interventions to the CoT and observe the resulting changes in final answers to evaluate CoT faithfulness” and that “Xiong et al. [53] utilize counterfactual interventions to investigate the faithfulness of the reasoning process.” Its own textual intervention is explicitly “Inspired by Lanham et al. [23]” (arXiv:2510.23482).

Trace economics: call counts are not evidence

Teams sometimes reach for a softer proxy: the agent made four visual calls, so it must have been examining the image carefully. The visual orchestration tax preprint dismantles that proxy. It identifies an orchestration-level redundancy mode in which semantically unchanged images are repeatedly reconstructed as image-conditioned requests at the VLM API boundary, and quantifies it on a worked ChartQA example: “SeeingEye’s translator issues four visual calls per query in the default inner tool loop, so M1t=4 and M2=0.75” (arXiv:2610.08170). Three of those four calls, by the paper’s redundancy metric, re-requested essentially the same visual evidence.

Two consequences follow. Practically, redundant visual calls are a real cost line, latency and token spend that buys nothing. Evidentially, call volume in a trace can be substantially waste rather than examination. When you audit, count distinct visual evidence acquired per query, not calls issued. And when a demo shows an agent making repeated zooms, the honest question is not “look how thoroughly it inspected the chart” but “why did the pipeline need to ask four times?”

The claims matrix

The governing rule for a shipping team: promise customers only what the highest level passed actually demonstrates. Each level requires its own test design, and passing one does not license claims at the next.

Read the middle row’s licensed claim literally. A passing perturbation suite shows the agent is sensitive to its evidence. Sensitivity is not the same as faithfulness on the next unseen input, which is why the licensed claim is deliberately phrased in terms of the test that ran, not as a property of the model.

The counter-evidence: audit, then train, then re-audit

Two results bound how far the skeptical reading should be pushed. First, the Grounding on Request paper finds that “modality control can be learned with very few parameters” (arXiv:2610.33841). Grounding failures are a trainable property, not a permanent model trait. That is good news with a sharp edge: any failure rate you measure today is version-sensitive. A targeted fine-tune can fix a grounding gap, and a routine model upgrade can reintroduce one, so flip-and-stay and redaction suites belong in CI, not in a one-time pre-launch report.

Second, perturbation is not the only audit path. The Visual Reasoning Tracer benchmark takes a trace-scoring approach: it “asks an MLLM to output a visual trace, a sequence of spatially grounded segmentation masks aligned with each reasoning step in its textual explanation,” scoring intermediate objects on the reasoning path rather than only the final target. Where perturbation tests causal dependence from the outside, VRT-style scoring checks whether the trace’s internal structure is grounded at all. For teams that can emit structured traces, the two are complementary: score the trace for coverage, then perturb to test whether coverage is causal.

What the evidence cannot settle

Three limitations should travel with any use of this checklist. The anchor result is an unreplicated single-team preprint, and its measurements cover two 7B-scale models, DeepEyes and Pixel-Reasoner, on V* Bench and HR-Bench 4K/8K; the two-level framing and the test designs generalize, but how unfaithful thinking-with-images models beyond those two, or at larger scales, actually are remains untested. The companion perturbation results, PDB-identifier shortcuts, lazy-grounding distractors, reasoning rigidity, come from text-heavy domains, and transfer of their effect sizes to document, chart and screenshot agents is assumed, not measured. And because modality control is cheap to train, every number in this literature has a shelf life.

My recommendation, conditioned on those limits: treat the visual trace as an audit surface, never as a grounding guarantee. Before a thinking-with-images feature ships, run image-swap tests to license input-level claims, run redaction and factual-distractor tests to license reasoning-level claims, and write the customer-facing language from the highest level that passed, nothing more. If a lost audit trail or an overclaimed demo would put a customer’s work at risk, the perturbation suite matters more than the demo’s persuasive zoom. The evidence here cannot tell you how often your agent fakes it; it can tell you exactly how to find out.

Frequently Asked Questions

What are the two checks to verify a VLM agent used an image?

First, swap or edit the image and see whether the answer changes; that tests whether the model used the image at all. Second, redact or perturb the evidence the agent’s tools returned and see whether the reasoning steps and the answer move; that tests whether retrieved visual content was actually integrated into the reasoning.

Why are factual distractors better than corrupted pixels for testing grounding?

A blurred chart is trivially detectable; a model can learn to flag it or simply fail safe. A chart that is real, well-formed, and irrelevant, retrieved by your own pipeline, factual in every pixel, is the harder test of whether the agent discriminates evidence by relevance or consumes whatever arrives.

What should teams count when auditing visual traces?

When you audit, count distinct visual evidence acquired per query, not calls issued. And when a demo shows an agent making repeated zooms, the honest question is not “look how thoroughly it inspected the chart” but “why did the pipeline need to ask four times?”