VLMs as Robot Safety Monitors: Where Collision Detection Still Fails

If you are deciding whether a vision-language model can carry collision-risk checks in a robot safety case, the measured answer right now is no: treat VLM contact judgments as advisory context, and keep certification in explicit geometric and physics-based layers until someone scores VLM contact detection on your real recordings. The sharpest evidence is TouchSafeBench, a physics-grounded benchmark released in 2026 that tested current VLMs on collision grounding and found that “the best Macro-F1 on either task is 56%, depth inputs are not automatically transformed into robot-body collision evidence, and robot–scene contact is consistently harder than human-contact risk.” An expert human reaches 74% on the same clips in the paper’s human reference, an 18-point gap between the best model and a human expert looking at identical footage, on the exact subtask a safety monitor exists to perform.

The practical consequence: a fluent captioner that describes a scene convincingly is not evidence that anyone, or anything, is checking whether the robot’s body touched the world. The burden of the safety argument stays with explicit collision-grounding representations, companion sensing, and humans, and any vendor pitch that routes contact-state sign-off through a VLM needs a scored evaluation on your own data before it enters the safety case.

What TouchSafeBench actually measured

TouchSafeBench is worth describing precisely, because its strengths and its limits both shape what the 56% number means. Per the paper: “We introduce TouchSafeBench, a physics-grounded benchmark for evaluating collision grounding in vision-language models (VLMs). Built in Habitat 3.0, TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social navigation and social rearrangement, with synchronized multi-view RGB-D, top-down maps, and simulator contact labels.”

The substrate matters. Habitat 3.0, described in “A Co-Habitat for Humans, Avatars and Robots”, provides accurate humanoid simulation, human-in-the-loop infrastructure via mouse/keyboard or VR, and the two collaborative tasks TouchSafeBench reuses: Social Navigation and Social Rearrangement. These are the deployment-facing settings, a robot moving through space shared with people, and a robot manipulating objects around people. Because the simulator knows the true contact state of every body in every frame, the benchmark can score a model’s claim of contact against ground truth rather than against another annotator’s judgment.

Two protocol details deserve attention before anyone quotes the headline number in either direction. First, read the metric as a per-class lens rather than a single score. The standardized evaluation is balanced, “450 balanced clips per task (150 per class),” and the paper reports raw accuracy alongside Macro-F1; averaging per-class F1 is what keeps a strong aggregate from hiding a weak class, and it is how the robot-scene blind spot documented below shows up in the numbers at all. When the paper says “its sampling is asymmetric by construction,” it is describing something else: Task 2 warning horizons differ by class, and only scene collision has a post-contact subtype because human contact ends an episode. Second, the paper is explicit that “TouchSafeBench is simulated, uses fixed Habitat tasks and humanoid appearances, and evaluates short clips rather than closed-loop intervention.” The 56% is a clip-level judgment score, not a measure of whether a VLM-driven system would actually stop a robot in time.

Where contact detection breaks first

The two failure loci the benchmark names are more actionable than the headline gap, because they tell you which checks to pull out of a VLM’s hands first.

The first is asymmetry by contact type: “robot–scene contact is consistently harder than human-contact risk.” Detecting that a robot has brushed a shelf, a wall, or a fixture is harder for these models than detecting that a human is at risk. This is the inverse of what a deployer might hope, since robot-scene contact is precisely the case where no human yelps, no one flinches, and the only witness is the monitoring system. A VLM that performs acceptably on “is the person’s arm near the gripper” may still be unreliable on “did the elbow clip the doorframe on that turn,” and the benchmark says this gap is consistent across the models evaluated, not an artifact of one weak model.

The second is the depth failure: “depth inputs are not automatically transformed into robot-body collision evidence.” This one defuses a common assumption in integration discussions. Giving a VLM synchronized RGB-D does not mean the model reasons about the robot’s body occupying a volume in that depth field. The model receives geometric information; the measured result says it does not reliably convert that information into a judgment about the robot’s own contact state. Vision fluency and physical accountability are separate capabilities, and the second does not come free with the first.

Which subtasks a VLM can plausibly hold

The evidence does not support a binary “VLMs are useless for safety” conclusion, and it does not support “VLMs are ready to monitor.” It supports a split. The measured failures are grounding failures, not vision failures: the models can describe scenes, and the deficiency appears when the task requires binding that description to the robot’s body and its contact state. That suggests the following allocation, which is my inference from the TouchSafeBench results plus the companion-sensing evidence below, not a result the paper itself states.

The last row is where the argument lands. A safety case is a documented claim that a system is acceptably safe, with evidence. A component measured at 56% Macro-F1 against a 74% human baseline, on the benchmark built to test exactly this capability, cannot be the evidence. It can be context around the evidence.

Why model confidence will not catch the failures

One natural response to a 56% score is to gate on confidence: let the VLM monitor, but escalate when it is unsure. Independent evidence from the VLM reliability literature undercuts that plan. In a study of medical vision-language models, the authors found a per-sample failure mode that evades exactly this defense: “Critically, Dangerous samples exhibit high accuracy (up to 99.6%) and low entropy, making them invisible to standard confidence-based screening.” The same work reports that flip rate negatively correlates with the Dangerous fraction (r=−0.89, n=10), meaning models that look more stable under rephrasing can be the ones harboring more of these confidently-wrong samples.

That result is from medical VLMs, not robot monitors, so treat it as analogical evidence about a model-class behavior, not a robot measurement. But the mechanism it documents, high-confidence errors that confidence thresholds cannot see, is the specific mechanism a confidence-gated safety filter assumes away. If your safety architecture includes a line that says “the VLM flags it when uncertain,” the burden is on you to show your model’s failures are uncertain. The published evidence points the other way.

What companion sensing buys, and what it costs

The geometric alternative has its own measured envelope, and honesty about it strengthens the case for layering rather than for any single layer. Robot Safety Monitoring using Programmable Light Curtains reports per-class detection rates over 20 trials each, from tests that included throwing objects into the robot’s workspace (Table II):

• Human walking: 20/20 (100%)
• Cap: 20/20 (100%)
• Paper ball: 18/20 (90%)
• LEGO 4x2 block: 15/20 (75%)

A light curtain is excellent at catching a person stepping into a cell and measurably worse at catching a small rigid object. That is not an argument against geometric sensing; a 100% rate on the intrusion class most likely to cause serious injury is exactly what you want from that layer. It is an argument that “geometric” does not mean “solved,” and that the layered configuration, VLM advisory, geometry certified, sensing retained, is what the evidence supports rather than a slogan.

The other bound on every layer is sim-to-real transfer. A benchmarking paper on robot policy evaluation for sim-to-real transfer notes that “changes in lighting and camera poses causes model success rate to degrade between 30–50%.” That figure comes from cited work on robot models under visual perturbation, not from TouchSafeBench itself, but it disciplines how much trust any simulator-scored monitor deserves in the field: a score earned in Habitat’s lighting with Habitat’s camera poses is not a score earned on your floor. And if anything processes the video before your monitor sees it, surveillance-video evaluation of vision-LLMs found that “accuracy generally drops by 2–11 pp with privacy, while FP rates often rise. VideoLLaMA-3 shows FP reductions under GAN filters.” Privacy filtering upstream of a safety monitor is not neutral plumbing; it moves both measured error rates.

For contrast on what explicit geometric assessment looks like when it is done as geometry, a marine route safety framework (GARSA) “assigned safety assessment values of 0.6 and 135 to two intra-port transit routes of 7,258 m and 7,845 m, respectively, with the higher value indicating a safer route.” Different domain, but the pattern transfers: numeric, comparable values derived from spatial analysis, which two engineers can argue about on the merits. That is the kind of artifact a safety case can hold.

A probe protocol before any VLM contact judgment enters the safety case

The field-level context makes this protocol necessary rather than optional. A survey of safety in embodied AI states it plainly: “The field currently lacks shared abstractions for thinking about safety across embodiments. There are no widely adopted taxonomies of failure that cut across platforms, nor common stress-testing protocols analogous to standard benchmarks in perception or language.” There is no certification regime that lets you inherit someone else’s VLM safety claim. You have to build the evidence yourself. A reasonable sequence:

1. Scored real-recording evaluation. Record your actual cells, label contact states against ground truth (sensor logs, physical inspection, human annotation with adjudication), and score the VLM with Macro-F1 and per-class F1 rather than raw accuracy alone, because contact events are rare by design and a majority-class answer can look accurate while missing every real contact. The driving domain shows what a scored naturalistic-event structure looks like: ScVLM evaluates safety-event understanding on SHRP 2 naturalistic data with “1,063 crashes, 774 tire strikes, 6,782 near-crashes, and 8,497 normal driving segments,” each event a 30-second front-view clip, classified into 16 conflict types. For robot contact, the recording and labeling work is yours.
2. Class-balance audit. Contact events are rare by design. Report per-class precision and recall, and be suspicious of any vendor number quoted as raw accuracy on an unspecified class distribution.
3. Split robot-scene from human-contact scoring. TouchSafeBench’s consistent asymmetry means an aggregate score can hide the class where the model is weakest and where no human witness compensates.
4. Perturb the inputs. Vary lighting and camera pose within your operating envelope, and measure degradation rather than assuming the lab score transfers. If privacy filters or compression sit upstream, measure through them, given the 2–11 pp accuracy drops and rising false positives measured in adjacent video work.
5. Attack the confidence gate. Before relying on escalation-on-uncertainty, look for high-accuracy, low-entropy failures in your own evaluation set. If they exist, and the medical-VLM evidence suggests they will, confidence screening is decorative, not a filter.
6. Closed-loop test, not just clips. TouchSafeBench evaluates short clips rather than closed-loop intervention. A monitor that scores well on clips can still be too slow or too unstable to interrupt a trajectory. One constructive pattern: H2RBench builds simulation environments “reconstructed from a real-world scene via a real-to-sim pipeline.” Inverting that pattern, reconstructing your cell in simulation to generate labeled contact variants of your real recordings, gives you probe coverage of configurations you cannot safely stage with hardware.

I would treat step one as a gate, not a formality. If a vendor’s safety monitoring claim cannot survive a scored evaluation on a few hundred of your own labeled clips, it was a task-definition claim resting on different class balances than your deployment, not evidence about your system.

What this evidence does not establish

The limitations cut both ways, and the counterweight deserves the same precision as the verdict. Every headline number here comes from 2,940 simulated indoor episodes with fixed Habitat tasks, fixed humanoid appearances, short clips, and asymmetric sampling. Real-world transfer is scored only on a contact-free binary judgment: the paper’s replication on THÖR-MAGNI recordings, with motion-capture ground truth, found GPT-5.5 at chance on current-state recognition (52.0 accuracy) and at 66.0 on early warning, but the recordings contain no physical contact, so contact-class field accuracy is unmeasured. That means the 56% ceiling may not bound real deployment performance in either direction: a model could do worse on real sensor noise, or a system tuned for a narrow cell could do better than a general model on a general benchmark. The adjacent-domain numbers, the 99.6% confident-error accuracy from medical VLMs, the 2–11 pp privacy-filter costs from surveillance video, the 30–50% lighting degradation from robot policy benchmarks, the light-curtain rates, are each measured in their own setting. They bound the shape of the problem; they do not measure your robot.

None of the evidence here establishes closed-loop intervention performance, latency or compute costs, any specific vendor system’s behavior, or positioning relative to ISO 10218 or ISO/TS 15066 requirements. If your safety case must cite those standards, none of this substitutes for that analysis.

The decision the evidence does support is specific. Keep collision-risk certification in explicit geometric and physics-based safety layers. Use VLM outputs as advisory context for human reviewers, where their descriptive strength helps and their grounding failures cost attention rather than contact. Hold human-contact and robot-scene contact checks in companion sensing or simulator-grounded geometry, knowing the light-curtain envelope degrades on small fast objects. And require a scored real-recording evaluation, with Macro-F1, split contact classes, perturbed inputs, and a closed-loop test, before any VLM contact judgment enters the safety case. The TouchSafeBench result is not that VLMs are bad at vision. It is that collision grounding is a separate, measured capability, and on that capability the best model tested is 18 points behind the human it would replace.

Frequently Asked Questions

What is the best Macro-F1 score for VLMs on collision grounding in TouchSafeBench?

the best Macro-F1 on either task is 56%, depth inputs are not automatically transformed into robot-body collision evidence, and robot–scene contact is consistently harder than human-contact risk.

How does human expert performance compare to the best VLM on the same clips?

An expert human reaches 74% on the same clips in the paper’s human reference, an 18-point gap between the best model and a human expert looking at identical footage, on the exact subtask a safety monitor exists to perform.

What are the detection rates for light curtains on different object types?

• Human walking: 20/20 (100%)
• Cap: 20/20 (100%)
• Paper ball: 18/20 (90%)
• LEGO 4x2 block: 15/20 (75%)