groundy
Models & Research

Do Multimodal LLMs See Like Humans? What HumanSightBench Tests

HumanSightBench tests perceptual gaps in multimodal LLMs using human-easy visual questions. Author-reported scores show models struggle, but the benchmark does not predict or

Published 4 references
A skeptical translucent green resin dinosaur reaches toward two overlapping ivory bird silhouettes. Bubbles, scuffs and articulated joints give the miniature a handmade texture, with hard shadows across a warm ivory background.
On this page9 sections

Use HumanSightBench to investigate a perceptual gap, not to select a production model by itself. The unreviewed HumanSightBench preprint, observed on arXiv on November 14, 2026, describes 2,876 short-answer visual questions designed to be easy for people and hard for models. Its authors report a best pass@3 score of 65.6% across eight evaluated multimodal LLMs, with no evaluated model exceeding 27.5% on Counting. These are author-reported, abstract-level results, and the supplied evidence does not identify the eight models.

That distinction matters for deployment. A team choosing a model for OCR, interface understanding, or visual inspection usually needs correct outputs on its own workload. HumanSightBench instead asks whether models struggle on visual questions constructed around human perceptual abilities. It can expose a useful failure class, but the supplied evidence does not establish that its scores predict product-task correctness.

What HumanSightBench actually tests

According to the HumanSightBench abstract, the benchmark contains 2,876 visual questions with short answers, organized into 7 top-level perceptual categories and 21 subcategories. Its stated construction guideline is unusually direct:

Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation.

“Human-easy” should not be read as “representative of every human visual task.” It describes the intended difficulty of these benchmark items. Nor does “model-hard” establish difficulty for untested models or future versions. It describes the authors’ evaluation of the eight unnamed systems.

Every included item must pass five inclusion gates:

  1. Question clarity
  2. Answer uniqueness
  3. Visual grounding
  4. Human solvability
  5. Text-only shortcut filtering

Those gates address common evaluation problems. A visually grounded question should require the image rather than background knowledge alone, while text-only shortcut filtering is intended to remove questions that can be answered without looking. Answer uniqueness and human solvability are especially important when an incorrect short answer is treated as evidence of a model failure.

The gate names are useful audit criteria, but the abstract does not supply their detailed implementation. It also does not describe how the human reference data were collected, how many people participated, or how agreement among people was handled. The available evidence therefore supports saying that human solvability was required, not that the benchmark provides a complete measurement of human perception.

Within this setup, errors are incorrect answers on human-validated visual tasks using the evaluated interface. That is a narrower definition than “seeing incorrectly,” and the difference becomes important when interpreting the scores.

What the reported scores support

The authors report that the best pass@3 result is 65.6% across the eight evaluated models. On Counting, they report a much lower ceiling: no model exceeds 27.5%. Both figures come from the preprint’s abstract, not an independent reproduction.

The narrow conclusion is that, under the authors’ setup, even the strongest reported result leaves many human-easy questions unanswered correctly, and Counting appears particularly difficult. That supports a cautious answer to the title’s question: on these tasks, the evaluated models do not match the benchmark’s human-easy premise.

It does not support naming a winning model. The supplied material omits all eight model identities and their individual scores. Nor does 65.6% represent a typical model score; it is explicitly the best reported pass@3 result. The 27.5% figure similarly applies to Counting, not to the whole benchmark.

The human side of the comparison also needs care. Human-easy-by-construction is a design requirement, not a reported human accuracy percentage. Without the full human-data methodology and results, the size and nature of a human-model gap cannot be decomposed further.

A low score is not necessarily a pure perception failure

The strongest caution comes from the benchmark’s own authors:

Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers.

This prevents a simple interpretation in which every wrong answer demonstrates that a model failed to see something. A single aggregate score can combine problems with visual processing, language interpretation, knowledge, and answer specification. The five inclusion gates are designed to control some of these issues, but the authors still identify them as possible error sources.

For a model-selection team, the practical response is not to discard the benchmark. It is to avoid treating its aggregate score as a complete diagnosis. A low result tells you where to inspect failures; it does not, by itself, identify the component that should be fixed or prove that another model will perform better on a product task.

That limitation also changes how HumanSightBench should influence optimization. Improving its score could involve changes unrelated to the visual behavior you care about. Item-level error analysis on representative failures is more informative than treating the headline number as a perception-only metric.

Human alignment and task correctness use different scoreboards

HumanSightBench sits among several benchmarks that ask different questions and produce scores that cannot be compared as if they were percentages of the same underlying ability.

BenchmarkEvaluation target and scoringAuthor-reported evidenceDecision it can inform
HumanSightBenchShort-answer accuracy on human-easy perceptual questions, summarized with pass@3Best pass@3 of 65.6% across eight models; Counting ceiling of 27.5%A diagnostic for gaps on the included perceptual tasks, not a standalone product selector
CogIP-BenchCorrelation between model outputs and human cognitive judgmentsNearly 0 correlation with humans on image memorability; average Spearman correlation below 0.5 on the other reported cognitive dimensionsWhether model judgments track human response patterns, rather than whether task answers are correct
Visual Aesthetic BenchmarkIdentification of the best and worst images under reordered candidate presentationsStrongest system succeeds on 26.5% of tasks across three random permutationsA specific aesthetic-judgment evaluation, not a general perception or task-completion score
MAGWeb-agent action and guide generation, evaluated through task performanceBest frontier configuration completes 37.4% of tasks; best 9B agent completes 13.2%Interface-agent task completion, rather than similarity to human perception

CogIP-Bench makes the scoring distinction especially clear. Its authors collected 800 training datapoints and 120 test datapoints for each cognition property. They report that all evaluated MLLMs, including open-source and API-based models, showed nearly 0 correlation with humans on memorability, while average Spearman correlations on other cognitive dimensions remained below 0.5. A correlation measures whether responses move with human judgments. It does not measure short-answer correctness, and a high correlation would not automatically make an output factually right.

The Visual Aesthetic Benchmark evaluates a different kind of alignment. It contains 400 tasks and 1,195 images spanning fine art, photography, and illustration. Its strongest reported system identified both the best and worst image correctly across three random candidate-order permutations on only 26.5% of tasks. That result concerns aesthetic judgment under the benchmark’s protocol, not OCR accuracy, counting ability, or successful interface operation.

MAG supplies the contrasting task-correctness axis. Built from success demonstrations released by OpAgent, it covers trajectories for 581 of WebArena’s 812 tasks and evaluates multimodal action and guide generation. Its authors report task completion of 37.4% for the strongest frontier configuration and 13.2% for the best 9B agent. Those figures should not be numerically compared with HumanSightBench’s 65.6%: the tasks, outputs, and success criteria differ.

When should human-likeness influence model selection?

Human-likeness is useful when it is part of the product requirement. If you need a model’s judgments to track human responses, a human-reference benchmark can measure something that ground-truth task accuracy misses. HumanSightBench, CogIP-Bench, and VAB each offer a different version of that target, so the choice still depends on whether you need perceptual answers, cognitive correlations, or aesthetic judgments.

Human-likeness is the wrong proxy when the requirement is simply to be correct. For OCR, the desired output is normally the right transcription; for an interface agent, it is the required action or completed state; for inspection, it is the correct product-specific decision. None of the supplied HumanSightBench evidence establishes fitness for those workloads. A model could score relatively well on perceptual alignment and still fail the deployment task, or perform the task correctly without matching human response patterns.

There is also an unresolved question about what “alignment” should reward. The supplied HumanSightBench abstract does not establish whether any scoring component rewards agreement with human error patterns. If a benchmark did reward that agreement, optimizing it could require reproducing human mistakes. That might be appropriate for a hypothetical system intended to simulate human responses, but undesirable for a system whose job is to catch defects or transcribe text accurately. Teams should inspect the full scoring methodology and decide which reference they want: typical human behavior, human consensus, or independently established ground truth.

What an internal evaluation still needs to measure

For a deployment decision, I would use HumanSightBench as a supplementary diagnostic and build the selection evidence around the product requirement. That means:

  • Define correctness for the actual output. Use task-appropriate targets for transcription, interface actions, inspection decisions, or other required responses. Do not substitute a perceptual-similarity score for those targets.
  • Test representative product inputs and failure conditions. The supplied research does not show that HumanSightBench’s human-easy items cover the difficulty or distribution of your workload.
  • Match the scoring protocol to deployment. If the product gets one attempt, report that setting separately from any multi-attempt diagnostic rather than relying only on pass@3.
  • Review errors by cause. The HumanSightBench authors’ list of perception, language priors, background knowledge, ambiguity, and underspecified answers provides a useful starting taxonomy for internal failure analysis.
  • Keep human alignment separate when it is a real requirement. Measure it explicitly, and specify whether agreement with human mistakes is desired. Do not assume that correctness and human-likeness always improve together.

Before citing HumanSightBench scores externally or using them to rank candidates, verify the full paper and released artifacts. The immediate checks are the eight model identities, human-reference-data collection, detailed pass@3 mechanics, category-level results, and whether the reported numbers can be reproduced. Until then, the defensible decision is bounded: use the author-reported results to justify investigating perceptual failures, especially Counting, but make the deployment choice with task-correctness benchmarks and internal evaluations that measure the behavior your product actually requires.

Frequently Asked Questions

What is the best pass@3 score reported for HumanSightBench?

Its authors report a best pass@3 score of 65.6% across eight evaluated multimodal LLMs, with no evaluated model exceeding 27.5% on Counting.

What error sources do the HumanSightBench authors identify?

Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. HumanSightBench preprintarxiv.orgAccessed
  2. CogIP-Bencharxiv.orgAccessed
  3. Visual Aesthetic Benchmarkarxiv.orgAccessed
  4. MAGarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy