How to Evaluate an LLM for Mental Health Conversations: MentalHealthBench

OpenAI released MentalHealthBench in September 2026: an open benchmark of 1,215 synthetic mental health conversations and 5,262 expert-authored rubric criteria, per Unite.AI’s coverage, co-created with more than 80 licensed experts from 22 countries. For teams shipping support bots, companion apps, or wellbeing-adjacent assistants, it is a runnable, open alternative to a refusal-rate audit: OpenAI’s announcement states that most prior evaluations in this domain focused primarily on emergency scenarios and measured success with broad, predefined criteria, leaving a gap in understanding how models perform across the full range of mental health conversations. The catch: every published score is vendor-reported, the grader is OpenAI’s own model, and a score measures rubric alignment, not clinical safety.

What a refusal-rate audit misses

The standard pre-launch gate for a supportive-conversation feature is some variant of a refusal audit: does the model decline self-harm requests, crisis roleplay, and medical advice it should not give? That gate answers one question, and it answers it on prompts where the harmful request is legible on the first turn.

Real supportive conversations do not look like that. The authors of BiGraph-Diffuse, a counseling-response paper on arXiv, make the structural point well: psychological disclosure is often progressive or defensive rather than direct. A user rarely states the acute problem in message one. A single-turn audit therefore cannot observe the behavior that most distinguishes a useful supportive response from a safe-but-useless one: whether the model asks what is going on before it advises, whether it escalates when a mid-conversation signal turns urgent, whether it gives the user a next step rather than a lecture.

The result of gating on refusals alone is a system that passes launch review and then, in production, produces responses that are technically non-harmful and practically inert: deflect, empathize vaguely, suggest professional help, repeat. Nothing in the refusal rate tells you whether responses seek context, preserve the user’s ability to decide for themselves, or say something actionable. Those are the behaviors MentalHealthBench was built to score.

What the benchmark actually is

Per OpenAI’s announcement, MentalHealthBench measures how AI systems respond in realistic mental health conversations across four headline behaviors: safety, seeking context, preserving user agency, and providing actionable guidance when appropriate. The expert cohort, per OpenAI, spans licensed psychologists and psychiatrists speaking 19 languages across nearly 20 subspecialties. The dataset and rubrics are released openly so outside teams can run their own evaluations. One dating note: OpenAI’s announcement page carries September 8, 2026 metadata, but its results charts are annotated “as of September 23, 2026”, and secondary coverage dates the release September 23; September 2026 is the publication window.

The construction details, reported by Unite.AI’s coverage of the accompanying paper, are what make the thing runnable rather than rhetorical:

1,215 synthetic conversations, per Unite.AI, spanning everyday wellbeing topics through urgent mental health emergencies. More than half exceed five messages, so most tasks are multi-turn.
Deliberate acuity mix (per Unite.AI): 53.5% non-acute, 18.2% high-acuity, 28.3% emergent. The authors are explicit that this mix is constructed for evaluation and does not represent how often such conversations occur in ChatGPT traffic.
Four user profiles (per Unite.AI): adults (68.1%), teens (21.2%), clinicians (5.8%), caregivers (4.9%).
Non-English coverage (per Unite.AI): 105 Spanish, 54 Hindi, 34 Arabic, 29 Portuguese conversations plus additional German, Italian, Persian, Indonesian, Turkish and Chinese conversations. The authors caution that language comparisons are descriptive only, because language cannot be isolated from acuity, topic, or culture in this design.

Each downloadable example carries an identifier, the conversation, rubric items, acuity, user profile, language, and prior-context fields. Seventy tasks (5.8%), per Unite.AI, include prior user context, which is the closest the benchmark comes to testing continuity. Every example also carries a canary string, and OpenAI asks that examples not be posted in plain text, to keep them out of future training corpora.

How expert-alignment scoring works, mechanically

This is where the benchmark differs most from a refusal audit, and it is worth understanding before reading any score.

Each conversation was reviewed by at least three experts in a three-stage process: two clinicians independently authored weighted criteria, a third adjudicated and refined them, and only criteria agreed upon by at least two experts and not contradicted by a third were retained, per the Unite.AI report. Every criterion carries a weight from -10 to +10, with magnitude indicating clinical importance. A response can therefore earn credit for behaviors a refusal audit never sees (asking a follow-up question, offering a concrete next step) and lose it for behaviors a refusal audit cannot detect (over-pathologizing, taking agency away).

Grading is automated: GPT-5.6 Sol at high reasoning effort assesses each model response against the expert criteria, with four independently sampled completions per task, and scores are reported as task-clipped rubric scores. The overall score decomposes into ten expert-defined behavior areas, enumerated by EdTech Innovation Hub: context and assessment, actionable guidance, clinical accuracy, interpretation and reframing, empathy and support, reality testing, urgency calibration, harm avoidance, agency, and communication.

That decomposition is the single most operationally important property of the benchmark, because models with similar overall scores can have different strengths, and different conversation types in your product load on different axes.

Reading the scoreboard, with its calibration anchors

All of the following are vendor-reported results from OpenAI’s own release, graded by an OpenAI model. GPT-6 Astra leads at 57.3% task-clipped, with GPT-6 Sol at 53.9% and Claude Opus 5.5 at 52.4%, per Unite.AI. GPT-6 Luna scored 50.2% and Muse Spark 1.3 47%, per EdTech Innovation Hub. GPT-4o (March 2025) sits at 32.1% and Gemini 2.5 Pro at 29.5%, per Unite.AI, with 95% confidence intervals in the paper. By subset, GPT-6 Astra leads on emergent conversations at 58.3%, while Claude Opus 5.5 leads on teen conversations at 57.0%, per Unite.AI.

Two reference completions calibrate what those percentages mean, and they matter more than the ranking:

Rubric-aware completions, written with the grading rubrics in hand, scored 99.0%, per Unite.AI. The paper describes this as a sanity check on the evaluation’s noise ceiling. A near-perfect score is achievable when you know the criteria.
Clinician-authored completions scored 38.5%, per Unite.AI. OpenAI attributes this largely to clinicians writing short responses as they would in an in-person exchange: a single question, a simple statement. The rubric rewards behaviors chat responses must spell out and therapy sessions do not.

Read together, these anchors say that 57.3%, the top task-clipped score per Unite.AI, is not a quality grade on a 0-100 clinical scale. It is a measure of alignment with a specific written-communication rubric, on which actual clinicians’ typical response style scores lower than the top commercial model. A shipping team should read a low score as “this model does not display the rubric’s preferred behaviors in writing,” not “this model would fail as a therapist,” and should read a high score with the same precision.

One more decomposition worth memorizing: per explainx’s analysis, Claude Opus 5.5 reportedly outscores GPT-6 Astra on context-seeking specifically, 56.9 versus 52.8, despite Astra’s higher overall. The headline number hides the axis your product most depends on. This is a pattern Groundy has seen elsewhere: aggregate benchmark scores routinely conceal per-dimension weaknesses, and a leaderboard rank answers a different question than the deployment asks.

The decision framework: which axis gates which conversation type

The practical output of the benchmark is not the leaderboard; it is a mapping from your product’s conversation mix to the axes that should block a launch. The dataset’s own structure suggests the mapping:

Conversation type in your product	Primary gate axes	Why
Non-acute wellbeing coaching (53.5% of the set)	Actionable guidance, empathy and support, agency	Users here want next steps; agency preservation prevents dependency and over-direction
High-acuity distress (18.2%)	Context and assessment, urgency calibration, reality testing	Misreading acuity is the failure mode; the model must gather context before advising
Emergent/crisis (28.3%)	Harm avoidance, urgency calibration, communication	Refusal-style harm checks still belong here, plus correct escalation
Teen users (21.2%)	Agency, urgency calibration, communication	Reviewed by clinicians with youth expertise; teen persona is system-message only
Caregiver/clinician users (10.7%)	Clinical accuracy, interpretation and reframing	Different standard of guidance than end-user support

Two caveats attach to this table. First, urgency calibration has a reported internal tradeoff: the paper finds tension between performance on emergent and non-acute conversations, and OpenAI says it errs on the side of caution so models handle emergent situations safely. A model tuned to maximize the emergent subset may over-escalate your non-acute traffic, treating a stressful week as a crisis. Gate on both subsets, not the blend. Second, the teen coverage has a structural gap: the teen persona (ages 13-17) is conveyed via a system message, which OpenAI notes is designed to work across providers but may not capture product-specific safeguards. If your product’s teen safety story lives in account-level controls, parental features, or classifier layers, this benchmark does not test them.

Reproducing a run on the open release

Because the dataset, rubric criteria, and grader configuration are public, reproducing a run is bounded work rather than a research project. This is an inference from the release’s structure, not a verified claim that your reproduction will match published numbers. What it requires, per the construction coverage:

Pull the dataset; each example includes conversation, rubric items, acuity, user profile, language, and prior-context fields.
Generate four independently sampled completions per task from your candidate model.
Grade with GPT-5.6 Sol at high reasoning effort against the expert criteria, producing task-clipped rubric scores.
Decompose the result across the ten behavior areas and the acuity/persona subsets relevant to your product; do not stop at the overall score.
Respect the canary-string rule: do not post examples in plain text anywhere a training crawl might reach, or the benchmark’s future signal degrades.

Note the dependence in step 3: the documented grader is OpenAI’s model. Nothing in the packet validates that grader’s fairness when scoring other providers’ outputs, so a reproduced run inherits the same conflict the published numbers carry, plus whatever bias your own pipeline adds.

Cheaper proxies, and where each fails

The benchmark’s three-expert consensus pipeline is exactly what most teams cannot staff: two independent clinician authors and an adjudicator per conversation. The open release transfers that cost to OpenAI for the shared dataset, but any custom conversations you add for your domain need new expert rubrics or the gate is weaker. Two cheaper proxies exist, and both have known failure modes.

LLM-judge pipelines without expert rubrics. The BiGraph-Diffuse authors show the shape of this: multi-judge automatic evaluation plus human evaluation by lay participants and licensed psychologists, with reported two-judge agreement of r=0.981 between Gemini-2.5-flash and GPT-4.1 Mini, per the arXiv paper. That is a single-author-team preprint result and should be treated as a reported claim. Even if it holds, high inter-judge agreement establishes consistency, not clinical validity: the judges agree with each other, which says nothing about whether either agrees with a psychiatrist. Use LLM judges for regression testing between candidate builds, where relative movement is what you need. Do not use them to establish an absolute safety claim.

Refusal-rate audits. Keep them, but only for the harm-avoidance axis in emergent scenarios, which is the one place a binary decline/escalate check genuinely maps to the risk. Everywhere else, they measure the absence of a bad behavior rather than the presence of a good one, and they structurally cannot see context-seeking quality in multi-turn progressive disclosure.

The broader pattern is worth stating: pre-deployment evals measure observed behavior under test conditions, and alignment claims based on them need deployment-time riders. MentalHealthBench is better evidence than a refusal audit, but it is evidence of the same limited kind.

What remains unverified

This section is not boilerplate; each item here changes how much weight a shipping decision can put on the benchmark.

Vendor conflict of interest. The benchmark was written by OpenAI, is graded by an OpenAI model, and ranks OpenAI’s models first, as explainx puts it; that does not make it wrong, but it makes independent replication matter more than usual, and none exists in the evidence packet as of this writing. Treat every named score as a vendor-reported figure, the same standing Groundy gives vendor-reported results on any other benchmark.
Expert alignment is not user helpfulness. A separate user study with 44 adults from 16 countries, limited to non-acute conversations, found user and expert rubrics aligned on just 25.7% of total rubric weight, with 1.0% directly contradictory, per Unite.AI; users emphasized practical next steps and tone, while experts emphasized context-gathering. A high score demonstrates expert-rubric alignment. Whether your users experience the response as helpful is a different measurement.
Not clinical effectiveness. Both OpenAI’s framing and secondary coverage state these are rubric-behavior scores, not measures of clinical outcomes, and ChatGPT is not a substitute for professional care. There is no outcome data in this packet at all.
Structural blind spots. Scoring assesses the final response of a conversation, so sycophancy and multi-week drift across a long user relationship are untested. Product-level safeguards (age gates, classifier layers, account controls) sit outside the system-message persona method. Community skepticism about the expert-selection process exists in the coverage and is not settled by anything in the benchmark itself.

Verdict

Adopt the open release as your pre-launch behavior-quality gate, and demote the refusal-rate audit to the one job it does well: harm avoidance on emergent scenarios. Run the 1,215-conversation set against your candidate model with the documented GPT-5.6 Sol grader and four sampled completions, then gate on the decomposed axes that match your conversation mix, not the overall score. Budget for expert review of any custom conversations you add, and use LLM-judge pipelines for cheap regression testing between builds rather than for absolute claims.

What the benchmark cannot tell you is at least as important as what it can. It cannot tell you whether your responses produce good outcomes for users, whether users find them helpful, whether your product’s safeguard layers work, or whether any published ranking will survive independent replication. As of September 2026, every number attached to MentalHealthBench originates with the vendor that also sells the top-ranked model. Use the rubrics and the pipeline now; treat the leaderboard as provisional until someone outside OpenAI reproduces it.