How to Read LLM Safety Evidence in Hausa Before You Ship

If your launch review accepts a vendor safety report written in English as evidence that an assistant or moderation classifier is safe in Hausa, Yoruba, or Kiswahili, you are shipping on an assumption the current evidence does not support. A fresh, unreplicated, single-team preprint from researchers affiliated with AIMS Cameroon and Bayero University Kano, Aligning Large Language Models for Safety in African Languages: A Case Study of Hausa, tested alignment configurations on one language spoken by over 200 million people (arXiv 2610.37245) and concluded that “no configuration we tested is safe, helpful, and fluent in Hausa.” That is one study of one language, and it is lab-based, not a deployment measurement. But it gives product, trust-and-safety, and localization leads something they did not have before: a concrete worked example of how to read per-language safety evidence, and of how easily the easiest metric misleads.

For a launch review, that case study and the surrounding research resolve into a short protocol: what evidence to require per language, which translated benchmarks still measure something, when native-speaker review is unavoidable, and how to write these requirements into procurement so a vendor’s “safe AI” claim has to survive contact with the languages your users actually speak.

What the Hausa study actually measured

The framing the authors give is blunt: large language models “are increasingly used by Africans as a personal assistant, yet the safety alignment techniques that govern their behavior are developed and validated almost exclusively on English and other high-resource languages” (arXiv 2610.37245). Their Hausa case study compared supervised fine-tuning (SFT) and direct preference optimization (DPO), the two standard alignment training recipes, and found they improve different things:

“SFT boosts both fluency and coverage of refusals, while DPO boosts refusal precision, but does so by optimizing a relative preference signal that provides no incentives for fluency; no configuration we tested is safe, helpful, and fluent in Hausa.”

Read that against the ceiling effect the paper reports. Refusal rate was already 1.000 for the untuned base model, whose refusals were almost uniformly low quality, and it saturated at 1.000 for every configuration, so neither technique bought frequency of refusal. In the authors’ summary, “SFT teaches the model how to refuse fluently in Hausa, while DPO teaches it when to refuse more precisely.” Concretely, DPO addressed the over-refusal seen under SFT, bringing the false-refusal rate on benign prompts from 0.405 down to 0.110, and paid for it with refusal quality that fell back near the untuned baseline. No configuration achieved all three properties at once. The trade-off is not hypothetical; it is the measured outcome of the two standard alignment recipes.

The authors’ general conclusion is the one to keep on a sticky note: “current alignment techniques teach models when to refuse more readily than how to do so fluently in underrepresented languages, and that refusal-rate metrics alone are an unreliable signal of safety in low-resource settings.”

Why refusal rates are the wrong headline number

A refusal rate is the easiest safety metric for a vendor to quote per language: feed the model harmful prompts in Hausa, count how often it declines, report a percentage. The Hausa study’s central warning is that this number can look reassuring while the underlying behavior is not. A model can refuse at a high rate and do it in broken Hausa that a user cannot parse, or refuse benign requests because its refusal trigger misfires outside English, or refuse in a way that technically counts as a refusal but communicates nothing useful about what happened or what the user can do instead.

There is a deeper reason to be skeptical of refusal as a single reported scalar. Interpretability work on the geometry of refusal found, “contrary to prior work, we uncover multiple independent directions and even multi-dimensional concept cones that mediate refusal” (The Geometry of Refusal in Large Language Models). Refusal is not one switch inside the model that training flips; it is a set of separable mechanisms. That matters for buyers because it means a vendor can tune or report one aggregate refusal number without the underlying behaviors (refusing the right categories, refusing fluently, not over-refusing) moving together. Where alignment data for a language is thin, the Hausa evidence shows the aggregate number simply stops tracking refusal quality.

So the first checklist item is not “does the model refuse in Hausa” but “does the vendor report refusal quality jointly with fluency, measured in Hausa, by someone who can judge Hausa.”

Translated benchmarks: what they still measure, and what they inherit

The most common way multilingual safety benchmarks get built is translation. XSafety, the clearest documented example, was constructed by gathering “several well-established monolingual safety benchmarks covering 14 types of safety issues” and employing “professional translators to translate them, resulting in a multilingual benchmark in 10 languages” (All Languages Matter: On the Multilingual Safety of LLMs), 2,800 instances in total.

A professionally translated benchmark is real evidence. It tells you whether the model’s safety behavior transfers to prompts phrased in the target language, against a defined harm taxonomy. What it cannot tell you is whether that taxonomy matches the harms your users actually face. A translated benchmark inherits its source-language taxonomy, and XSafety’s source is largely Chinese, not English: 13 of its 14 safety issue types come from a Chinese LLM safety benchmark, with only Commonsense Safety drawn from an English set. The builders did ask translators to adapt sentences to each target culture, but adapting sentences does not add categories. Harm categories that are specific to a market, whether local political sensitivities, region-specific fraud patterns, or culturally specific slurs and harassment registers, are only covered if they happened to be in the source taxonomy. And XSafety’s ten languages are “the ten most widely-used languages,” which is a different set from the languages of Northern Nigeria, Niger, and Ghana.

For African languages specifically, the dedicated resources are thin. The systematic review LLM Safety Alignment in Low-Resource Languages, which shares its first authors with the Hausa case study, notes that “benchmark resources for African languages remain limited,” naming LSR, which evaluates cross-lingual refusal degradation in West African languages including Yoruba, Hausa, Igbo, and Igala, and UbuntuGuard, described as the first African policy-based safety benchmark. The Hausa study’s related work adds AfriGuard, “a culturally-informed dataset comprising ten African languages, including Hausa, for safety intent classification, harm classification, and refusal detection” (arXiv 2610.37245). That so short a list is what the field has to offer tells you both that usable starting points exist and that the shelf is close to empty.

The practical reading: a translated-benchmark score is evidence about the source-language harm taxonomy expressed in the target language. Treat it as a floor, not a certification.

Jailbreak numbers travel with their attack method

Here the findings pull in opposite directions, and how you handle that is itself a launch-review skill.

The systematic review reports that “low-resource languages exhibit cross-lingual jailbreak failure rates more than seven times higher than those observed in high-resource baselines” (arXiv 2608.14626v1). Read as a headline, that says low-resource languages are seven-plus times more dangerous.

A direct attack study, Multilingual jailbreaking of LLMs using low-resource languages, found something that sounds nearly opposite: “the overall jailbreak rate is relatively low with only the DeepSeek model exceeding 10%, namely, 17.4% for isiZulu and 13.0% for Kiswahili” (arXiv 2605.18239v1), and the authors conclude that “simply translating prompts into low-resource languages is insufficient to bypass current LLM safety guardrails.”

These are not necessarily contradictory measurements; they are measurements of different attacks against different model sets, aggregated differently. A review-level failure rate across many studies, attack families, and models is not comparable to a single-turn translation attack against a specific set of recent models. The lesson for launch review is mechanical: never accept a jailbreak percentage without its attack method, model set, and language set attached. A vendor quoting the low single-turn numbers is showing you resistance to an attack that recent models mostly withstand anyway. A vendor quoting the seven-times figure is quoting a review aggregate. Neither one, alone, describes your deployment. What you want is in-language red-team results, in your target language, with the attack methods enumerated, run against the model version you are actually buying.

The per-language safety evidence checklist

The table below is the working artifact. Each row is a class of evidence a vendor or internal team might present, what it genuinely measures, the failure mode the research documents for low-resource languages, and how to treat it in a launch review.

Where native-speaker review is unavoidable

Every row in that table except one can be gamed, approximated, or inherited from English. The exception is fluency-qualified refusal. The Hausa result is precisely that the two standard alignment techniques split this property: SFT bought fluency, DPO bought refusal precision, and neither delivered both. The closest automated proxy is the study’s own LLM-judged quality score, and its authors flag the limit themselves: the judgments “are done by a single LLM judge (Gemini) and not by annotation by many native speakers,” and “the automatic judging of fluency and appropriateness of a low-resource language is vulnerable to its own mistakes” (arXiv 2610.37245). Their stated next step is evaluation by native Hausa speakers.

For a launch, this means native-speaker evaluation is not a nice-to-have audit step; it is the dependable measurement for the dimension the Hausa study found failing. Scope it narrowly if budget is tight: refusal and escalation messages in the top harm categories for your product, reviewed by speakers from the region you are serving, before launch and after each model version change. That is a bounded, fundable amount of work. Skipping it means the one property the anchor study measured as broken goes unmeasured in your product too.

Running this in launch review and procurement

The operational move is to convert the checklist into disclosure requirements. In launch review, treat English-only safety validation as insufficient for any African-language market and require, per target language: refusal quality measured jointly with fluency, in-language jailbreak results with attack methods named, harm-taxonomy coverage that distinguishes translated categories from target-language-native ones, and native-speaker review sign-off.

In procurement, the same items become contract language: the vendor reports refusal, fluency, and harm-taxonomy coverage per target language, for the model version being purchased, or the “safe AI” claim in the sales material is read as English-scoped. This is a request for disclosure, not an accusation. Nothing in the evidence certifies any named commercial model as safe or unsafe in Hausa or any other language, and it includes no per-language refusal-quality or fluency measurements for the closed commercial assistants a buyer would purchase; the direct attack study reports harmful-response rates for those systems, which is a different quantity. The Hausa study does measure refusal quality and fluency in Hausa, but for its open-weight base model, Qwen2.5-7B-Instruct, and the authors’ fine-tunes of it. What the evidence establishes is that the burden of proof currently sits in the wrong place: vendors validate in English, and users in Kano or Accra absorb the difference.

One judgment I would stand behind: if a vendor cannot produce any per-language safety evidence for a language you are launching in, the honest options are to fund the native-speaker evaluation yourself, gate the launch to lower-harm features, or delay. Accepting an English report as coverage is not a fourth option; it is the first option with the measurement skipped.

What one Hausa preprint cannot tell you yet

Hold the limits clearly. The anchor is a single-team, unreplicated, lab-based case study of one language. Its SFT/DPO findings describe the configurations that team tested, not every model or training pipeline. The seven-times jailbreak figure is a review aggregate whose methods differ from the direct attack study that found single-turn translation attacks mostly ineffective; both are method-bound. The African benchmark landscape is a few named resources, the broadest of which, AfriGuard, spans ten African languages. Per-language refusal-quality and fluency measurements for closed commercial assistants do not appear in this evidence base; the direct attack study reports harmful-response rates for named models, not how fluently or appropriately they refuse. The Hausa study’s own quality scores cover its open-weight Qwen2.5-7B-Instruct base and the authors’ fine-tunes, not the closed assistants a buyer would purchase. What survives those limits is the reading skill and the ask. Refusal rates alone mislead in low-resource settings. Translated benchmarks inherit their source-language harm taxonomy. Jailbreak numbers mean nothing without their attack method. Fluency-qualified refusal requires a human who speaks the language. And a vendor safety claim that cannot name its languages is an English claim. Those conclusions rest on how the methods work, not on one preprint. The Hausa study’s contribution is to make the gap concrete enough to write into a launch checklist, which is exactly where it belongs.

Frequently Asked Questions

What is the main finding of the Hausa safety study regarding alignment techniques?

SFT boosts both fluency and coverage of refusals, while DPO boosts refusal precision, but does so by optimizing a relative preference signal that provides no incentives for fluency; no configuration we tested is safe, helpful, and fluent in Hausa.

Why are refusal rates considered an unreliable safety metric in low-resource languages?

current alignment techniques teach models when to refuse more readily than how to do so fluently in underrepresented languages, and that refusal-rate metrics alone are an unreliable signal of safety in low-resource settings.

What specific limitation do the authors note about their automated fluency judgments?

the judgments “are done by a single LLM judge (Gemini) and not by annotation by many native speakers,” and “the automatic judging of fluency and appropriateness of a low-resource language is vulnerable to its own mistakes”