English refusal rates do not transfer to Hindi, Bengali, Marathi, or Punjabi, at least not on the evidence any team should currently accept. IndicSafeEval, a persuasion-based jailbreak evaluation posted to arXiv on 2026-09-03, reports that model safety behavior varies by language, by persuasive phrasing, and by risk category, which means a refusal rate measured on English prompts is not transfer evidence for Indic-language deployment. Teams shipping into those markets need native-language persuasive probes and per-language baselines of their own.
Everything in this article rests on a single author group’s preprint: IndicSafeEval (arXiv:2609.03781), submitted by Saikat Mondal, v1 dated 2026-09-03 and v2 dated 2026-09-04, with acceptance to Findings of EMNLP 2026 stated by the authors on the arXiv page and not independently confirmed. There is no independent replication, no vendor response, and no critique in the public record as of 2026-09-08. Nothing here is a regulatory requirement or a compliance mandate. What follows is a deployment-audit posture that the preprint argues for, useful on its own terms regardless of how this one paper fares under peer scrutiny.
What is IndicSafeEval, actually?
IndicSafeEval is a persuasion-based jailbreak evaluation framework that crosses ten safety-critical content categories with six human-like persuasive strategies across four Indian languages, producing 7,200 adversarial prompts, according to the paper’s abstract.
The construction matters more than the count. Most jailbreak evaluations probe models with direct harmful queries: ask for the disallowed thing, measure whether the model refuses. Persuasive jailbreaks work differently. They wrap the harmful request in rhetorical framing, appeals to authority, emotional pressure, manufactured urgency, the kinds of phrasing a motivated human would actually use. Crossing that attack style with a language axis produces a grid: ten harm categories, six persuasion strategies, four languages (Hindi, Bengali, Marathi, Punjabi). Ten times six times four is 240 cells per model, and the paper reports 7,200 prompts total, which works out to thirty prompts per cell.
That grid shape is the paper’s real contribution to practice. A single aggregate refusal rate, the number most vendor safety cards report, collapses all three axes into one figure. The IndicSafeEval authors explicitly argue this is a mistake, reporting that some harm categories are “significantly more susceptible to persuasion-based jailbreaks than others.” If category A holds up under persuasion and category B collapses, an aggregate number will report something in between and tell you nothing about either.
The arXiv page lists the paper at 38 pages with 7 figures and 33 tables, and the abstract carries a warning that it contains example data that may be offensive or harmful. The authors state an implementation is available via a linked repository, though the cleaned abstract page does not expose the repository URL, so the code’s existence is author-stated rather than verified here.
What does the preprint actually report, and what does it not?
The preprint reports a black-box evaluation of several open-source LLMs across the language, persuasion, and risk-category axes, and its headline finding is that a model “does not behave equally safely across all languages and prompt styles.” What it does not report, in any publicly accessible summary, is which models scored what.
Read that second sentence again before citing this paper in a planning doc. The abstract evaluated open-source models. It does not claim that closed frontier models were tested. Any sentence of the form “IndicSafeEval shows GPT-class or Claude-class models fail in Hindi” is an extrapolation the paper’s own abstract does not support. The direction of the finding, that safety varies with language and phrasing, is reported. The magnitude, per model, is not present in the abstract text, and no independent group has replicated or challenged the numbers as of 2026-09-08.
The headline qualitative findings are three. First, safety performance depends on the language of the request. Second, it depends on how the request is phrased, specifically on persuasive cues. Third, vulnerability is uneven across risk categories, so aggregate refusal rates conceal per-category gaps. All three come from the author group’s own abstract, and all three are stated at the level of the model population evaluated, not as per-model scorecards.
The evidence gaps are equally concrete. The fetched abstract page contains zero refusal-rate percentages. There is no author block in the cleaned abstract text, only the submitter name. The EMNLP 2026 Findings acceptance appears in the arXiv comments field, which is author-controlled metadata, not conference confirmation. And the evaluation method is black-box, meaning the authors probed model outputs without access to internals, which is the correct posture for a deployment-relevant audit but also means the paper says nothing about why the safety behavior varies, only that it does.
Why would English refusal rates fail to transfer?
English refusal rates fail to transfer because language, persuasive phrasing, and risk category each move safety behavior independently, so a number measured on one point of that grid says little about the rest of it.
The mechanism is not exotic. Safety training data, refusal exemplars, and red-team prompt sets are overwhelmingly English. This follows a pattern older than LLMs: as Wikipedia’s overview of multilingualism notes, software development nearly always uses English due to its status in computing, and commercial software is typically released in English first with multilingual versions derived from the English original. Safety evaluation inherited that default. A model’s refusal behavior in Hindi is, in most training pipelines, a byproduct of whatever cross-lingual generalization happened to occur, not a deliberately trained and measured property.
Persuasive phrasing compounds the problem because it is language-specific at the level of idiom, not vocabulary. The persuasion strategies that work on a person (appeals to family obligation, deference to elders, religious framing, urgency conventions) differ across cultures, and the phrasings that instantiate them differ across languages. Translating an English persuasive jailbreak into Bengali flattens exactly the feature the attack relies on: native persuasive texture. A translated test set measures whether the model refuses awkward translated phrasing. It does not measure whether the model refuses the phrasing a native speaker would actually write.
The risk-category axis is the third independent variable, and the one most likely to surprise teams that have only ever read aggregate safety numbers. The preprint reports that some categories of harmful content are significantly more susceptible to persuasive jailbreaks than others. A deployment team that measures one refusal rate per language is still averaging over this variance. A model can look acceptable on nine categories and be wide open on the tenth, and the tenth might be the one that matters for your product and your market.
There is also a population argument hiding under the technical one. Multilingual speakers are believed to outnumber monolingual speakers worldwide, and in India schoolchildren routinely learn multiple languages depending on where they live. English-only safety evidence is not a reasonable proxy for “most users.” It is a proxy for the users safety teams found cheapest to test.
What must a multilingual deployment audit cover before shipping?
A multilingual deployment audit must cover native-language persuasive-attack probes per language, per-language refusal baselines, and per-category harm taxonomies, because each axis the audit skips is a dimension along which your measured safety number can silently diverge from production behavior.
Concretely, the audit shape IndicSafeEval demonstrates has five components:
| Audit axis | English-only practice | What a multilingual audit requires |
|---|---|---|
| Language coverage | English prompts, sometimes machine-translated variants | Native-authored prompts in each deployment language (Hindi, Bengali, Marathi, Punjabi at minimum for those markets) |
| Attack style | Direct harmful queries | Persuasive jailbreak strategies written by native speakers, per language |
| Risk taxonomy | One aggregate refusal rate | Per-category refusal rates across a harm taxonomy (the preprint uses ten categories) |
| Evidence language | Translated test sets | Native-language probes; translation flattens the persuasive texture being tested |
| Baselines | A single launch-time number | Per-language, per-category baselines tracked across model versions |
Two of these rows deserve emphasis because they are where teams most often cut corners.
The translated-set row is the big one. Machine translation of an English red-team corpus is cheap and feels like coverage. It is not coverage of the thing the preprint measures. The attack mechanism is native-language persuasive phrasing; a translated prompt is, by construction, phrasing no native speaker would produce. If your audit passes on translated prompts, you have learned that the model refuses stilted Bengali, which was never the threat model.
The baselines row matters because safety behavior drifts across model versions. A per-language, per-category baseline measured at launch is the only way to detect that a fine-tune or a system-prompt change degraded refusal rates in Marathi while leaving English untouched. Without the baseline, that regression ships silently.
How should you read a vendor safety card that cites only English benchmarks?
A safety card citing only English benchmarks is evidence about English-language behavior, and should be read as silent on every other language the model will be deployed in.
This is a reading discipline, not an accusation. Vendor safety evaluations are usually competent within their scope. The scope is the problem. When a card reports a refusal rate on an English jailbreak suite, the honest interpretation is “this model, under these English-language attack conditions, refused at this rate.” Extending that number to Hindi or Punjabi is an inference the card does not support, and after IndicSafeEval, it is an inference with published counter-evidence against it, however preliminary that evidence is.
The marketing layer makes this worse. Capability claims about multilinguality travel without safety qualifications: Merriam-Webster’s auto-compiled 2026 usage examples include a Forbes line from 2026-09-02 touting “multilingual communication” as a foundation-model strength, with no safety caveat attached. Capability in a language and safety in a language are separate training outcomes with separate evidence requirements. A model that converses fluently in Bengali has demonstrated the first and said nothing about the second.
Meanwhile, the operational burden lands on the deploying team. Language-industry trade coverage such as MultiLingual, which has covered the business of language since 1987, currently discusses operationalizing language under the label “LangOps,” but its coverage touches no model-safety auditing. There is no mature vendor category that will run your multilingual red team for you, and no fetched evidence that any compliance regime currently demands one. The audit exists only if you build it.
The practical reading rule for safety cards is therefore three questions. Which languages were probed? Which attack styles were used, direct queries or persuasive framing? And is the reported number aggregate or per-category? A card that cannot answer all three is a partial document, and deployments into non-English markets on the strength of it are running on unproven transfer.
So what is the right posture for a team shipping into these markets?
Treat English-measured refusal rates as unproven for Indic-language deployment, and run native-language persuasive-attack probes per language and per risk category before shipping, in roughly the 10-category × 6-strategy × 4-language shape IndicSafeEval demonstrates, while stating plainly in your own documentation that this posture rests on a single preprint and is a best practice, not a compliance requirement.
The cost is real and should be named. Native-speaker red teaming across four languages and ten harm categories is not a weekend project; it requires hiring or contracting native speakers with red-team judgment, building per-category taxonomies, and standing up baseline tracking across model versions. For teams that assumed translated test sets were sufficient, the preprint raises the audit cost of multilingual rollout from “run the translator over the English corpus” to “build a parallel red-team capability per language.” That is the honest price of the finding, and hand-waving it would be a disservice.
The strongest limitation cuts the other way, and it is substantial. Everything here rests on one author group’s preprint, evaluated black-box on open-source models in four languages, with no full text in the fetched record, no per-model scores in the abstract, and no independent replication as of 2026-09-08. Transfer of the specific findings to closed frontier models, to other Indic languages, or to other language families is unestablished. The EMNLP Findings acceptance is author-stated. If the paper’s numbers weaken under replication, the specific claim about these four languages weakens with it.
What survives regardless is the audit shape. The principle that translated test sets do not substitute for native-language persuasive probes follows from the attack mechanism itself, not from any single paper’s refusal percentages. The principle that aggregate refusal rates hide per-category gaps is arithmetic. And the habit of discounting English-only safety cards for non-English deployment is sound epistemics whether or not arXiv:2609.03781 holds up. Build the audit for those reasons, cite the preprint as the current evidence, and revisit the per-language numbers when the full text and independent replications land.
Frequently Asked Questions
Does IndicSafeEval cover closed frontier models like GPT-4 or Claude?
No. The abstract explicitly states the evaluation covered ‘several open-source LLMs’ only. Any claim that the results apply to closed frontier models is an extrapolation not supported by the paper’s own text, and no independent replication has tested those specific models as of 2026-09-08.
Why is machine-translated red-team data insufficient for Indic-language safety audits?
Translation flattens the native persuasive texture that drives jailbreak success. A translated prompt measures refusal of stilted phrasing, not the idiomatic appeals to authority or urgency a native speaker would use, which is the specific attack vector the benchmark targets.
What is the minimum audit scope for a product launching in Hindi and Bengali markets?
You need native-authored persuasive probes for each language, not just English translations. The audit must track per-category refusal rates separately, because the preprint shows vulnerability varies significantly across harm types, meaning an aggregate score can hide critical gaps in specific categories.
How should teams interpret vendor safety cards that cite only English benchmarks?
Treat them as silent on Indic-language behavior. The card proves safety under English attack conditions only. Extending that refusal rate to Hindi or Punjabi is an unsupported inference, especially given published evidence that safety behavior diverges across languages and phrasing styles.
What is the primary limitation of relying on IndicSafeEval for compliance decisions?
It is a single preprint with no independent replication or per-model scores in the abstract. The EMNLP acceptance is author-stated, not independently confirmed. Teams should use it as a best-practice audit shape, not as a regulatory mandate or verified compliance standard.
