Can Your LLM Red-Team Corpus Jailbreak Diffusion Language Models?

Probably, yes. If your security team owns a jailbreak corpus built against autoregressive LLMs and you are triaging a diffusion language model for production text generation, the working assumption should be that the corpus transfers, but through an indirect pipeline rather than by pasting prompts, and that your prompt-level defenses will not close the gap. That assumption rests on one self-reported preprint, RMT-Jailbreak, which transfers existing autoregressive jailbreak attacks to diffusion language models by first inducing a harmful response from a source autoregressive model, then masking that response into a sparse template for the target dLLM to reconstruct. The transferred attacks reach 61.3–76.5% attack success on HarmBench and 77.0–84.0% on JailbreakBench (arXiv 2610.31331), and on HarmBench success remained at 69.5%–72.0% even under a Self-reminder defense (arXiv 2610.31331).

The practical consequence is a budget shift, not a panic. Your existing corpus is still worth rerunning against any candidate dLLM before go-live; the rerun order is what changes, because the claimed transfer mechanism tells you which prompts are most likely to port. And because the one prompt-level defense tested did not reliably suppress the attack, residual risk belongs in deployment-time controls rather than in model choice. All of this is one preprint’s self-report rather than an established finding. That is enough to schedule a rerun, and not enough to certify any specific production model.

The claim, precisely

The preprint’s core statement is worth quoting exactly, because it is narrower than its title:

“Based on this observation, we propose RMT-Jailbreak, a general framework that transfers existing autoregressive (AR) jailbreak attacks to dLLMs through their induced harmful responses.” (arXiv 2610.31331)

Two things follow from the wording. First, the mechanism runs through a source model, not the target: “Given a harmful query, RMT-Jailbreak first applies an off-the-shelf AR jailbreak method to an autoregressive model to obtain a query-specific harmful response” (arXiv 2610.31331). That response is converted into a high-mask-ratio template with sparse tokens retained at roughly uniform intervals, filtered so that no explicitly harmful token remains visible, concatenated with the original query, and handed to the target dLLM, which fills in the masks through bidirectional generation. The implementation defaults to AutoDAN against Mistral-7B-Instruct-v0.2, with vicuna-7b-v1.5 as a second source model in the generalization tests. So the attack does not need your candidate dLLM to fall for your original prompt at all; it needs an autoregressive model somewhere that will.

Second, “transfers” is a claim about the framework’s self-reported performance across named model pairs, not a universal property of diffusion architectures. The paper evaluates four target dLLMs (LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-Instruct-7B and MMaDA-8B-MixCoT) on three benchmarks. Applied directly to LLaDA-Instruct on HarmBench, AutoDAN succeeds 11.5% of the time and the RA attack 9.8% (arXiv 2610.31331). Routed through response templates induced from Mistral-7B-Instruct-v0.2, the same attacks reach 76.0% and 41.3% respectively (arXiv 2610.31331). Across all four targets, RMT-Jailbreak reports 61.3–76.5% attack success on HarmBench, 77.0–84.0% on JailbreakBench, and StrongREJECT scores of 56.7–70.4, all self-reported (arXiv 2610.31331). If your procurement review asks “what percentage of our corpus ports,” the paper’s answer has two halves. Direct transfer “generally yields limited and highly variable performance,” in the paper’s words, and the appendix adds that this resistance “is neither uniform nor absolute” (arXiv 2610.31331). ReNeLLM, which constructs nested or rewritten prompts, reaches 78.0% on LLaDA-Instruct, 76.0% on LLaDA-1.5 and 74.0% on Dream-Instruct when applied directly on JailbreakBench, with 49.0% on MMaDA-MixCoT, all self-reported (arXiv 2610.31331). The second half is the paper’s central claim: any prompt that can still elicit a harmful response from some AR model becomes raw material for a masked-template attack. Note what the ReNeLLM numbers do to the rerun order: nested and rewritten prompts are instruction-structure payloads, and on at least one benchmark they already succeed on dLLMs without any template pipeline.

The same paper makes a second claim that matters more for test design than for headlines: the diffusion architecture itself may carry a distinct safety risk. The self-reported observation is that

“Uniform masking exposes a distinct safety risk of bidirectional generation: even when most harmful content is masked, distributing the remaining visible tokens throughout the response provides local contextual constraints for reconstructing unsafe content.” (arXiv 2610.31331)

In plain terms: an autoregressive model generates left to right, so a refusal at the start of a sentence can cut off everything after. A masked diffusion model generates across the whole response at once, filling in blanks with bidirectional context. If scattered harmful tokens survive masking at distributed positions, the model’s own fill-in process can reconstruct the unsafe content around them. Whether that happens in any model you are evaluating is an open research question; the mechanism is a hypothesis with a testable shape, not a settled fact.

What you already own

The reason this preprint should not trigger a from-scratch program is that AR-era red-team corpora are, by construction, enumerable and rerunnable. Two examples from the literature show what “enumerable” means in practice.

The Latent Jailbreak benchmark constructs its suite combinatorially: per its authors, combining “the 13 templates with 16 words related to protected groups and 2 positional types (prefix and suffix)” yields “a total of 416 possible latent jailbreak prompts” (arXiv 2307.08487). A corpus built as templates × slots × positions is a rerun manifest, not a folder of screenshots. You can regenerate it, diff results across targets, and trace any failure back to the template family that produced it.

The Jigsaw Puzzles (JSP) attack shows a second kind of portable asset: prompt mechanics that depend on instruction following rather than on any specific model. Per the paper, a JSP prompt “explicitly instructs the model not to generate the concatenated question but to provide responses directly based on the fraction of each turn” (JSP, arXiv 2410.11459). A jailbreak whose payload is an instruction about output structure has no intrinsic dependence on left-to-right generation. Whether it works on a dLLM is an empirical question, but there is no architectural reason to assume it fails, which is exactly why it belongs early in a rerun list.

The inference I would draw, and it is an inference rather than a measured finding: prompts that encode their payload in instruction structure (JSP-style) and prompts generated from compositional templates (Latent Jailbreak-style) are the cheapest to rerun and the most likely to interact with the claimed harmful-response-template mechanism. Prompts that depend on token-level tricks tuned to a specific AR tokenizer go last, because their transfer depends on details the preprint does not report.

Architecture-specific tests your corpus does not have

Porting the corpus is necessary but not sufficient, because the uniform-masking observation describes a failure mode with no AR analogue. A dLLM evaluation needs at least one test family that does not exist in your current suite: constrained harmful completions, where scattered fragments of unsafe content appear at distributed positions in a partially masked response and the model is asked to fill the rest. That test design follows directly from the quoted masking passage above. The paper’s own numbers make the case for funding it: on LLaDA-8B-Instruct, masking a harmful response at ratios from 0.3 to 0.9 still yields 85.5–97.75% attack success, all self-reported (arXiv 2610.31331). The mask-pattern ablation turns the mechanism claim into direct evidence: at a fixed ratio of 0.8 on LLaDA-8B-Instruct, uniform masking beats contiguous masking by 58.3 ASR points on HarmBench, 18.0 ASR points on JailbreakBench and 60.4 StrongREJECT points, all self-reported (arXiv 2610.31331). Because the ratio is held fixed, the gap comes from where the visible tokens sit, not from how many are visible. It is the one place where I would spend genuinely new budget, because no transferred AR prompt exercises it.

There is also a cross-modal precedent for treating alignment as a weak barrier rather than a wall. Cited in a 2025 paper, “a single adversarial image can bypass the safety mechanisms of alignment-tuned LVLMs and induce harmful outputs” (arXiv 2505.21556). That result is about vision-language models, not text dLLMs, so it establishes nothing about your candidate model. What it does establish is a pattern: alignment tuning has repeatedly failed to block inputs crafted to route around it, in modalities where the bypass was eventually demonstrated. The burden of proof sits on “our alignment handles transferred attacks,” not on the attacker.

What the defense results actually show

The defense experiment is broader than a single number, and the fuller picture is worse for prompt-level defenses than the HarmBench headline alone. Self-reminder reduced attack success only on HarmBench, where it “declines modestly but remains at 69.5%–72.0%” (arXiv 2610.31331). On JailbreakBench, “attack success instead increases for all three models, while StrongReject shows no reduction,” and the authors conclude that the defense “does not consistently suppress the attack” (arXiv 2610.31331).

Against complacency: a prompt-level defense that leaves the large majority of attacks succeeding on one benchmark, loses ground on a second, and holds steady on a third is not a control you can sign off on. Against fatalism: the HarmBench decline shows transferred attacks do respond to their deployment setting, so layering is not pointless. But responding is not stopping, and a safety reminder demonstrably does not stop this attack.

When your rerun produces numbers, demand the reporting discipline the attack literature already uses. PBI-Attack, for example, reports success against a named baseline (arXiv 2412.05892): a 94.9% success rate on MiniGPT-4 versus 87.5% for the next-best method UMK, both self-reported by its authors. That is the yardstick: attack success rate, named model, named defense, named baseline. RMT-Jailbreak’s own tables already follow it, pairing each direct attack rate with its response-mediated rate on each named target. A rerun report that says “the model refused most prompts” without a rate against a baseline is not comparable to anything, including next quarter’s rerun.

Harness and tooling: reuse exists, with a gap you fill yourself

On the harness side, JailbreakBench is the natural slot for reruns because it explicitly accepts the attack category this article is about: per its authors, “We accept any jailbreaking attack, including white-box, black-box, universal, transfer, and adaptive attacks, and any jailbreaking defense, all of which are compared using the same set of evaluation metrics” (JailbreakBench, arXiv 2404.01318). Shared metrics are what make a dLLM rerun comparable to your historical AR results instead of a fresh island of numbers.

The gap: the framework’s quoted target list is autoregressive-only, naming “Vicuna-13B-v1.5 (Zheng et al., 2023), Llama-2-7B-chat-hf (Touvron et al., 2023), GPT-3.5-Turbo-1106 (OpenAI, 2023), GPT-4-0125-Preview (OpenAI, 2023)” (JailbreakBench, arXiv 2404.01318). Reusing the harness means adding dLLM targets yourself and owning the integration risk that comes with that.

On the tooling side, the dLLM framework is described by its authors as “an open-source framework that unifies the core components of diffusion language modeling—training, inference, and evaluation—and makes them easy to customize for new designs” (arXiv 2602.22661). Per the same paper, it lets users reproduce models like LLaDA and Dream, and ships recipes for converting BERT-style encoders and autoregressive models into diffusion language models. That overlaps usefully with the attack evidence: RMT-Jailbreak’s four targets include LLaDA-8B-Instruct, LLaDA-1.5 and Dream-v0-Instruct-7B, so the models this framework helps you stand up are the same class the attack paper evaluated. Two cautions attach. First, the preprint’s target list does not include Mercury or Gemini Diffusion, so its numbers do not describe those systems. Second, a conversion recipe demonstrates that you can construct a dLLM from an AR model; it says nothing about whether the converted model inherits the source’s safety behavior. Treat converted models as fresh targets.

The other direction, briefly

Do not conflate two research lines that share the word “diffusion.” DiffusionAttacker is presented by its authors as “an end-to-end prompt rewriter for jailbreak attacks, representing the first application of diffusion language models to this task” (arXiv 2412.17522), using a seq2seq diffusion model for flexible token modification. That is a dLLM attacking conventional LLMs. RMT-Jailbreak is conventional jailbreak techniques attacking dLLMs. The directions have different implications for you: the first expands the attacker’s toolkit against models you already run; the second determines whether a new model class you might adopt inherits your existing threat model. Both argue for rerun discipline, but only the second is this article’s question.

The porting checklist

Putting the evidence together, this is the order I would run, with the caveat that the ordering is reasoned from the mechanism claim, not measured:

1. Instruction-structure prompts first. JSP-style items whose payload is an output-format instruction (arXiv 2410.11459). Cheapest to rerun, no tokenizer dependence, plausibly compatible with harmful-response templating.
2. Compositional template families second. Regenerate your Latent Jailbreak-style suites from their template × slot × position definitions (arXiv 2307.08487) so every dLLM failure is traceable to a template family.
3. Uniform-masking reconstruction tests, newly written. Partially masked responses with harmful fragments at distributed positions, per the mechanism described in arXiv 2610.31331. This is where new budget goes, because nothing in your corpus covers it.
4. Tokenizer-tuned AR tricks last. Their transfer depends on implementation details the preprint does not report.
5. Report everything against named baselines and named defenses, in the style of PBI-Attack’s rate-versus-baseline reporting, inside a shared-metrics harness like JailbreakBench with your own dLLM targets added.

Where the residual risk goes

The economics argument is the part none of the cited papers frames, so label it as reasoning: if a jailbreak template transfers across architectures, then publishing it once creates a standing attack against every model it reaches, present and future. Model choice cannot retire that risk, because the next model may inherit the template for free. What can absorb it is deployment-time control: input and output filtering you operate, rate and context constraints, monitored reruns each time the model or the corpus changes.

Here the vendor assumption deserves direct skepticism. Red-teaming of the Stable Diffusion safety filter found that “the filter is obfuscated and poorly documented. This makes it hard for users to prevent misuse in their applications, and to understand the filter’s limitations and improve it” (arXiv 2210.04610). That is an image-model precedent, not a result about any text dLLM, but it is a concrete warning against writing “vendor filter handles transferred jailbreaks” into a risk register on trust.

What this does not establish

Everything transfer-related rests on one preprint’s self-reported experiments: four open-weight target dLLMs, two source AR models, three benchmarks, one team. The 69.5%–72.0% figure is the post-defense range on HarmBench only (arXiv 2610.31331); the same defense increased success on JailbreakBench and left StrongREJECT scores unchanged. Nothing here tests Mercury or Gemini Diffusion, and nothing here tells you how any specific model behaves against your particular corpus. The right posture is a scheduled rerun and a deployment-controls budget line, with a re-check for replication attempts and vendor responses before this analysis hardens into policy. The checklist and the masking-test design will still be valid when those replications land; the transfer rates may not be.

Frequently Asked Questions

Did the Self-reminder defense stop the transferred attacks?

Self-reminder reduced attack success only on HarmBench, where it “declines modestly but remains at 69.5%–72.0%” (arXiv 2610.31331). On JailbreakBench, “attack success instead increases for all three models, while StrongReject shows no reduction,” and the authors conclude that the defense “does not consistently suppress the attack” (arXiv 2610.31331).

What new test is needed for diffusion language models?

A dLLM evaluation needs at least one test family that does not exist in your current suite: constrained harmful completions, where scattered fragments of unsafe content appear at distributed positions in a partially masked response and the model is asked to fill the rest.