Frontier AI procurement teams usually frame safety as a jailbreak problem: if the model resists adversarial prompts, it is safe enough. Anthropic’s public record over the past year suggests the problem is split in two. One failure mode is behavioral: the model emits harmful output from an ordinary, non-malicious request. The other is institutional: the vendor declines a mission because of its corporate structure, usage policy, or sales restrictions. Both can block a deployment, but they require different due diligence. Confusing them means optimizing for the wrong failure.
The capability threat model
In the capability threat model, the model is dangerous in normal operation. A reviewer or end user asks for innocuous help with code, infrastructure, biology, or security, and the model returns output that enables harm without any adversarial framing. The user’s intent is not the signal to catch; the model’s underlying competence is. Prompt filters tuned to detect jailbreaks, prompt injections, or misuse cues will miss this, because the request reads like legitimate work.
The right controls here are capability evaluation, scoped deployment, and output review. Anthropic publishes a Responsible Scaling Policy for this class of problem. Teams need to know not just how the model behaves under attack, but what it can produce when asked straightforwardly. That requires evals that assume the prompt is benign and test the output itself, plus governance over where the model is allowed to run unsupervised.
The policy threat model
In the policy threat model, the model may pass every red-team exercise and still be unavailable. The risk is not that the model fails at runtime; it is that the vendor refuses the customer or the use case before the first token is generated.
Anthropic is a public benefit corporation and says it is dedicated to securing AI’s benefits and mitigating its risks, according to its homepage. That structure is not decorative. In September 2025, the company announced it would stop selling its products to groups majority-owned by Chinese, Russian, Iranian, or North Korean entities, citing national security concerns, as recorded in its Wikipedia entry. This kind of restriction is contractual and political. A procurement team can have perfect technical results and still find the door closed because of who they are or what they plan to do. The diligence target shifts from model weights to governance documents and usage-policy redlines.
Where the China-hacker disclosure fits
The closest verified analog to a capability concern in the cached sources is Anthropic’s November 2025 disclosure that hackers sponsored by the Chinese government used Claude in automated cyberattacks against roughly 30 organizations. The attackers bypassed safeguards by framing their requests as defensive testing, according to the same source.
That episode is adjacent to the capability threat model but not the same thing. The users were malicious actors misrepresenting their purpose. The model did not spontaneously produce harmful output from a benign prompt; it was tricked into treating an offensive task as defensive work. That is a guardrail-evasion problem, not a proof that ordinary coding questions expose latent dangerous capability. The controls it points to are stronger use-case verification and abuse monitoring, not just capability evals.
What a benign-prompt capability incident would imply
The two models are easy to conflate because both involve harmful output. The difference is in the trigger. In a genuine capability incident, a legitimate reviewer asks an ordinary question and the model reveals competence that was not supposed to be on offer. No deception is required. The implication is that the safety bar is not adversarial robustness but the model’s baseline behavior.
That scenario is theoretically important for frontier-model buyers. It means jailbreak-resistance scores are insufficient. A team purchasing a coding model should ask what the vendor knows about benign-prompt outputs in security-relevant domains, and what human review or automated filtering sits between the model and production. The absence of a verified incident in the current sources does not make the question optional.
What procurement teams should actually check
For teams evaluating a frontier model in a regulated or government-adjacent context, the checklist should follow from Anthropic’s verified posture rather than from a generic safety checklist.
First, read the vendor’s governance and published sales restrictions before running benchmarks. Anthropic’s public benefit corporation status and its September 2025 decision to stop selling to certain state-owned entities mean some missions are excluded regardless of model performance. If your intended use case or customer base falls into those categories, the procurement risk is not technical failure; it is that the vendor will decline the mission outright. Discovering this after a benchmark cycle wastes engineering time and political capital.
Second, ask for evidence of benign-prompt capability evaluations, not just adversarial robustness scores. The November 2025 China-hacker disclosure shows that requests framed as legitimate security work can produce offensive output. Buyers should understand what output filtering, human review, and scope limits apply to coding, infrastructure, or dual-use tasks. A red-team certificate is not a substitute for knowing what the model does when it thinks the request is normal.
Third, treat the vendor’s public safety commitments as a durable procurement signal. Anthropic states that it builds AI to serve humanity’s long-term well-being and puts safety at the frontier. That posture is on the record. It should be mapped against your intended use cases early, not treated as marketing language that can be negotiated away.
The central procurement question is not whether one safety metric is high enough. It is whether the risk you are managing is a behavior the model might exhibit or a customer the vendor will not serve. Anthropic’s record makes the second risk real. Buyers who plan for only the first are auditing the wrong thing.
Frequently Asked Questions
Which Anthropic product lines and buyer categories are actually restricted under its public commitments?
The company’s disclosed products include Claude (Opus and Sonnet 4.6), Claude Code, Cowork, Claude Design, Bun, and the Mythos line announced in April 2026. Its verified restrictions cover buyers majority-owned by Chinese, Russian, Iranian, or North Korean entities, plus the broader US federal supply chain after the Pentagon designated Anthropic a supply-chain risk on March 5, 2026.
How does the confirmed Anthropic-US government dispute differ from a jailbreak-driven ban?
The anchored episode began February 24, 2026, when Defense Secretary Pete Hegseth threatened to invoke the Defense Production Act and remove Anthropic from the Defense Department supply chain by February 27 unless it allowed unrestricted use of Claude. Anthropic refused, and President Trump then ordered agencies to stop using its models; the dispute centers on surveillance and autonomous-weapons safeguards, not on a coding prompt that bypassed safety filters.
What should procurement teams do if a frontier vendor is formally designated a supply-chain risk?
Treat the designation as a termination event, not a negotiation. Anthropic responded that it had ‘no choice but to challenge … in court,’ so buyers should map alternative models and contract exit clauses before any legal resolution. Relying on a single frontier vendor becomes a continuity risk once a dispute moves from policy disagreement to formal exclusion.
What is the closest verified analog to a benign coding prompt exposing dangerous capability?
There is none in the available sources. The November 2025 China-hacker disclosure involved state-backed actors framing offensive work as defensive testing, which required deliberate misrepresentation. A genuine benign-prompt incident would need no such framing, and none of the available sources confirms that type of event for any Anthropic model.