Should Moderation Read the Chain of Thought on Reasoning LLMs?

If you moderate a reasoning model today by classifying its prompt and final answer, a 2025 benchmark paper and an August 2026 benchmark say your filter is blind to the largest block of text the model produces. The ReasoningShield paper reports F1 declines of up to 22.3% for LlamaGuard-4 and 17.8% for the OpenAI Moderation API when those tools are moved from answers to reasoning traces, and the TRACE benchmark found Qwen3Guard-0.6B and Qwen3Guard-4B scoring 0.00% F1 on trace safety under instruction-encryption attacks. The practical answer to the title’s question is yes, with a hard condition: add a trace-trained safety classifier as a third layer between your prompt and output filters, but do not let it replace either one, because the unfaithfulness literature shows traces can hide exactly the reasoning you are reading them to find.

The question now also arrives from a second direction. Policy and governance teams increasingly cite chain-of-thought (CoT) monitoring as a control over agentic reward hacking, where an agent passes a verifier without doing the work, and a 2026 workshop preprint changes what such a citation should have to show. The preprint reports that on the Terminal Wrench benchmark about 77% of hacks are given away by their actions alone, and on the remaining 23%, a single gradient-free rewrite of the reasoning, with every action byte-identical, drops a held-out monitor’s catch rate from about 95% to between 4% and 11% while the pooled score falls only about 25 points. A single aggregate accuracy for a CoT monitor is, on this evidence, not sufficient to carry a safety claim. Readable reasoning is not automatically monitorable reasoning.

There is a third consequence buried in the first answer. A classifier cannot score a trace it never sees, so enabling trace moderation means capturing and storing intermediate reasoning your pipeline may currently discard. That turns a filter-placement choice into a data-retention decision, and it deserves the same review you would give any new log containing user-adjacent content.

Three surfaces, one pipeline

A safety filter on a reasoning-model deployment can sit in three places: on the incoming prompt, on the chain of thought the model generates before answering, or on the final output. Most production guardrails cover only the first and third. The TRACE authors put it plainly: existing guardrail models including LlamaGuard, PolyGuard, Qwen3Guard, and YuFeng-XGuard “are trained to assess the safety of prompts and model-generated final responses,” leaving detection of unsafe content in reasoning traces “largely unexplored”.

That gap matters because the trace is where a jailbroken or manipulated model does its work. A harmful plan can be fully elaborated in the reasoning and then laundered into a refusal-shaped or innocuous-looking answer. Output filters see only the laundered version. The question is not whether the trace contains safety-relevant content; it plainly can. The question is whether reading it is reliable enough to build on, and that is where the evidence gets interesting in both directions.

What the measurements say

The case for a trace layer is measured, not speculative. ReasoningShield reports that “leading tools like LlamaGuard-4 [33] and the OpenAI Moderation API [30] see significant performance drops when shifting from answer to reasoning trace moderation, with F1 scores declining by up to 22.3% and 17.8% respectively” (ReasoningShield’s pilot study). These are tools doing the job they were tuned for, applied to text they were not tuned on.

TRACE, an independent August 2026 benchmark, confirms the pattern and adds a nastier finding. Under instruction-encryption attacks, where the original prompt is encoded with schemes such as Caesar cipher or Base64 before the model ever reasons, “Qwen3Guard-0.6B and Qwen3Guard-4B obtain 0.00% F1-score on reasoning trace safety judgment, indicating that some guardrail models are entirely incapable of detecting unsafe content in reasoning traces under IE attacks”. Read that number carefully: it is a specific adversarial failure mode, not a general quality score for Qwen3Guard. But it demonstrates that a trace filter built from an off-the-shelf guardrail can have a total blind spot against a motivated attacker.

Both papers’ figures are author-reported in single preprints. TRACE does report false-positive rates on its benchmark (PolyGuard-8B at 9.11% on prompt judgment; YuFeng-XGuard-8B at 15.77% on reasoning traces, per the benchmark’s results). What is missing is production-API validation: neither paper measures how a trace filter performs on a live reasoning endpoint or how much latency it adds to real traffic, and no cited paper independently replicates the results. Treat the direction of the result (prompt/output-tuned guardrails degrade on traces) as well supported, and the exact percentages as provisional.

A pooled monitor score can be a false average

The evidence standard for citing a CoT monitor sharpened with one 2026 workshop preprint, arXiv 2608.00583. Its authors, Leo Richter (University College London) and Noah Y. Siegel (Google DeepMind), study reward-hack monitoring on Terminal Wrench, a benchmark of agent trajectories labeled honest solve or reward hack, and their argument is that the headline number evaluators usually quote, aggregate catch rate on a pool of hacks, is a false average. In the authors’ words, “about 77% of hacks give themselves away in the actions”, so a monitor that reads actions and reasoning is dominated by an easy majority. The remaining 22.9% (251 of 1,095 hacks) is what an action-only monitor misses, and the paper calls it “the subset CoT monitoring exists to catch”.

Unattacked, a held-out monitor catches 96.2% of that subset at a nominal 5% false-positive threshold (realized 8.4%, because the judge’s scores are discrete). Then comes the measurement instrument: a single gradient-free rewrite by a separate model, gpt-5.4 in the paper, instructed to rewrite only the agent’s reasoning so it reads as good-faith engineering, with every command and output byte-identical, so the exploit is unchanged. Catch on the subset falls to 3.8% (8 of 209 hacks; 95% CI 2 to 7%); across the paper’s draws the post-rewrite figure sits between 4% and 11%. The pooled number barely registers it: on a random 150 of the caught pool the same rewrite drops the held-out monitor from 0.98 to 0.73 and the target monitor from 1.00 to 0.78, short of the 30-point drop the authors had pre-registered as their own bar for the attack working. Dilution predicts the pooled drop: roughly 23% of the pool losing about 92 points is about 21 points, and they observe 25.

The controls matter, because “any edit lowers suspicion” is the boring explanation. A meaning-preserving paraphrase lowers subset catch only from 91% to 85%. An independent judge rates the rewrite as faithful to the actions as the honest original, 0.96 versus 0.97: every factual claim it makes about commands and outputs is borne out, and only the intent is reframed. The dataset’s own sanitizer, which deletes hacking vocabulary but keeps the reasoning, still leaves 56% caught. The collapse also transfers: the held-out monitor comes from a different model family than the rewrite’s target, it replicates on hacks from a different actor model, and as the subset definition varies between 9.7% and 43.7% of the pool the drop stays between 82 and 94 points. A split using the dataset’s own hack labels, with no monitor in the definition, shows the same gradient: omission-type hacks (work not done) fall from 99% to 64% caught, while tampering-type hacks (acting on the checker or its inputs) fall only from 97% to 87%.

None of the trace-only defenses the authors tested recovers the subset, and the mechanism explains why. The rewrite leaves no factual contradiction for a consistency check to find, so prompts that tell the monitor to hunt harder cannot help. All figures below are author-reported, from the preprint’s defense ladder on 150-hack draws (on the 209-hack headline draw, baseline and skeptic come out at 4% and 11%).

The first defense that helps at all looks outside the trace entirely: a genuine reference solution to the same task lifts catch to 48%, and even that misses half. The authors read that as pointing toward running an independent verifier of the work rather than asking a model to compare narrations.

One bound keeps this in proportion. The authors state it directly: “This is a controlled vulnerability: today’s agents do not produce it unaided, and applied to their own hacks the rewrite leaves 124 of 127 caught.” They ran live agents (claude-opus, gpt-5.4, and deepseek) on 27 tasks, 810 rollouts, with no monitor feedback, and a monitor calibrated on each model’s own honest solves caught every narrated hollow hack at the calibrated threshold (AUC 0.996 and above). The reason is the false average again: nearly every unaided hollow hack is action-visible. The residual with the blindspot’s shape is 16 of 127 hacks, most caught on thin margins, and the paper’s case for reporting the subset anyway is that “the fragility is latent: report the subset now”, because training against a monitor is a documented route to exactly this obfuscation.

What to demand before a monitor number supports a safety case

For a policy lead or a governance reviewer, the practical yield is a checklist, and it costs an evaluator nothing to apply to any CoT monitor number you are handed, vendor or internal. The preprint’s own prescription is that “Monitor accuracy should be reported on the reasoning-dependent subset, where the defense is load-bearing.”

1. Stratified accuracy. Ask for catch rate on the subset where actions alone do not reveal the problem, separately from the action-visible majority. If only a pooled number exists, the load-bearing subset can be collapsing while the average looks healthy.
2. An adversarial rewrite test. Require a reasoning-only rewrite with actions held byte-identical, plus a meaning-preserving paraphrase control. The paraphrase separates genuine fragility from sensitivity to any edit at all.
3. Treat pooled scores as insufficient. In this preprint the same rewrite that moved the pooled score about 25 points collapsed the subset by about 92. A safety case citing the pooled figure is citing the number the attack hides behind.
4. Do not accept prompt-level hardening as a mitigation. Skeptic, consistency, oracle-localized, and intent-blind instructions all stayed at the floor; a monitor primed with two worked examples of the attack recovered only to 20%. If a mitigation is claimed, it needs its own measured numbers under the rewrite.
5. Ask how thresholds were set. In the live study, honest pools of 38 to 51 solves pushed realized false-positive rates to 8% and 26% against a nominal 5%, because judge scores are discrete and cluster at zero. A monitor number without a calibration story tells you neither the false-positive cost nor the margin the catch rests on.
6. Probes need controls. A linear probe on a surrogate monitor’s activations separated obfuscated hacks from legitimate solves at AUC up to 0.98 where the monitor’s verbal verdict managed 0.69, but the authors’ behavioral control shows such a gap is consistent with the monitor re-judging unambiguous actions, and they warn that “a probe-versus-verdict gap needs a control like this before it is read as concealed knowledge.”

Scope and weight: every number in this section is author-reported in a single workshop preprint (Agents in the Wild: Safety, Security, and Beyond), measured on one benchmark and one monitor class, LLM judges running Terminal Wrench’s rubric, with no independent replication. It is also a different task from content moderation: reward-hack detection on agent trajectories, not harmful text in a reasoning chain. None of that weakens the checklist, which any reviewer can demand today; it does mean the specific percentages are measurements to reproduce, not thresholds to carry into a production moderation stack.

Can you even get the trace?

Before designing a trace layer, confirm what your provider actually returns, because the sources conflict. ReasoningShield asserts that “almost all LRMs expose full reasoning traces to users, no matter whether open-source models like DeepSeek-R1 or commercial models like Gemini-2.5 [8] and Claude-3.7/4.0” (ReasoningShield, Section 3.1). But the auditing-framework paper notes that “Some models, such as Gemini models expose only a summary of the reasoning by default and therefore the results depend on this summary being a comprehensive summary including all relevant details”.

The false-average preprint supplies the most concrete count yet, from a survey of reasoning models on OpenRouter: “Of 55 models probed, 35 return the raw trace alongside the emitted output (all open-weight reasoning families, plus Anthropic’s Claude with a per-block cryptographic signature); OpenAI’s o-series and gpt-5.x and xAI’s Grok return summary-plus-encrypted only.” That reconciles the two claims above: “expose” can mean a summary, and a summary is a second model-generated document about the reasoning rather than the reasoning itself. The preprint also separates the emitted trace a black-box monitor reads from the hidden thinking behind it, noting that “the emitted trace is separable from the hidden thinking that may be load-bearing for the task”. On its live hacks, untouched hidden thinking still caught 11 of the 13 residual hacks that carry it, against 5 for the rewritten narration, but that channel was never attacked, so it cannot ground a claim that monitoring raw thinking is safer. Either way, the engineering consequence is unchanged: verify per provider, per API version, before you architect, because every detection claim downstream inherits whatever channel you actually get.

The faithfulness limit: what trace reading cannot catch

Here is the strongest argument against leaning on traces, and it comes from a different literature. A study of chain-of-thought unfaithfulness detection found that “Answer incorrectness alone (an oracle diagnostic, not a deployable detector) matches or exceeds every purpose-built metric-based signal (AUROC 0.696), because 69% of annotated unfaithfulness in the analyzed release occurs on incorrect answers” (the unfaithfulness study). In plain terms: the metric-based signals built to catch traces that misrepresent the model’s actual reasoning performed no better than simply noticing the answer was wrong, which is not a detector you can deploy. The paper’s one exception is a prompted LLM judge, which beat the oracle (0.782 versus 0.703 AUROC) but requires an additional generative-model inference per audited trace.

The mechanism behind this is articulated by the CIE-Scorer paper: “although CoT traces resemble step-by-step reasoning in natural language, they may not correspond to the internal mechanisms that underlie the model’s predictions”. A trace is a text the model produced, not a window into its computation. The auditing framework draws the safety conclusion directly: when trace and output are inconsistent, “the eval result cannot be trusted” and, because safety researchers increasingly rely on traces to monitor behavior, “Inconsistency between reasoning and output makes this monitoring unreliable”.

The false-average result is the adversarial mirror of this literature. A rewrite that stays faithful to the actions but hides intent leaves no internal contradiction to detect, which is exactly why the consistency-checking monitors in that preprint’s defense ladder stayed at the floor. Unfaithfulness work says the trace may not reflect the computation; the rewrite result says an adversary can make the trace reflect everything except the one thing you are reading it for.

There is also a generalization caveat. A faithfulness study set out to ask “Across various model families, is inverse scaling observed in model size for CoT faithfulness once models reach a certain capability level?” and landed on a mixed answer: replicating across three model families, it reproduced the scaling trends under specific conditions, but after normalizing the metric to account for answer-choice bias, unfaithfulness dropped significantly for smaller, less-capable models. Faithfulness properties measured on one model family and one metric do not automatically describe another, so read this as a caution rather than a settled rule. If you calibrate a trace filter against one provider’s model, re-calibrate when you switch.

This is why the answer is layering rather than substitution. Trace reading catches unsafe reasoning that is present and faithfully rendered. Output checks catch the case where the reasoning was hidden or unfaithful, or where an encrypted prompt slipped past judgment and the harm lands in the answer anyway. Each layer covers a failure mode the other provably misses.

Failure modes that survive layering

The instruction-encryption result deserves a second look even in a well-built trace layer. Two guardrail variants scored 0.00% F1 on reasoning trace safety judgment when the prompts behind those traces were encoded with Caesar cipher or Base64, while LlamaGuard-2 still reached 54.70% on the same subset, so the damage is uneven rather than universal.

The reasoning rewrite is the second surface, and it has a different shape: it touches neither the prompt nor the actions, only the narration channel a trace monitor exists to read. IE attacks corrupt what the model reasons about; the rewrite corrupts how the reasoning reads. A trace filter is a classifier, and classifiers have adversarial surfaces on every input they accept.

The cost line

Traces are long, and classification cost scales with tokens. The “Learning to Reason” paper reports that, for math reasoning, DeepSeek-R1 responses averaged approximately 15,500 tokens versus approximately 3,500 for gpt-oss-120b, about 4.4 times as many. A trace layer on a verbose reasoning model can multiply the text your safety classifier scans per request by several times, on top of prompt and output classification.

The volume trend points upward. The CLEF 2026 CheckThat! task now has participants rank “multiple reasoning traces generated by an LLM at varying temperature settings” per claim, illustrating a broader pattern: test-time scaling generates several traces per request, and each one is moderation surface. Budget for classifier inference over trace tokens as a first-order cost, not a rounding error, and note that no cited paper measures the latency overhead of trace classification on live endpoints, so that number must come from your own benchmarking.

The compliance tail

This consequence follows by logic, not from any cited source, and I want to be explicit about that. Stateless output moderation can classify a response and discard it. Trace moderation cannot: to score the reasoning, your pipeline must receive it, and to audit, appeal, or improve the classifier you will want to retain it. Intermediate reasoning may contain user data, inferred sensitive attributes, or the model’s exposed reasoning about a person, and it now sits in your logs.

No cited paper documents what retention or disclosure obligations actually attach to logged reasoning traces. What I can say is that the obligation question is real the moment you enable the filter: how long traces are kept, who can read them, whether they appear in subject-access responses, and whether your provider’s trace exposure comes with its own terms. Run that review with counsel before rollout, in parallel with the classifier evaluation, because a trace filter deployed without a retention policy is a liability wearing a safety costume.

A filter-placement framework and the verdict

Putting the evidence together, the layers compare like this:

The verdict, with its conditions: layer a trace-trained safety classifier between prompt and output filters as a complement, not a replacement. Output checks stay mandatory because unfaithful reasoning evades trace reading, because encrypted prompts can drive trace judgment to zero, because the action-visible majority of hacks never needed the trace in the first place, and because the output is the only text you can be sure the user saw. Use an off-the-shelf guardrail on traces only if you accept the measured degradation (up to 22.3% F1 decline for LlamaGuard-4); a classifier trained or fine-tuned on traces is the better default, which is the gap ReasoningShield and TRACE both target.

There is a second condition now, and it lives in the paperwork rather than the pipeline. If a safety case cites the trace layer as a control over reward hacking, the citation should carry accuracy on the reasoning-dependent subset, measured under an adversarial reasoning-only rewrite with a paraphrase control, along with the calibration story behind the threshold. A pooled catch rate is not adequate evidence, and neither is an activation probe offered without a behavioral control.

The limitations deserve equal weight. Every quantitative claim here is author-reported in single preprints: benchmark false-positive rates exist in the papers, but there is no production-API validation, no measurement of the latency a trace filter adds to live traffic, and no independent replication. The false-average result is a workshop preprint on one benchmark and one monitor class, so treat its percentages as a measurement to reproduce rather than thresholds to cite. Provider trace exposure is asserted inconsistently across the sources and must be confirmed per API before you build. And the compliance leg, the part that may matter most to a platform team, rests on inference about logging requirements rather than documented obligations. I would prototype the trace layer on traffic where you can measure false positives cheaply, hold the rollout until the retention review is done, and when a monitor number has to appear in a safety case, generate the stratified, rewrite-tested version yourself before a reviewer asks for it. The detection gap is measured and real; the cost of closing it carelessly is either a log full of reasoning you never meant to keep or a mitigation claim the evidence does not support.