Per-Decision Evidence vs Snapshot Audits: What AI Agent Governance Must Log

When an agent’s action is challenged, in an incident, a dispute, or a regulator’s request, the artifact that settles it is never the Model Card. A preprint protocol called AIREP (arXiv:2608.21363), observed in the arXiv feed on 2026-09-18, argues that every governed runtime decision should leave four distinct records rather than one collapsed audit event. This moves the burden of proof in agent governance from pre-deployment review to runtime evidence, and the bill arrives as retention: verbose traces collide with privacy and storage budgets.

What a snapshot cannot tell an investigator

The snapshot side of governance is candid about its own limits. A study on translating EU AI Act requirements into executable compliance pipelines uses a blunt example: “A Model Card reporting a 3% hallucination rate stays unchanged when production climbs to 9%” (arXiv:2609.20016). The authors frame this as purpose, not deficiency, because “documentation artifacts were designed to communicate, not to gate a deployment.” The same paper builds the other half of the snapshot regime, a pre-deployment suite that does gate, at a measurable cost in pipeline time.

Neither artifact answers the questions an investigator actually asks. A Model Card describes a model version in aggregate. A pre-deployment suite describes one rollout’s test results. When a dispute arrives, the questions are narrower and timestamped: which governance decision was made, was the instruction dispatched, did the action execute, did anything observe the resulting state. Snapshot evidence can supply background, and even work on auditing model internals targets the model rather than any single decision. Nothing sampled at review time can testify about one decision at runtime.

One decision, four events

AIREP’s core move is a taxonomy. The text states the problem directly: “Runtime-governance evidence is often collapsed into one audit event even though materially different events may have occurred: a governance decision may have been made, an instruction may have been dispatched or received, an action may or may not have executed, and a resulting state may or may not have been observed.”

Split those out and each carries a different truth condition. A governance decision is a policy judgment: the runtime chose to allow, restrict, or route an action. Dispatch and receipt are message-level facts: the instruction left the governor and arrived at the executor. Execution is a side-effect fact: the action ran, failed, or never started. Observed state is an epistemic fact: something checked the world afterward. Collapse them and the log cannot distinguish an approved-and-executed action from one that was approved, dispatched, and silently failed. Suppose a refund agent’s guardrail approves a deletion instruction, dispatch succeeds, execution errors out, and nothing observes the result. A single “governed” event reads like a clean run; four events show approval without completed effect.

The scope is deliberately narrow. The protocol says it “addresses a narrower question: how can evidence about an individual governed AI-runtime decision and its subsequent control, execution, and observed-effect lifecycle be represented.” It is not a replacement for Model Cards or OpenTelemetry traces; it is the connective record between them. The granularity instinct parallels a shift already visible in authorization design, where per-action authorization grants move the approval event from install time to the moment of action.

Why NOT_RUN must not become PASS

The taxonomy says what to record. A second rule says how missing records read: “A measurement declared NOT_RUN cannot become PASS; missing evaluation evidence remains NOT_MEASURED rather than success.”

Most log schemas treat silence as success. If a safety check never ran, no line is written, and downstream readers infer a pass. Fail-closed vocabulary forbids that inference: the states are named, sticky, and negative by default. This matters most in exactly the scenario governance cares about, because a flaky pipeline or a hostile actor that suppresses evidence otherwise looks identical to a clean run. The decision whose evidence went missing is usually the one under investigation, and a schema where absence reads as success makes the investigation unwinnable by construction. Note the boundary here: this is a requirement inside a preprint’s profile, not something any of the surveyed tooling is shown to enforce.

What a working implementation looks like

The building blocks are ordinary. A Show HN post describing an open-source Article 12 logging library for Node applications on the Vercel AI SDK lays out a concrete pattern. The library “wraps the model in middleware, automatically logs every inference call to structured JSONL in your own S3 bucket” (Show HN post) and “chains entries with SHA-256 hashes for tamper detection, enforces a 180-day retention floor, and provides a CLI to reconstruct a decision and verify integrity.”

Four properties matter more than the stack. Capture happens in middleware, so logging is not left to each caller. Storage is deployer-owned, which keeps the evidence with the party that will be asked to produce it. Entries are hash-chained, so later tampering is detectable rather than invisible. And reconstruction is a first-class operation with a CLI, not an archaeology project. The goal sits adjacent to work on reversible execution traces, which records agent runs as editable, replayable structures for debugging; both treat the run record as an object with operations, rather than as exhaust.

Two caveats keep this honest. The library logs inference calls, and the sources reviewed do not show it emitting AIREP’s four distinct event types, so it demonstrates the mechanics of per-decision logging rather than protocol compliance. And the 180-day floor is that project’s enforcement choice. The post does assert that Article 12 requires “automatic event recording and six-month retention for high-risk systems,” so the author reads that floor as a statutory duty; the reading is the poster’s, not something the sources reviewed establish, so treat the number as a design precedent, not law.

What per-decision evidence buys in practice

Phoebe, an SRE incident-investigation finite state machine for LLM agents served over MCP, shows the payoff in code rather than argument. Its conclude action is gated: “phase must be verify, you need evidence from ≥2 distinct telemetry backends, and at least one probe must have run during the verify phase” (the project). An investigating agent cannot close an incident on a single telemetry source or without an active check during the verification phase. Whatever one thinks of the specific thresholds, the structure is the point: conclusions are gated on classes of evidence, and the gate is mechanical rather than judgment left to the model.

Map that discipline onto the three requests that actually arrive. Incident investigation: reconstruct the decision across all four events with verified integrity, and “did the guardrail fire or did the action just fail” becomes answerable instead of debatable. Dispute resolution: a customer claims the agent deleted their data, and the dispatch and execution records either support the claim or they do not. Regulator requests: produce the record for a specific decision rather than a statistical characterization of the model. None of the sources reviewed shows a regulator accepting such a record, so acceptance is an open question, not a demonstrated outcome.

There is also a quieter effect. Burden of proof shifting to operators is already the direction of agent governance generally: directories demand that agents prove identity, and logging duties demand that deployers prove behavior. Per-decision evidence is the technical complement, the record that makes the demanded proof producible.

Side by side: what each regime catches

The retention collision nobody has measured

The protocol’s weak point is not structure but volume. Per-decision records carry instructions and observed state at minimum, which is exactly the content privacy regimes worry about, and they accumulate per decision rather than per deployment. Lossy compression solves that by deleting the fields that make reconstruction possible, which breaks the audits the traces exist to support. The tension is real and currently argued rather than measured: none of the four sources quantifies the privacy exposure or the storage cost of per-decision retention, and no source reviewed reports a deployment that ran the arithmetic.

One partial technique appears inside AIREP itself. The protocol permits a platform-specific verification token to be hashed and preserved as platform evidence “without being reinterpreted as proof that a model is safe, aligned, independently evaluated, or AIREP-Authenticated.” Hashing lets a record carry the fact of a platform check without carrying the token’s claims, which is a minimization move. It is also a narrow one. Hash chaining detects tampering, and hashed tokens compress references, but neither makes a verbose payload private. If the payload is needed for reconstruction, integrity techniques do not substitute for a retention policy. The floor has to be chosen deliberately, and before copying anyone’s number, including the 180-day one above (Show HN post), the actual Article 12 retention obligations for your deployment need checking; the sources reviewed do not establish them.

Does runtime evidence replace the pre-deployment gate?

No, and the cost figures explain why the question arises at all. The compliance-pipeline paper reports that “the full pre-deployment suite adds roughly 45 minutes to the chatbot’s pipeline, 32 of them in Robustness Testing, which can run asynchronously behind a deployment hold for latency-sensitive rollouts” (arXiv:2609.20016). Forty-five minutes per rollout is a real cost, and the async pattern is the standard mitigation. Runtime logging does not remove it, because the two layers answer different questions: the gate asks whether to ship, the record asks what shipping did. A team that dropped pre-deployment review in favor of runtime evidence would trade a bounded, schedulable cost for unbounded retrospective exposure, with nothing in the sources reviewed suggesting any auditor would accept that trade. What logging costs in request latency is likewise unquantified in the sources reviewed; the visible cost axis is retention, not runtime overhead.

The minimum event set worth logging

The synthesis, assembled from these four sources rather than from any adopted standard: log every governed agent decision as distinct events.

• One governance-decision record: which policy or actor decided, and what it decided.
• Dispatch and receipt records for the instruction, so message-level facts survive separately from intent.
• An execution record with an explicit outcome, including failed and never-started as first-class states.
• An observation record for the resulting state, or an explicit marker of its absence.

Give every status field fail-closed vocabulary, chain entries cryptographically so tampering is detectable, keep a path that rebuilds a single decision end to end, and set a retention floor you can defend in writing. Pair all of it with asynchronously-gated pre-deployment review rather than replacing it. Each source contributes one piece: AIREP the taxonomy and semantics, the pipeline paper the cost of the gate, the logging library the capture pattern, Phoebe the conclusion-gating discipline.

What this evidence cannot settle

AIREP is an author-reported preprint, with no independent adoption, benchmark, or regulatory endorsement anywhere in the sources reviewed. The absence of adoption statements is not proof of non-adoption, but nothing here supports calling it a standard. The 180-day retention floor and the 45-minute suite figure each come from a single project or case study. The open-source tooling demonstrates that per-decision logging is implementable today, not that it is legally sufficient or interchangeable with AIREP-style records. And the protocol’s own disclaimer bounds the entire argument: logged, hashed evidence supports reconstruction of what happened, and says nothing about whether the model was safe. That boundary belongs in front of every claim built on this foundation, because the moment a runtime record is sold as proof of safety, it stops being evidence and becomes marketing.