A survey posted to arXiv as arXiv:2605.23955, “From Accuracy to Auditability: A Survey of Determinism in Financial AI Systems,” surfaced in research feeds on 2026-08-28, two days after its v4 revision landed on 2026-08-26, with an uncomfortable finding for compliance teams: temperature 0 controls only the sampler, while the reproducibility failures that matter in financial AI live in hardware, batching, and parallelism configuration below it. The practical consequence is that a greedy decode config does not produce auditable outputs, and audit evidence has to be built at the pipeline level instead.
What actually breaks in financial AI reproducibility?
The survey’s core contribution is a failure taxonomy across the three modalities that dominate financial AI, and its central claim is that nondeterminism in these systems is mechanical, rooted in hardware and architecture, rather than a simple consequence of stochastic sampling.
That reframing matters because most internal guidance on reproducibility was written for the sampling story. Set a seed, fix a temperature, containerize the environment, and the model becomes repeatable. The survey’s taxonomy says each modality breaks somewhere else:
Tabular models fail at the explanation layer. The failure mode the survey names is post-hoc explanation variance: the model’s prediction may be stable while the ranked list of reasons attached to it is not. The authors ran first-party experiments on public financial datasets quantifying explanation rank instability in credit scoring, according to the preprint. For a credit decision pipeline, this is the uncomfortable case. The approve or decline bit can reproduce perfectly while the reason codes, the part a customer dispute or an adverse-action review actually examines, shuffle between runs. An institution that validates only prediction stability has validated the part of the system nobody contests.
Graph networks fail through stochastic sampling and temporal asynchrony. Financial graph models, fraud rings, transaction networks, counterparty exposure graphs, are typically trained and explained on sampled neighborhoods of a much larger graph. Sampling introduces run-to-run variance by design, and temporal asynchrony adds a second axis: the graph itself changes between the time a model is trained, the time a decision is made, and the time an auditor reconstructs the decision. Reproducing a result requires reproducing the graph state, which is a data versioning problem as much as a model problem. The survey’s experiments reach this modality too: the authors measured prediction flip rates in GNN-based fraud detection on public financial datasets, per the abstract. That is a step past the tabular case. In credit scoring the decision bit can hold while the reasons shuffle; in the fraud-detection experiments the prediction itself varies between runs.
LLM-based agentic workflows fail through batch-dependent divergence and trajectory drift. These are the failure modes that make the temperature 0 shortcut look weakest, and they get their own section below.
The taxonomy is useful even before any of the survey’s numbers are trusted, because it assigns each failure mode to a layer of the stack. Explanation variance is a post-processing problem. Graph asynchrony is a data-state problem. Batch divergence is a serving-infrastructure problem. Teams that treat “the model is stochastic” as one problem tend to buy one control, usually a seed and a temperature setting, and that control addresses at most one row of the taxonomy.
Why isn’t temperature 0 the same as determinism?
Greedy decoding removes sampling randomness and nothing else; the divergence the survey measures happens in the arithmetic underneath the sampler.
The mechanism is worth stating precisely, because it explains why “we set temperature to 0” fails under re-run even when nothing about the model or the prompt changes. Floating-point addition is not associative. The order in which partial results are summed changes the last bits of the result, and the order in which a GPU sums partial results depends on how work is scheduled: how many requests are batched together, how kernels are tiled, how matrix multiplications are split across devices. This is documented framework behavior: PyTorch’s reproducibility notes warn that identical results are not guaranteed across platforms and hardware configurations, and the framework ships deterministic-algorithm controls because some CUDA operations are nondeterministic by default. Two runs of the same prompt with the same weights and the same decode config can land on slightly different logits because the batch they were served in had a different composition. Slightly different logits usually produce the same token. Occasionally, near a decision boundary, they do not. One different token early in a generation changes everything after it, because each token conditions the next.
The survey’s experimental analysis includes exactly this kind of failure: tensor-parallel-induced output divergence in LLM entity extraction, per the preprint. Tensor parallelism splits large matrix operations across multiple GPUs and recombines partial results through collective reduction operations. The reduction order is a property of the parallel configuration, not of the model or the prompt. Change the number of shards, or let a serving stack rebalance load, and the arithmetic changes while every user-visible setting stays fixed. Greedy decoding cannot help, because the nondeterminism never passed through the sampler.
Trajectory drift compounds this in agentic workflows. A single LLM call that flips one token near a boundary is a nuisance. An agentic pipeline, where each step’s output becomes the next step’s input, a plan, a tool call, a retrieved document, turns one flipped token into a branch. The second call now sees a different prompt, produces a different plan, calls a different tool. The survey names trajectory drift as a distinct failure mode of LLM-based agentic workflows, and it is the mode most relevant to the financial use cases currently being piloted: multi-step extraction, reconciliation, and research agents where the output is a chain, not a single answer. Auditing the final answer of such a chain without auditing the trajectory is auditing a coin flip’s landing without checking the toss.
The survey’s own framing, per its abstract on arXiv, is that mechanical nondeterminism sits in hardware and architecture. That is the sentence compliance teams should pin to the wall. The stochastic sampling story is the one vendors and tutorials teach because it has a one-line fix. The mechanical story has no one-line fix, which is precisely why it belongs in audit planning.
Which metrics can an auditor actually read?
The survey proposes a layered evaluation framework that links modality-specific divergence metrics, RBO, D_cos, TDI, and PSD, to audit readiness, rather than proposing a single reproducibility score.
The layered structure is the honest part of the proposal. A single aggregate reproducibility number would hide the taxonomy: a pipeline could be stable in prediction and unstable in explanation and still score well. Instead, the framework attaches a metric family to each failure mode. Per the survey, the named metrics are:
RBO, rank-biased overlap, is the natural fit for the tabular failure mode. RBO is an established measure of similarity between ranked lists that weights top ranks more heavily than deep ones and handles lists of different lengths. Applied to credit-scoring explanations, it quantifies the survey’s explanation rank instability directly: run the explanation pipeline twice, compute RBO between the two ranked reason lists, and a value well below 1.0 says the reasons a reviewer sees depend on which run produced them. The top-weighting matches the audit reality, since reviewers read the first few reasons and rarely the tenth.
D_cos, cosine distance, measures divergence in vector-valued outputs, which covers embeddings and any explanation or extraction output expressed as a direction in an embedding space. It is cheap to compute, interpretation is well understood, and it fits the graph modality where node representations are the object of interest.
TDI and PSD are named by the survey as further per-modality divergence measures, linked to audit readiness alongside RBO and D_cos, per the preprint. The abstract names both without saying what either measures, so the safe characterization ends where the abstract does: two more scalar divergence measures in the same family. Teams adopting the framework should read the preprint’s metric definitions directly rather than trusting any summary, including this one.
The operational pattern the framework implies is more valuable than any individual metric. Divergence is measured, not assumed. A team picks the metric matching its modality, runs the pipeline N times under production-like serving conditions on its own data, and records the distribution of divergence scores as audit evidence. That converts reproducibility from a claim in a model card into a number with a history. It also produces the awkward conversations early: an entity-extraction pipeline whose D_cos distribution widens when the serving cluster is reconfigured has a reproducibility problem that no decode config will fix, and it is better to learn that from a metric than from a regulator’s re-run.
What does each control actually pin?
No single control produces reproducibility; each one fixes one layer of the stack and leaves the others open, so controls have to be matched to failure modes.
The practical verdict that follows and the survey’s taxonomy combine into a mapping that a model-risk team can use as a scoping table:
| Control | What it actually fixes | What it leaves open |
|---|---|---|
| Temperature 0 / greedy decoding | Sampling randomness at decode time | Batch-dependent divergence, tensor-parallel reduction order, trajectory drift |
| Pinned random seed | Repeatable RNG draws (sampling, shuffling, graph neighborhood sampling) | Everything below the RNG: kernel scheduling, floating-point reduction order |
| Pinned model version or snapshot | Identical weights across runs | Serving-stack divergence; weight changes you cannot observe behind a hosted API |
| Fixed batch size and batching policy | Stable reduction order at serving time | Vendor-side batching you do not control; cross-hardware differences |
| Fixed tensor-parallel configuration and hardware | Parallelism-induced divergence, including the entity-extraction effect the survey measured | Layers above the hardware: prompts, retrieval state, agent trajectories |
| Per-modality divergence metrics (RBO, D_cos, TDI, PSD) | Detection: quantifies drift on the institution’s own data | Nothing, measurement detects divergence rather than preventing it |
Two rows deserve emphasis. The pinned-model-version row has a hole in it for API-hosted models: an institution can pin a version string while having no visibility into whether the weights behind the endpoint changed. Whether weights change behind a pinned version is not something an institution can observe from outside, so treat it as a procurement question rather than an engineering assumption: does the contract guarantee weight stability, and what evidence would show a change?
The metrics row is the other one teams underweight. Pinning controls are preventive, and every preventive control fails silently when the layer below it shifts. Measurement is what turns “we pinned everything we control” into “here is the divergence we observe in production, and here is its trend.” A reviewer can work with the second sentence. The first is an assertion.
How much weight does a preprint carry against regulatory text?
The survey (DOI: 10.48550/arXiv.2605.23955) is a moderated, non-peer-reviewed arXiv preprint, so its framework is engineering guidance with preprint evidentiary weight, and nothing in this article’s grounding establishes what any regulator requires.
This section exists because the distinction gets blurred constantly in vendor content and consulting decks, and the blur is expensive. A team that implements the RBO, D_cos, TDI, PSD framework and describes it to its validators as “what auditors expect” has overstated the evidence by at least one tier.
The verifiable facts about the repository are plain: arXiv is an open-access repository whose e-prints are approved for posting after moderation but are not peer reviewed, per Wikipedia’s arXiv entry. arXiv’s own about page confirms submissions are moderated for topicality and scholarly value and that material is not peer-reviewed by arXiv. Moderation filters for whether a submission is on-topic and plausibly scholarly. It does not check whether the experiments replicate. The repository hosts more than three million articles across eight subject areas, per the same about page, and the organization itself is mid-transition to an independent nonprofit after decades of partnership with Cornell University. None of that changes the moderation model.
On the regulatory side, the honest statement is a negative one. What an SR 11-7-style model-risk review accepts as validation evidence is a question this article does not answer; it makes no claim about what the Federal Reserve’s model-risk guidance or any other regulatory text requires, and links the guidance only so readers can check the source themselves. Any version of the survey’s framework presented as compliance doctrine should be treated with the same skepticism as a benchmark in a vendor whitepaper. Teams operating under formal model-risk management obligations should take the survey’s taxonomy to their validators as a proposal, not as a citation.
The first-party experiments the survey reports, explanation rank instability in credit scoring, prediction flip rates in GNN fraud detection, tensor-parallel divergence in entity extraction, are the authors’ own results, not yet independently replicated as of 2026-08-29. That does not make them wrong; the mechanisms involved are well understood in the systems literature. It makes them unconfirmed, and it means the right posture is to use the taxonomy as a hypothesis generator for your own pipeline testing rather than as established fact about your vendor’s stack.
What should a model-risk reviewer take from this?
Treat temperature 0 as a sampling control, not a reproducibility control, and treat reproducibility as a property of the pipeline that must be measured, not a property of the decode config that can be asserted.
The checklist that follows from the survey’s taxonomy, with its evidentiary tier attached:
- Classify the system by modality first. Tabular, graph, or LLM-agentic. The failure mode and therefore the control and the metric all follow from that classification. A credit-scoring model with post-hoc explanations lives in the first row; a multi-step extraction agent lives in the third.
- Pin what the failure mode actually lives in. Seeds and temperature for sampling; model snapshots for weights; batching policy and tensor-parallel configuration for serving-level divergence; graph state snapshots for temporal asynchrony. The table above is the scoping tool.
- Measure divergence on your own data, under production-like serving. RBO for ranked explanations, D_cos for vector outputs, and the survey’s TDI and PSD where they fit, per the framework. Divergence measured on a vendor demo dataset is marketing; divergence measured on your data, across re-runs and reconfigurations, is evidence.
- Record serving configuration, not just decode config. Batch policy, parallelism layout, hardware class. The survey’s entity-extraction result, parallelism-induced divergence that by construction sits below the sampler, is the argument for why this belongs in the audit trail.
- Label evidence tiers explicitly. Preprint framework, institutional measurements, and regulatory requirements are three different things. Conflating them is how audit findings happen.
The strongest limitation on all of this deserves to be stated flat. Every technical claim here rests on that single moderated, non-peer-reviewed preprint. The survey’s taxonomy is plausible, mechanistically coherent, and immediately useful as a testing agenda. It is not yet doctrine, not yet replicated, and not a substitute for asking your own pipeline the same questions.
That is, in the end, the answer to the title. Temperature 0 won’t save the audit because the audit was never about the sampler. It is about whether the institution can re-run the decision, explanation and trajectory included, and get the same answer for the same reason. The survey’s contribution is showing how many places that breaks below the decode config. What an institution does about it is engineering, measurement, and evidence labeling, and none of those fit in a decode parameter.
Frequently Asked Questions
Does the survey’s framework apply to small, single-GPU models?
The tensor-parallel divergence mechanism described in the preprint requires multi-GPU collective reductions, so single-GPU deployments are immune to that specific failure mode. However, the survey’s taxonomy still applies to tabular explanation variance and graph temporal asynchrony, which persist regardless of hardware parallelism.
How does this preprint differ from peer-reviewed reproducibility standards?
The survey is a moderated arXiv preprint, meaning it has not undergone independent peer review or replication verification. Unlike established regulatory texts or peer-reviewed journals, its metrics (RBO, D_cos, TDI, PSD) carry engineering guidance weight only and lack the evidentiary standing required for formal compliance citations.
What operational change is required for LLM agentic workflows?
Teams must shift from auditing only the final output to logging the full trajectory of intermediate steps, including tool calls and retrieved documents. This is necessary because trajectory drift, where a single token flip branches the entire workflow, cannot be detected by comparing final answers alone.
Can pinned model versions guarantee weight stability for API-hosted models?
No, pinning a version string does not prevent silent weight updates behind a hosted endpoint. The body notes that institutions cannot observe weight changes from outside the API, making this a procurement and contractual issue rather than an engineering control that can be verified via standard monitoring.
