Should AI Agents Prepare Tax Returns? Reading Dayjob's 10.8% Result

A tax firm deciding whether an AI agent may draft returns or client advice should not start from the 10.8% pass rate Surge AI reported for its new DAYJOB: Tax benchmark. Start from the condo. In the vendor’s previewed scenario, an agent answered the client’s actual question correctly while never, across five attempts, raising a deadline that could quietly forfeit an exclusion worth up to $250,000 (vendor’s announcement). That failure shape, an omission that reads as finished work, is what changes the control design for any firm considering AI-assisted preparation. A wrong number on a workpaper is findable. An issue the agent never raised requires the reviewer to redo the professional judgment the agent was supposed to supply, which is exactly the cost the deployment was meant to avoid.

One caveat up front, because it governs everything below: every DAYJOB number cited here is author-reported by Surge AI, the benchmark’s own vendor. The grading rubric, pass definitions, and environment difficulty are not independently verified. And the legal-duty layer, Circular 230 due diligence, IRC 6694 preparer penalties, state board rules, appears in no source reviewed for this article; this piece is about evaluation and control design, and the regulatory analysis must be sourced from primary authorities before any firm acts on it.

What DAYJOB: Tax measures, and what 10.8% actually is

According to Surge AI’s announcement, the benchmark contains 50 environments across eight areas of tax practice. Agents receive client requests and supporting records, then produce tax returns, workpapers, technical reviews, or client advice. It is the third release in the DAYJOB family, after Healthcare and Finance, and its stated target is not tax knowledge in isolation but assignments involving “decisions that aren’t fully specified in the initial instructions.”

The headline scores are low, and the scoring rule explains part of why:

The healthcare and finance figures come from the benchmark family’s academic companion paper, which covered 30 model configurations from 13 developers. Two things follow. First, the tax number is a pass@1 score at a deliberately strict bar. As the announcement puts it: “We report pass@1 at a 95% rubric threshold. An attempt passes when it satisfies at least 95% of the task’s criteria, which assess calculations, supporting evidence, consequential issues, and the completed deliverable.” Low scores partly reflect that stringency, not absolute incapacity. Second, the sibling-domain scores are not tax-comparable, and they sit at a different bar: the paper’s headline metric is strict pass@1, where an attempt passes only if it meets every criterion, while the tax score already allows an attempt to miss up to 5% of criteria under the vendor’s threshold. The tax number is lower even with the more forgiving rule. A 24.7% healthcare pass rate, the paper’s headline result for Claude Opus 5.5, says nothing about whether an agent will catch an expiring exclusion window.

Note also what the rubric already concedes: “consequential issues” is a scored criterion. The vendor knows the risk class exists. The question is whether any aggregate score can tell you how a specific agent handles it.

The condo-sale case: the question answered, the deadline never raised

Here is the scenario as the vendor describes it. Anna moved out of her condo in August 2024 and asked whether closing in October 2026 or January 2027 would be better for taxes. An agent reviewed both options and correctly concluded that either date would satisfy the two-of-five-years residence requirement for the exclusion. But because Anna moved out in August 2024, she stops satisfying that requirement around the end of August 2027; a delayed sale beyond that point could lose eligibility entirely. The agent never said so.

The vendor’s summary of the run: “Across five attempts, Astra never identified the deadline. Opus identified it in all five attempts.” (Astra and Opus are the vendor’s names for the two agents in this case study.)

Both halves of that sentence matter, and neither permits the comfortable reading. Against blanket skepticism: the omission is agent-specific, not universal. Opus found the issue every time, so “AI cannot do professional judgment” is not supported. Against deployment confidence: the same vendor page reports a second case in which Opus itself was the one that missed, assuming a separating couple was unmarried across all five attempts while Astra flagged the need to verify marital status in four of five. Model choice shifts which omissions you get; it does not eliminate them. There is no model on the evidence here that a firm could designate “the safe one” for issue spotting.

Omission is the pattern, not the outlier

The companion paper documents the same failure shape in finance, and it is worth reading closely because it shows the omission coexisting with otherwise strong work. In a fund-reporting task, the paper reports that GPT-5.6 Sol (max) “found both pricing-hierarchy exceptions, which net to a $132,000 understatement, and recommended holding the reports. It did not question the Nerovexa price, and its final message stated ‘No other release blocker was identified,’ leaving the $25.7 million overstatement in place.”

That is the control problem in one sentence. The agent did real analytical work, caught real errors, made a defensible recommendation, and then affirmatively declared the review complete while a larger error sat untouched. A reviewer checking the agent’s stated conclusions would find them all correct. The failure is only visible to someone who re-derives the issue list independently, from the raw records, without reading the agent’s work first.

This is an inference from the evidence, not a measured finding on live engagements: omission-type failures are structurally harder to review than commission-type errors, because nothing in the deliverable points at the gap. The agent’s output is complete, coherent, and signed off internally by its own final message. The companion paper’s case and the vendor’s condo scenario are consistent with that account, but two documented cases from one benchmark family establish a risk class, not a frequency.

Why tax punishes omissions more than most domains

There is a structural reason to expect omission risk to be acute in tax specifically. Research on metamorphic testing of tax preparation software describes a US tax code of nearly 9,000 pages in all: over 2,600 pages of statutes, with IRS regulations, rulings, and clarifications making up the rest. The paper also quantifies the dependency problem: “97% of references in the Internal Revenue Code (IRC) are internal. Internal dependencies can lead to legal avoidance, when the tax code is not adhered to as intended and can occur when the IRC is misinterpreted.”

An eligibility rule that expires because its conditions sit behind cross-references elsewhere in the code is the canonical tax omission. The condo case is a small, clean instance; real engagements have more places for the dependency to break.

Nor can a firm outsource the completeness check to the model’s own ability to generate verification logic. Work on few-shot generation of metamorphic specifications for tax software found “the model makes between 1-2 mistakes per property on average when generating predicates, variables and operators.” If the checks themselves carry that error rate, “have the agent verify itself” is not a control.

There is a constructive counterpart. The TenForty framework, presented at the 14th Annual IRS/TPC Joint Research Conference, characterized 33 metamorphic specifications across five domains of the US individual income tax return: properties that must hold regardless of input, which can be tested mechanically. That is a template for firm-built, per-assignment-type checks that do not depend on any vendor rubric. It is also, notably, an existence proof that independent verification methods for tax logic are a live research area rather than something a firm must invent from nothing.

The reviewer is the weakest control in the loop

Suppose a firm accepts that review must include independent issue discovery. The next problem is whether the reviewers retain the skill to do it. A design-space analysis of agent systems cites independent research (Shen and Tamkin, 2026) finding that developers in AI-assisted conditions score 17% lower on comprehension tests. That evidence comes from software development, not tax review, so it should be treated as a warning, not a measurement of your staff. But the mechanism transfers uncomfortably well: the skill most likely to atrophy under AI assistance is precisely the skill required to catch what the assistant left out. A firm that routes all first-pass judgment to an agent is training its reviewers out of the competence its control design depends on.

Then there is the signature. Research on governance of multi-agent AI systems frames the general problem: “it is often unclear who should be held responsible – the model developers who created the foundation model, those who fine-tuned and deployed agents, the platform operators who orchestrated interactions, or the agents themselves (which lack legal personhood).” How tax practice resolves that question, who signs what and what due diligence requires of them, is precisely the legal layer absent from the sources reviewed here. It has to be answered from primary authorities before an agent drafts client work, not assumed in either direction.

What to require before an agent touches client work

The evidence supports a specific control posture. None of it requires rejecting AI assistance outright; all of it rejects treating any vendor score as a deployment case.

1. Evaluate consequential-issue spotting as its own axis, per assignment type. Surge AI’s rubric folds consequential issues into an aggregate pass/fail at a 95% threshold. That aggregate cannot tell you whether an agent surfaces unraised issues on, say, partnership loss analyses specifically. The condo and marital-status cases show the same agents succeeding and failing on this axis depending on the scenario. Run your own omission-focused evaluation on each assignment type you intend to delegate, with seeded issues the client never mentions, before approving that type.

2. Redefine sign-off as independent issue discovery, not conclusion-checking. The $25.7 million fund case, reported in the companion paper, shows an agent’s stated conclusions can all be correct while the deliverable is wrong. A review that verifies what the agent wrote will pass that deliverable every time. The reviewer needs to derive the issue list from the source records first, then compare. This is the expensive control, and it is worth being honest about what it costs: it recreates a substantial share of the first-pass judgment the agent was bought to perform. For assignment types where omissions dominate the risk, that cost may erase the economic case. Better to learn that in a pilot than in a malpractice claim.

3. Set firm-level pass thresholds per assignment type before citing any vendor score. An author-reported 10.8% pass rate at the vendor’s 95% rubric bar on 50 environments of unknown difficulty mix is marketing context, not a capability statement. Decide internally what pass rate, on your own evaluation, makes delegation of each assignment type rational given your review costs, and hold vendors to your measurement rather than adopting theirs.

4. Protect reviewer skill deliberately. Given the comprehension finding from AI-assisted development work, rotate staff through unassisted engagements and treat issue-spotting drills as maintenance of a control, not as training overhead.

I would run any pilot on assignment types where a missed issue is recoverable: internal workpapers, draft analyses with mandatory second-preparer review, work that can tolerate a restart. Client-facing advice on irreversible decisions, like Anna’s closing date, belongs at the end of the queue, after the per-type evaluation has shown the agent catches what the client does not ask about. That ordering follows from the failure shape; it is my recommendation from this evidence, not a measured result.

What this benchmark cannot tell you

Three limits should travel with any use of these numbers. Every figure is author-reported by the benchmark’s own vendor; the rubric and pass definitions have no independent verification, and 50 environments from one family cannot support firm-level capability claims in either direction. The documented omissions are agent-specific and scenario-specific, so they argue for evaluation, not for a verdict on any named model. And the legal frame, what due diligence requires of a signing preparer, what penalties attach to an understated position, what state boards expect of supervised AI use, is entirely absent from the sources reviewed here. This article tells you how to evaluate the agent. What the law requires of the human who signs is a separate research job, and it should be done before the pilot, not after.