Gemini 4 Argon's #1 Vals Index Claim: What the Benchmark Tests

Google’s launch-day claim checks out on the leaderboard, and it still should not change your routing. On October 13, 2026, Google’s Gemini 4 Argon launch post reported Argon “ranking #1 on the Vals Index, which measures model performance across critical economic sectors like finance, coding, legal, and tax work.” The administrator’s own board backs the standing: Vals AI’s leaderboard shows Gemini 4 Argon first at 68.90% as of October 13, 2026, ahead of Claude Sonnet 5.5 at 67.04% and Claude Opus 5.5 at 66.97%.

The ranking is real. The question is what it ranks. Read closely, the Vals Index is a GDP-weighted composite in which finance and coding carry roughly 89% of the weight, six of its eight component benchmarks are private, the whole index was re-versioned 18 days before Argon’s launch, and the margin over second place is smaller than the swing a single scoring rule applies to one of the runner-ups. That combination makes the #1 a useful screening signal for finance- and coding-heavy agentic work, and a poor basis for routing legal, tax, or anything else on its own.

What the Vals Index actually measures

Vals describes the index as “A single measure of AI’s potential economic impact — agentic model performance across finance, coding, legal, and tax tasks, weighted by each sector’s share of U.S. GDP.” The weighting is explicit:

Vals_Index = (8.0 × Finance + 5.6 × Coding + 1.2 × Legal + 0.5 × Tax) / 15.3

Simple arithmetic on the published formula: finance is about 52% of the weight, coding about 37%, legal about 8%, tax about 3%. This is a deliberate design choice, not a flaw. If you believe model procurement should track where the money is, weighting by GDP share is a defensible way to build one number. But it means the index answers a specific question, “which model performs best across the economy’s biggest knowledge-work sectors,” and not the question most buyers actually ask, which is “which model is best for my workload.”

A legal-first buyer reading that Argon ranks #1 on the Vals Index is reading a composite in which legal work determines 1.2 of 15.3 points of weight. Argon could hypothetically trail competitors on the legal component and still top the index on finance and coding strength. Vals publishes the formula openly, which is to its credit; the burden of noticing what the formula implies falls on the reader.

Inside the harness: mostly private, with a scoring footnote that matters

The index aggregates six private and two public benchmarks, which Vals frames as providing “high signal into the real-world tradeoffs between capability, latency, and cost of deploying AI systems.” Private components have a real advantage against contamination, since unpublished test data cannot leak into training corpora as easily. The cost is verifiability: buyers cannot independently run six of the eight components to check a sector score. You are trusting the administrator’s harness, grading, and data hygiene for three-quarters of the composite.

The more instructive detail is a footnote on the leaderboard. Vals discloses that some Sonnet 5.5, Opus 5.5, and Fable 5.1 scores “include some component tasks served by a fallback model after a provider refusal; counting those as failures lowers them to 65.85%, 65.05% and 64.59% respectively, with the order unchanged. Gemini 4 Argon had no fallbacks.”

Fable 5.1 drops to 64.59% under the stricter rule as well; its headline score sits outside the top three and is not the comparison that matters here.

Look at the margin through that footnote. Argon leads Sonnet 5.5 by 1.86 points. The fallback rule alone swings Opus 5.5 by 1.92 points. In other words, one defensible scoring-policy choice, made by the administrator, moves a rival’s score by more than the entire gap between first and second place. Vals reports the order is unchanged either way, and Argon’s clean refusal record is genuinely informative for production routing, where a provider refusal mid-task is an operational event, not a rounding error. But a lead that fits inside one scoring rule is not the kind of lead that should survive contact with a procurement decision by itself.

The 18-day-old scoreboard

Timing sharpens the picture. Vals Index v2.1 shipped on September 25, 2026, eighteen days before the Argon launch. The update added a Tax sector (Tax Agent Bench at weight 0.5, moving the denominator from 14.8 to 15.3) and replaced Terminal-Bench 2.1 with Terminal-Bench 4.0 in the coding bucket. Per Vals, “Terminal-Bench 4.0 is substantially harder, so index scores are lower than under v2.”

Two consequences follow. First, any Vals Index number quoted before late September is not comparable to 68.90%, because the formula and one component both changed. If a competitor’s marketing or an older analyst note cites a v2 score, it is measuring a different index. Second, the v2.1 configuration is only 18 days old, so there is no long history of this exact setup to contextualize the spread between models. An earlier v2 revision, from August 13, 2026, had already removed SWE-Bench Verified from the coding bucket because, in Vals’ account, SWE-Bench had become saturated, and added Excel financial modeling, code migration (including COBOL), and two legal benchmarks.

The component construction also deserves a look. For Code Migration, Vals scores a fixed subset: 50 of 120 published CLI migration tasks plus all 10 COBOL tasks, weighted 75% CLI and 25% COBOL. That is a reasonable editorial choice, and it means the index’s code-migration number will not match the standalone benchmark’s published number. When you see a model’s Code Migration score quoted elsewhere, you cannot assume it is the number feeding the index.

None of this is an accusation. Versioned, weighted, curated indices are legitimate instruments. It is a reminder that an index is a product with a changelog, and a #1 claim inherits the changelog.

Why even honest test scores overstate competence

Step back from Vals specifically. The broader evidence on agentic coding benchmarks suggests that test-based grading systematically flatters models, which matters because coding carries 37% of this index.

The strongest adversarial result comes from SWE-ABS, which applies adversarial benchmark strengthening to SWE-Bench Verified and finds that “19.78% of accepted patches are semantically incorrect”; under strengthened tests, the top agent’s score falls from 78.80% to 62.20%. Patches that pass the provided unit tests turn out, on closer inspection, to be wrong at a rate no buyer would tolerate. Test-pass rate and task success are correlated, not identical.

Contamination is the second channel. SWE-Bench+ found that neither SWE-Bench Lite nor SWE-Bench Verified “addresses the solution leakage problem”: in 18 Lite and 37 Verified instances, the solution was posted directly in the issue description or its GitHub discussion, so the answer ships inside the benchmark’s own input, ready to be copied rather than solved. The same paper reports that over 94% of the full dataset’s issues were created before the studied models’ knowledge cutoffs, which it frames as potential data leakage. And SWE-bench-Live documents the staleness side: SWE-bench and its derivatives “have not been updated since their initial releases,” and because models train on massive inscrutable corpora, static datasets sit at persistent risk of having been included in training data.

These papers examine SWE-bench-family benchmarks, not the Vals Index components directly, and that distinction matters. Vals’ stated reason for dropping SWE-Bench Verified was saturation (“SWE-Bench has become saturated”), and its stated purpose for private components is real-world signal rather than contamination control; whatever contamination resistance the private components gain is a side effect, and reading it as a design motive would be my inference, not Vals’ claim. Either way, it would be wrong to import the 19.78% figure as an estimate of error inside the Vals Index. What the literature establishes is directional: test-based, static, public benchmarks inflate and decay, and any index containing public, test-graded components inherits some of that risk at an unknown rate. The private components avoid contamination but are not auditable for grading inflation from the outside. There is no configuration that escapes both concerns at once.

A useful frame comes from a 2026 position paper on perishable evaluation knowledge, which argues that evaluation scores should be treated as epistemic claims with formality, scope, and validity windows: results apply to the tested distribution rather than universally, and they expire as contamination accumulates and distributions shift. A launch-day index standing is a claim with a shelf life, and its scope is the administrator’s task distribution, not your workload.

How is this different from SWE-bench or AIME?

Academic benchmarks like SWE-bench and AIME test one capability on a public task set with a public methodology, and you can run them yourself. The Vals Index is a different instrument: an agentic, multi-sector composite, weighted to GDP, built mostly from private benchmarks, administered by one company, and versioned like a product. Neither is better in the abstract. The academic sets are reproducible but narrow and increasingly contaminated; the sector index is broader and fresher but rests on the administrator’s self-reported methodology.

That last point is the strongest limitation of this analysis, and of the #1 claim itself. Everything established about the index’s construction, components, weights, and scoring policy comes from Vals AI’s own methodology page. The page does publish more than rankings: cost and runtime per model, where Argon at $15.68 per test in 46m33s undercuts Sonnet 5.5 ($21.34, 1h18m) and Opus 5.5 ($32.14, 1h19m). It also names component leaders, among them Argon at 65.40% on Finance Agent v2; what it does not publish is each model’s full component breakdown, so Argon’s standing on the legal and tax components is not visible on the board. As of October 15, 2026, no independent evaluation of the Vals Index is cited here, and there is no evidence either way about whether Google has any involvement in the index. That absence is a gap in the sources, not proof that none exists. The #1 rests on a single administrator-run leaderboard, quoted by the vendor that benefits from it.

There is also a rubric-fit problem at the sector level. A single legal score assumes legal work is one thing. PersonaMatrix, studying legal summarization evaluation, finds that “prevailing task-based evaluators overlook divergent user and stakeholder needs.” A compliance team, a litigator, and a client reading the same summary need different things from it; one composite number cannot arbitrate between them. That is an argument about legal evaluation specifically, but it generalizes to any heterogeneous professional domain compressed into a sector score.

When should a sector ranking override your own evals?

Almost never, but the exceptions are real:

• When you have no evals and need a prior. If you are standing up a finance or coding agentic workflow for the first time, an index weighted roughly 89% toward those sectors is a reasonable starting shortlist. Screening is what it is built for.
• When the operational details match your constraints. Argon’s zero-fallback run is decision-relevant if your workloads cannot tolerate provider refusals mid-task. That is arguably more useful than the headline score.
• When your workload is misweighted in the index. Legal and tax together carry about 11% of the formula. If that is your workload, the index is mostly measuring someone else’s problem, and a sector ranking should not override even a modest internal eval.

What the ranking cannot tell you: whether Argon is best at your tasks, or how its component scores distribute. The board names component leaders only, and those leaders are informative in themselves: Argon, first overall, leads just one of the eight components (Finance Agent v2 at 65.40%), while Claude Sonnet 5.5 leads two coding components and Claude Fable 5.1 leads Tax Agent at 77.64%. A model can top a weighted composite without leading the component you care about, and the disclosed leaders show exactly that here. What the board does publish is cost and runtime, on the administrator’s own numbers: Argon at $15.68 per test and 46m33s, cheaper and faster than either runner-up. Treat those as reported figures to check against your own pricing and workloads.

A checklist for any launch-day benchmark claim

The Argon launch is a good occasion to build the habit, because sector-branded indices put the verification burden on buyers: the claim arrives with the launch post, and independent replication lags by weeks or never arrives. Five questions, reusable for the next one:

1. What is the task construction? Public or private components, who grades, and can you run it yourself? Here: six private, two public, administrator-graded.
2. What is the weighting, and does it match your workload? A GDP-weighted index is a finance-and-coding index with legal and tax attached.
3. How sensitive is the result to scoring policy? Ask what refusals, fallbacks, and partial credit do to the ranking. Here, one rule swings a rival by more than the winner’s margin.
4. How fresh is the version? This ranking sits on an index re-weighted 18 days earlier; cross-version comparisons are invalid.
5. Who administers it, and who benefits? The methodology is self-reported by Vals in every particular; nothing establishes Google’s involvement or independence either way. Treat vendor-quoted, administrator-run standings as reported claims until a third party replicates them.

Verdict

Treat Gemini 4 Argon’s #1 Vals Index standing as a screening signal for finance- and coding-heavy agentic workloads, not a routing decision. The standing is confirmed on Vals’ own leaderboard, the zero-fallback run is a genuine operational data point, and the same board’s cost and runtime columns favor Argon: $15.68 per test in 46m33s, against $21.34 and $32.14 for the runners-up. The index’s GDP-weighted design is transparent. Against that: the 1.86-point margin is smaller than one scoring rule’s effect on a rival, six of eight components are not buyer-reproducible, the index was re-versioned less than three weeks before launch, and, as of October 15, 2026, no independent evaluation of the Vals Index is cited here. If Argon is on your shortlist for finance or coding agents, it earned the eval slot. Run your own tasks, weight them by your workload rather than the U.S. economy, and let the index do what indices are for: telling you where to look first, not what to buy.

Frequently Asked Questions

What is the exact formula for the Vals Index weighting?

Vals_Index = (8.0 × Finance + 5.6 × Coding + 1.2 × Legal + 0.5 × Tax) / 15.3

What are the cost and runtime figures for Gemini 4 Argon compared to runners-up?

Argon at $15.68 per test in 46m33s undercuts Sonnet 5.5 ($21.34, 1h18m) and Opus 5.5 ($32.14, 1h19m).