groundy
Models & Research

Gemini 4 Neon Ranks #2 and #3: Which LLM Leaderboard Should Drive Routing

Gemini 4 Neon ranks #2 and #3 on vendor-reported indices. Use cost-adjusted placement to filter candidates, not route traffic, unless frontier reasoning is required.

Published 7 references
On this page7 sections

At a fixed cost-and-latency ceiling, use leaderboard rank to filter candidates, not to route production traffic. For Gemini 4 Neon, the more useful signal is cost-adjusted placement at the thinking level your workload actually needs. Every Google capability score and cost delta cited here is vendor-reported from Google’s November 18 launch post: Neon ranks #3 overall with a score of 61 on the Artificial Analysis Intelligence Index at xhigh, and #2 on the Vals Index with Pareto-frontier placement. The separate Agent Duel numbers are an independent projection, not measured Neon battles. If your workload requires frontier reasoning, however, the recommendation changes: use the xhigh capability result as an eligibility filter first, and consider cost second.

That distinction prevents a common routing expense: paying for capability headroom that a workload never uses. Neon’s three launch verdicts are not necessarily contradictory. They answer different questions, at different settings, with different kinds of evidence.

Three verdicts, three different questions

Leaderboard verdictWhat it can tell a routerEvidence behind the Neon resultAppropriate use
Artificial Analysis Intelligence Index: #3, score 61, at xhighA raw capability ranking at one demanding settingVendor-reported in Google’s launch postFilter candidates for difficult reasoning workloads
Vals Index: #2, with Pareto-frontier placementA potentially useful quality-versus-cost trade-off signalVendor-reported; the supplied evidence does not describe the methodology, runs, or task conditionsShortlist for a cost-constrained trial, not automatic deployment
Agent Duel: projected Elo 1,066 versus Opus 5.5 at 1,078How speed can affect modeled win rate alongside model identityIndependent analysis, but explicitly projected from assumed Carbon-level performance and observed serving speedForm a hypothesis for latency-sensitive workloads; do not treat it as a measured Neon result

An Elo and an intelligence-index score are not interchangeable units. The first estimates relative performance under a win-rate model; the second produces a capability score under its own evaluation design. Nor does a Pareto frontier simply mean “second-best model.” A frontier preserves trade-offs between objectives rather than collapsing them into one order: among the compared options, improving one objective requires giving ground on another.

That makes frontier placement relevant when cost or latency is constrained. It does not make the placement self-validating. In Neon’s case, no fetched source independently confirms the Vals frontier, and the launch post does not supply enough methodology to audit what the placement represents.

Thinking level is part of the model configuration

Neon exposes four thinking levels, which Google says let teams adjust intelligence and token spend by workload. Its headline scores should therefore be read as results for model-plus-setting combinations, not as one invariant property called “Neon.”

The distinctions matter:

  • The #3 Artificial Analysis ranking and score of 61 are vendor-reported at xhigh.
  • The default high setting carries Google’s claim of 32% lower cost per task than Gemini 4 Carbon.
  • At high, Google also reports Pareto-frontier placement on both the Vals Index and Agent Arena, with roughly 50% to 80% lower cost per task than top competing frontier models.
  • At xhigh, Google reports Neon’s highest intelligence score and a cost per task at least 36% below competing models at their highest setting.

Combining the #3 xhigh rank with the 32% high-setting saving would produce a neat launch summary and a bad routing assumption. A team running high has not purchased the measured xhigh configuration, while a team that needs xhigh cannot assume the default setting’s cost and quality trade-off applies unchanged.

Evaluation conditions extend beyond thinking level. Google’s vendor-reported SWE-sweep result is 39.0% in Google Antigravity versus 12.8% on the mini-SWE harness. Those figures should not be averaged or treated as interchangeable estimates of coding ability. They illustrate why a routing evaluation needs to hold the harness and configuration steady before comparing models.

What the independent speed reading changes

The most useful independent observation comes from micjm’s launch-day analysis: Neon was being served through OpenRouter at 127 tokens per second, compared with 46 tokens per second for Carbon. The author characterizes Neon as an incremental benchmark improvement with a very large speed improvement.

That is one serving observation, not a durable latency guarantee. It nevertheless matters because the Agent Duel modeling finds speed roughly as important as model identity in determining win rate. A faster model with similar answer quality can therefore close a preference or Elo gap without establishing superior reasoning.

The projection proceeds by assuming Carbon-level performance at the observed 127 tokens per second. It produces an Elo of 1,066, just below Opus 5.5’s 1,078 snapshot. The uncertainty is more informative than the apparent closeness: Neon’s projected win rate against Opus is 48.3%, counting ties as half a win, and the 90% interval for the projected Elo difference runs from -94 to +68. That is statistically a coin flip, not evidence of Neon dominance or a measured equivalence result.

For an interactive coding or support session, the speed weighting may reflect something users genuinely notice. For a batch extraction job, immediate token throughput may matter less than successful completion within a cost ceiling. The Agent Duel projection is consequently a prompt to measure the right workload property, not a universal ordering of the three models.

When should frontier placement beat raw rank?

Frontier placement should outrank raw capability rank when four conditions hold:

  1. The workload has a firm cost or latency ceiling. A model outside that ceiling is not a viable default, however high its capability score.
  2. A cheaper configuration already clears the task’s quality requirement. Additional benchmark headroom has no routing value if it does not improve completed work.
  3. The comparison uses consistent settings and evaluation conditions. Thinking levels, harnesses, and task definitions cannot be mixed to obtain the preferred answer.
  4. The frontier’s cost and latency assumptions resemble the deployment. Otherwise, the placement is a shortlist signal rather than a transferable result.

Research on cost-aware selection supports the second condition. In a study of text-classification trade-offs, the authors report that for deterministic, fixed-label ontology-style classification, scaling to large generative models did not improve outcomes while substantially increasing latency uncertainty and operating cost. They recommend Pareto-frontier projections and a parameterized utility function rather than one combined score.

That finding does not show Neon will win a classification workload. It shows why rank-first selection can waste money when the task is already inside a cheaper model’s competence. The exception is equally important: if only an xhigh-class configuration can complete the hardest reasoning step, raw capability becomes the gate. A cheaper frontier position cannot compensate for failed work.

A worked reading of Neon, Carbon, and Opus 5.5

Consider a hypothetical team with a fixed per-task budget and an interactive latency target. Assume its own evaluation finds that Neon at high, Carbon, and Opus 5.5 all clear the required quality bar. That quality result is an assumption for the example, not a finding from the launch sources.

Under those assumptions, Neon’s #3 xhigh rank should not decide the route. The team would start by testing Neon at high because Google reports a 32% per-task cost advantage over Carbon at that setting, while the launch-day OpenRouter observation showed substantially higher serving speed. Those figures justify a trial order, not a production guarantee. The team still needs its own cost per successful task and latency measurements under load.

Opus 5.5 cannot be eliminated from this comparison using the Agent Duel projection. The supplied evidence contains no comparable Opus cost-per-task figure, and the projected 48.3% Neon win rate does not establish a quality winner. If the team’s tasks resemble the Agent Duel setting, the projection makes a head-to-head trial reasonable. It does not supply the result of that trial.

Now change the assumption: suppose the evaluation finds that only xhigh-level reasoning reliably completes the work. The routing choice should reverse. Neon’s xhigh capability rank becomes relevant to eligibility, and Google’s vendor-reported claim of at least 36% lower cost per task than competitors at their highest settings becomes useful only among configurations that pass. The correct rule is therefore conditional: frontier placement beats rank after the quality bar is met; rank regains priority when capability itself is the binding constraint.

Why routing research favors a budget-weighted utility

The launch numbers become more useful when placed inside a selection policy rather than treated as a final answer.

ProgRouter finds that greedy policies which always select the strongest model can exhaust their budgets early, while conservative policies can underuse strong agents. It also reports an unfavorable quality-cost trade-off for single-model policies in the multi-agent workflows it studies. The routing lesson is not to choose the weakest or cheapest model by default. It is to reserve expensive capability for queries where it changes the outcome.

Scope formalizes that approach. It predicts whether a model will answer a query correctly and estimates token cost, then maximizes a utility function weighing accuracy against cost according to the user’s budget preference. This is the operational version of knowing your ceiling before reading a leaderboard: the same model can have different utility for two teams even if its rank is identical.

Deployment conditions add another constraint. RouterWise reports that each model’s latency is a non-linear function of allocated resources and of the request load induced by the routing policy itself. A model can therefore be the right choice under one deployment and the wrong choice under another. A router optimized against fixed leaderboard latencies can produce slow, infeasible, or suboptimal assignments once deployed.

Cost is similarly conditional. In a concurrency-aware costing study, the authors report effective costs from $0.21 to $15.25 per million output tokens on identical H100 hardware, depending on load. They also caution that published serverless per-token prices are not conditioned on a production service-level objective, with no enforceable per-request p99 guarantees for time to first token or inter-token latency. Google’s per-task savings claims may be accurate under their test conditions, but those conditions are not supplied, so the percentages cannot be transferred directly into a production budget.

The routing decision, with the evidence limits attached

I would use Neon’s frontier and cost claims to decide what to trial first at a fixed ceiling, and its xhigh rank to decide whether it belongs in a frontier-reasoning trial. I would use the Agent Duel projection to prioritize a speed-sensitive comparison with Opus 5.5, not to declare a winner.

The durable routing record is a small set of measurements from your own workload: quality by thinking level, cost per successful task, latency under expected load, and the point at which a stronger configuration improves completion rather than merely increasing headroom. The routing papers support that framework, but their different model pools and benchmarks do not provide Neon-specific performance evidence.

As of November 20, 2026, every Neon launch score in this comparison remains vendor-reported, the Agent Duel result remains a projection whose uncertainty interval spans zero, and the Vals methodology and Pareto placement lack independent confirmation in the supplied evidence. Those limitations do not make the leaderboards useless. They define their proper job: narrowing the experiment, while your workload determines the route.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. Introducing Gemini 4 Neonblog.googleAccessed
  2. Gemini 4 Neon looks like a top modelmicjm.comAccessed
  3. Text-classification trade-offs studyarxiv.orgAccessed
  4. ProgRouterarxiv.orgAccessed
  5. Scopearxiv.orgAccessed
  6. RouterWisearxiv.orgAccessed
  7. Concurrency-aware costing studyarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy