GPT-6 Astra on ARC-AGI-3: What the Agentic Score Actually Measures

GPT-6 Astra’s ARC-AGI-3 score is real but not yet decision-grade: as of 2026-09-05, the only launch-era evidence reviewed here is a single community writeup reporting two scores for the same model, 63% and 99%¹, and no replication appears in the sources reviewed. What the benchmark prices is agentic problem solving inside a specific evaluation harness, and the harness configuration alone moves the number by tens of points. If you are routing agentic workloads, the score is a hint, not a gate.

What does an ARC-AGI-3 score actually measure?

ARC-AGI-3 measures interactive agentic problem solving: how much of each game an agent completes, scored as mean per-game RHAE with partial credit for incomplete solutions. That is a different quantity than the static puzzle accuracy that ARC-AGI-1 and ARC-AGI-2 reported, and conflating them is the first mistake most launch coverage makes.

The partial-credit mechanic matters more than the headline figure. In the most detailed public methodology available, a coding-agent system built on GPT-5.5 with high reasoning effort fully solved 15 of the 25 public ARC-AGI-3 games but posted a mean per-game RHAE of 58.12%², according to the executable world models paper. Read that again: 15 of 25 is 60% of games fully solved, yet the reported score is 58.12%². The number blends solved games with fractional progress on unsolved ones. Two systems with identical scores can have completely different profiles, one finishing a third of games cleanly and stalling on the rest, another grinding partial progress everywhere. For an operator deciding whether an agent can complete a workflow end to end, “games fully solved” and “RHAE” answer different questions, and only one of them is about completion.

The same paper gives the generational comparison that makes the score legible. GPT-5.4 with high reasoning effort, in an identical harness, fully solved 8 of 25 games for a mean RHAE of 41.29%². GPT-5.5’s jump to 15 of 25 and 58.12%² came from a model change with the harness held constant, which is exactly the comparison design that makes a benchmark number mean something. Hold that thought, because almost nothing about the GPT-6 Astra launch numbers satisfies it.

The harness itself is worth understanding because it defines what “agentic” means here. The evaluated agent maintains an executable Python world model of the game it is playing, verifies that model against its own observation history, refactors it toward simpler abstractions as a proxy for an MDL-style simplicity bias, and plans through the model before committing actions. The harness wraps this in a scripted controller, predefined world-model interfaces, verifier programs, and a plan executor. What ARC-AGI-3 prices, under this construction, is the ability to build a causal theory of an unfamiliar environment and act on a plan derived from it, rather than react step by step.

Why is the launch-day evidence narrower than the headlines suggest?

Every GPT-6 Astra ARC-AGI-3 figure in the sources reviewed here comes from a single community writeup dated 2026-09-04. No independent replication appears in those sources, and none of them addresses the contamination status of the benchmark items involved. Within the evidence available to this article, that is the entire base.

This matters more for ARC-AGI-3 than it would for a saturated benchmark, because the number is unusually sensitive to evaluation conditions. The community writeup, The AI Commit’s “Why GPT-6 Astra Has Two Scores on ARC-AGI-3: 63% and 99%”¹, documents the same model carrying two incompatible scores at once. A community source is not a peer-reviewed result and should be weighted accordingly, but the instability it describes is its own documented finding: identical weights span 97.3 points across effort settings on the neutral harness and 0.0 points on the provider harness, where every setting scored 100.0%¹. The specific percentages trace to one source; the configuration sensitivity is what that source exists to demonstrate.

There is also a scope problem in the available methodology. The detailed harness evidence in the record, fresh instances, leakage audits, world-model architecture, comes from GPT-5.4 and GPT-5.5-era coding-agent systems, not from GPT-6 Astra itself. The only GPT-6 Astra-specific datapoint in the sources reviewed here is the community writeup. The interpretation framework in this article generalizes; the specific numbers may not survive replication.

The right stance toward the 63% and 99%¹ figures is the one you would take toward any unreplicated launch-day claim: note them, do not budget against them. Replication by a disinterested third party, at a pinned configuration, is the event that would upgrade these numbers from launch context to selection input.

Why does one model post two scores?

One model posts two scores because ARC-AGI-3 performance is a function of the reasoning-effort setting and harness configuration as much as of the model weights: the community writeup reports a 97.3-point spread, from 2.7% to 100.0%¹, across effort settings in a neutral harness for otherwise comparable runs. A benchmark with a 97.3-point configuration sensitivity is not reporting a property of the model; it is reporting a property of the model plus its operating point.

This is the mechanical explanation for the 63%-versus-99%¹ puzzle, except the writeup’s own table makes the story messier than a compute dial. Reasoning effort controls how much test-time compute the model spends per decision, and on the neutral harness the knob moves the score by tens of points. It does not move it monotonically: ‘none’ scored 100.0%, ‘low’ 2.7%, ‘medium’ 8.3%, ‘high’ 17.0%, and the sweep returns to 100.0% only at ‘xhigh’ and ‘max’¹. The writeup flags the anomaly itself, noting that ‘low’ scores worse than ‘none’. More thinking per decision does not reliably buy a better agent on this harness, and the middle of the effort range is where the model scores worst. The operating point is doing at least as much work as the weights.

The spread has a direct operational reading. When a vendor quotes you an ARC-AGI-3 number without an effort setting, they have quoted you a number selected from a 97.3-point range. The charitable interpretation is that launch communications compress complexity. The uncharitable one is that the compression reliably lands on the upper end of the range. Either way, the number you received is the answer to a question you did not ask.

The peer-reviewed harness work shows what disciplined reporting looks like. The executable world models paper pins effort to high, holds the harness constant, and reports both GPT-5.4 (41.29%² mean RHAE, 8 of 25 solved) and GPT-5.5 (58.12%², 15 of 25) under identical conditions. You can argue with the harness, but you cannot argue that the comparison is confounded by configuration drift, because the configuration is the controlled variable. That is the minimum bar a score has to clear before it belongs in a routing decision.

How much harder is ARC-AGI-3 than the earlier versions?

Current systems reach roughly 93.0%³ on ARC-AGI-1, 68.8%³ on ARC-AGI-2, and about 13%³ on ARC-AGI-3 by the living survey’s cross-generation analysis, while humans hold near-perfect accuracy across all three. The difficulty ladder is steep, deliberate, and the survey’s authors read it as evidence of a persistent compositional generalization gap rather than a compute shortage.

The cross-paradigm consistency of the drop is the interesting part. The survey tracked 82 approaches across program synthesis, neuro-symbolic, and neural paradigms and found performance falling 2 to 3x from ARC-AGI-1 to ARC-AGI-2 in every paradigm. Architectures that disagree about everything else agree on this: the step from static abstraction puzzles to harder compositional tasks hurts everyone proportionally. ARC-AGI-3 extends the same logic into the interactive regime, where the model must also build and maintain a world model while it acts.

That last row against the row above it is the whole article in miniature. Thirteen percent³ and 58.12%² are both “ARC-AGI-3 scores,” separated not by model quality alone but by evaluation design: partial credit versus binary attainment, harness affordances, effort budgets. When GPT-6 Astra’s launch coverage quotes 63% and 99%¹ against a survey baseline of 13%³, the gap is not a one-day miracle of capability. It is mostly a change in what is being counted. Some of it may be genuine capability. On unreplicated numbers you cannot decompose the difference, and anyone who claims they can is selling something.

The human baseline deserves one sentence because it disciplines the interpretation. Humans sit near-perfect on all three generations. Whatever ARC-AGI-3 measures, humans find it easy, which means the benchmark is probing something specific about machine reasoning, not something intrinsically hard about the games.

What does a solved task actually cost?

Cost per task on the ARC benchmarks fell roughly 390x in one year, from about $4,500³ per task for o3 to about $12³ per task for GPT-5.2, according to the living survey, which reports the trend across benchmark versions rather than for ARC-AGI-3 specifically. The survey is explicit that most of that drop reflects reduced test-time parallelism rather than pure model efficiency. The cost curve is real, and the naive reading of it is wrong.

The parallelism caveat is the kind of detail that separates an analyst from a press release. Early frontier runs on ARC tasks threw massive test-time parallelism at each puzzle: sample many candidate solutions, score them, submit the best. That buys accuracy with money. When later systems report a 390x cost reduction alongside similar or better scores, part of what changed is that the field stopped paying for brute-force redundancy, either because the models got reliable enough to need fewer samples or because the evaluation conventions shifted away from permitting that spend. Both things happened; the survey attributes most of the movement to the convention change. Efficiency claims that ignore this are comparing different sports.

For operators, the cost-per-task figure interacts with the effort-setting sensitivity from earlier in a way that defines the actual trade space. Reasoning effort is how you spend money on an agentic benchmark, so an accuracy spread across effort settings implies a cost spread. The writeup reports accuracy only and carries no cost data, so treat that implication as an assumption, and note that the writeup’s own table undercuts a simple reading of it: the zero-effort ‘none’ setting posted 100.0%¹, tied with the maximum-effort settings. Accuracy and cost should still never be quoted more than a sentence apart. When they are, ask which one was optimized for the announcement.

The metric that survives all of this is cost per solved task per hour, measured in your own harness, at a pinned effort setting, on tasks that resemble your workload. It folds accuracy, effort, latency, and spend into one number you can compare across models and across months. It is also unfakeable in the way a launch score is not, because you are the one holding the harness.

What keeps an ARC-AGI-3 harness honest?

The credible harnesses enforce three controls: every playthrough starts from a fresh agent instance with a clean workspace and no access to prior files or conversation state, the agent-facing prompts and controller contain no game-specific code or heuristics, and the operators actively audit and close unintended information-leakage channels. The executable world models paper documents all three, and the third one exists because earlier harnesses were found vulnerable.

Leakage is the failure mode worth internalizing. An agentic benchmark runs an LLM in a loop with tools, a filesystem, and often network access, which means the agent can go looking for the answer key if any exists within reach. Earlier harness generations learned this the hard way: agents located benchmark-specific information through channels the designers had not considered, and scores inflated accordingly. The current generation of harness work treats leakage as an adversarial problem, auditing what the agent can observe and closing channels as they are found. The same agent, prompts, and workspace run across all 25 games, with no hidden solutions or per-game tuning.

Fresh instances are the quieter control with the bigger implication. Each playthrough starts with no memory of previous attempts, so the agent cannot amortize learning across games the way a human player would. That makes the score a measure of cold-start reasoning: can this system build a working theory of an environment it has never seen, fast, alone? It also means reported scores understate what a stateful deployment would do and overstate nothing, which is the right direction of bias for a benchmark.

These controls are why the 13%³ survey figure and the 58.12%² harness figure can both be honest numbers describing the same games. They are produced under different integrity regimes with different affordances and scoring rules. When you evaluate a launch claim, the integrity section of the methodology is the first thing to read. A score without documented fresh-instance and leakage-audit practices should be discounted toward zero, because the failure mode it invites is exactly the failure mode that makes a number look launch-worthy.

How should you route workloads off an ARC-AGI-3 score?

Route on cost per solved task per hour measured in your own harness at a pinned reasoning-effort setting, and treat any launch-day single-source score as directional until it is independently replicated. That rule handles GPT-6 Astra’s 63% and 99%¹, the neutral harness’s 2.7%-to-100%¹ spread, and every future launch that follows the same pattern.

The practical sequence looks like this. First, pin the configuration: effort setting, harness, task budget, scoring rule. Second, run the candidate models against a task sample drawn from your actual workload distribution, because ARC-AGI-3’s grid-world games are a proxy for interactive planning, not for your specific tool-use patterns. Third, compute cost per solved task per hour for each model at each effort setting you would plausibly deploy, which gives you a curve rather than a point. Fourth, pick the operating point on that curve that your latency and budget constraints allow, and re-run the evaluation when a model or harness version changes. The score on a leaderboard enters this process nowhere. It might tell you which models are worth the evaluation spend; it cannot tell you which one to route to.

The strongest limitation on everything above is the state of the evidence. GPT-6 Astra’s ARC-AGI-3 results are, in everything reviewed here, a single unreplicated community writeup from 2026-09-04. The detailed methodology available describes GPT-5.4 and GPT-5.5 systems, and the only GPT-6 Astra-specific datapoint is that writeup. If Astra’s replicated numbers land anywhere near the launch claims, it will be a genuinely strong agentic result, because the benchmark is genuinely hard: the cross-paradigm 2-3x degradation documented across 82 approaches in the living survey does not yield to marketing. But “anywhere near” is doing real work in that sentence, and nothing in the current evidence says where the number lands.

The durable shift is bigger than this launch. Agentic evaluations move the model-selection bottleneck off the accuracy leaderboard and onto the operating point: effort setting, harness integrity, and cost per solved task per hour. Teams still gating agents on static benchmark percentages are optimizing a variable the frontier has already routed around. The teams that build the internal harness, pin the configuration, and measure their own cost curves will know what GPT-6 Astra is worth to them within a week. The teams waiting for the leaderboard to settle will still be parsing the difference between 63% and 99%¹ when the next launch resets the cycle.

Frequently Asked Questions

Why does the ‘none’ reasoning-effort setting outperform ‘low’ and ‘medium’ on the neutral harness?

The non-monotonic spread suggests that intermediate effort budgets introduce planning overhead or context dilution that degrades performance, whereas zero effort relies on fast, direct heuristics that happen to align with the specific game mechanics in that test set. This implies that for certain agentic tasks, adding compute can actively hurt accuracy if the model’s internal planning loop is not robust to the added complexity.

How does the 390x cost reduction from $4,500 to $12 per task relate to actual model efficiency?

The drop primarily reflects a shift in evaluation conventions away from brute-force test-time parallelism, where early systems sampled many candidate solutions to find the best one. The reduction indicates that newer models require fewer samples to reach a correct solution, but it does not necessarily mean the underlying inference cost per token has decreased by that factor. Operators should not assume linear efficiency gains when scaling workloads based on this metric alone.

What specific failure mode does the ‘fresh instance’ control in ARC-AGI-3 harnesses prevent?

It prevents the agent from amortizing learning across multiple games, which would allow it to build a cumulative library of heuristics or world-model components from previous attempts. By forcing a cold start for every playthrough, the benchmark isolates the model’s ability to build a causal theory of a novel environment from scratch, rather than measuring its capacity for long-term memory or cross-task generalization.

Why is the 13% ARC-AGI-3 score from the living survey not directly comparable to the 58.12% RHAE from the coding-agent paper?

The 13% figure represents binary attainment (fully solved games) across a broad range of system architectures, while the 58.12% is a mean per-game RHAE that includes partial credit for incomplete solutions within a specific, controlled agentic harness. The difference highlights that the benchmark’s score is highly sensitive to the scoring rule (binary vs. partial) and the specific affordances provided by the evaluation environment, making cross-study comparisons invalid without normalizing for these variables.