Benchmarks disagree on GPT‑6 Astra, but its human‑beating efficiency on ARC‑AGI‑3 pulls François Chollet’s AGI forecast forward
Two narratives are taking shape around OpenAI’s GPT‑6 “Astra.” Epoch AI’s Epoch ECI puts Astra clearly at the top. Artificial Analysis’s Intelligence Index treats it as level with its predecessor. The wrinkle drawing attention away from leaderboard politics is Astra’s run on ARC‑AGI‑3. There it showed unusually efficient, symbolically compact world modeling and, in some setups, completed levels in fewer moves than the median human. Those results, and the role of vendor tooling in producing them, have ARC Prize co‑founder François Chollet saying timelines should be moved earlier, even while ARC stresses that benchmark success is not proof of AGI.
Quick definitions you’ll need
- ARC‑AGI‑3, an ARC Prize benchmark that places models into unfamiliar, unexplained game worlds and evaluates short‑horizon exploration, adaptation, and causal modeling.
- Harness (vendor vs. ARC in‑house), the testing scaffold around a model: state preservation, summarizers, tool integrations and caching. Different harnesses can change measured performance, token usage, and wall‑time dramatically.
- PRO‑LONG, a sandboxed code‑execution framework used in some runs to let agents execute and persist code (reported as a red‑teaming/development partner in some Astra runs).
- Lean‑verified proof, a machine‑checkable formal proof verified by the Lean theorem prover; passing Lean means the formalization and proof satisfy the prover’s logical checks.
How the scorecards diverge
The aggregators tell different stories. Epoch AI’s Epoch ECI, an aggregate of “more than 50 benchmarks, ” according to Epoch AI, rates GPT‑6 Astra at 169, ahead of 267 model entries. Artificial Analysis’s AA Intelligence Index gives Astra 61 points, the same score recorded for GPT‑5.6 “Sol.” Anthropic’s Claude Fable 5.1 sits at 66 in AA’s ranking.
Reported Epoch ECI snapshot: Astra 169; Sol 162; Fable 5.1 163; Opus 5 162. Reported Artificial Analysis snapshot: Astra 61; Sol 61; Fable 5.1 66; Opus 5 63. Those headline numbers reflect different benchmark mixes, weighting and model cutoffs. Aggregators choose different suites and priorities, so differences are inevitable.
Astra’s standout: ARC‑AGI‑3 and surprising efficiency
Using ARC Prize’s in‑house scaffold, Astra reached 62.7% on ARC‑AGI‑3. For comparison, reported scores include Sol at 7.8% and Opus 5 at 30.2%.
“Astra creates a dense, compact symbolic world model to complete ARC‑AGI‑3 environments.”, ARC Prize (tweet, Sept 3, 2026, as reported)
Two practical facts from the ARC‑AGI‑3 runs matter for any business evaluating these results.
- Efficiency vs. cost paradox: On ARC’s standard scaffold, increasing explicit reasoning settings raised the score while lowering total token usage and cost. Reported figures show cost dropping from $49, 791 with no explicit reasoning to $26, 098 at maximum reasoning, while the ARC‑AGI‑3 score climbed from 35.2% to 62.7%.
- Human baseline and moves: ARC ran roughly 500 human testers, with no pre‑screening, to establish median moves per level. Testers were paid $115 per 90‑minute session plus $5 per game solved. ARC reported that, on OpenAI’s vendor harness, Astra cleared 96% of levels in fewer moves than the human median and, averaged across those solved levels, used a little over half the moves of a typical human.
François Chollet highlighted Astra’s emergent shorthand DSL and compact symbolic modeling on X, calling it “highly efficient, on‑the‑fly symbolic world modeling for each game and level” and describing the shorthand as “essentially a game‑specific algebraic notation.” He said the arrival of this capability was “about twice as fast” as he expected and that the pace of progress will surprise many people, while reiterating, as ARC has, that solving ARC‑AGI‑3 is not proof of AGI.
Why harnesses change everything
The harness, the software and tooling around a model, has become the single most important variable in cross‑model comparisons. ARC reported that runs using OpenAI’s vendor harness were about 3.66× faster and used 49% fewer tokens than runs on ARC’s in‑house harness across 167 comparable game‑reasoning pairs. OpenAI also reported higher success numbers under different harness conditions, with some vendor‑setup comparisons reported as 99.9% success. The takeaway: model comparisons that do not specify harness and tool access can be misleading.
This works both ways. Vendor harnesses can improve performance through engineering like summarizers, caching and persistent state. Observers including Chollet also noted Astra seems to internalize some harness‑like behaviors, constructing compact, algebraic internal representations it can reuse. What used to be external engineering appears, in specific tasks, to be migrating into the model itself.
Symbolic DSLs and agentic code writing
Two technical patterns underpin Astra’s ARC‑AGI‑3 edge and matter for automation planning.
- Internal symbolic notation: Astra often invents a compact, algebra‑like shorthand to represent objects, positions and operations within a game level. That notation acts like an internal scratchpad the model uses to plan and compress state.
- Agentic use of code execution (PRO‑LONG): When granted access to a sandboxed code executor called PRO‑LONG, Astra wrote and executed code libraries to automate repetitive subtasks and to persist utilities across attempts. Runs that used PRO‑LONG blurred the line between a single API call and a small stateful agent with executable tooling.
For businesses, those traits matter. Internal abstractions and reusable tooling reduce iteration and error when a model handles multi‑step workflows. They also raise design questions about provenance, audit trails and reproducibility.
FrontierMath Erdős: formal proofs and expensive correctness
Epoch AI’s FrontierMath Erdős benchmark uses a strict protocol: a $300 inference budget per problem and Lean‑verified formal proofs as the scoring requirement. Epoch reports GPT‑6 Astra produced two Lean‑verified solutions out of 68 open Erdős problems under that budget, a reported 3% score. The two scored runs listed by Epoch included one attempt costing $218 and taking 15 hours, another costing $247 and taking 16 hours.
Epoch documents the engineering overhead. A natural‑language proof that fits in a short paper can expand to millions of lines of Lean during formalization. Epoch cites an 18‑page proof ballooning to about 1.2 million lines of Lean. Epoch also notes additional out‑of‑protocol, higher‑budget attempts produced more solutions but were excluded from the official score. The $300 per‑attempt rule is the scoring baseline in Epoch’s public materials.
Other benchmark signals, mixed but telling
- ARC‑AGI‑2 (abstract visual puzzles): Astra 95.0%; Sol 92.5%; Fable 5.1 90.0%; Opus 5 90.4 (reported figures).
- ARC‑AGI‑1: Astra reported at 98.5% at an xhigh reasoning effort setting, and 97.5% at max (a non‑monotonicity that ARC did not publicly explain in detail, as reported).
- Coding Agent Index (reported): Astra reached 67 points while using roughly one‑third of Sol’s token usage; Fable 5.1 led with 70.
- AA‑Omniscience hallucination metric (reported): Astra’s hallucination rate dropped from 92% to 51% under some evaluated conditions.
- Regressions (reported): Astra lost about 80 Elo points on GDPval‑AA v2 and showed slips on certain banking, SciCode, and long‑context reasoning tasks.
The pattern is clear: Astra is not uniformly superior. It shines in short‑horizon causal tasks, internal symbolic modeling and some coding contexts, but shows regressions elsewhere. Aggregator rankings diverge because their suites emphasize different capabilities.
Cost, pricing and the compute‑efficiency tradeoff
Price per token and per‑task cost tell two sides of the story. Reported OpenAI pricing made Astra “two and a half times as much per unit of processed text, ” which was described as making a typical task “roughly 75 percent more” costly when measured by raw per‑token price. At the same time, Astra reportedly needs only one third of Sol’s compute steps for some tasks and one fifth of Opus 5’s, meaning fewer total tokens and fewer steps are needed to finish work.
Concrete example from the reported ARC runs: Astra’s ARC‑AGI‑3 testing cost roughly $26, 000 on ARC’s standard scaffold. Under vendor harness conditions Astra completed many levels with fewer tokens and less wall‑time; ARC reported vendor harness runs were about 3.66× faster and used 49% fewer tokens than ARC’s in‑house runs for comparable solved pairs. Net cost per solved task therefore depends on two variables: the vendor’s per‑token price and the model’s token efficiency on your workflow.
What to ask vendors and what to demand as a buyer
Benchmarks are increasingly sensitive to harnesses, hyperparameters, and tool access. Before you sign a purchase order or re‑architect a workflow, obtain the following:
- Run artifacts: per‑level token traces, wall‑time, and harness configuration for any reported benchmark numbers.
- Reproducible harness scripts: the exact scaffold used for vendor‑reported runs so you can replicate results in your environment.
- Cost breakdowns: what’s included (inference, human evaluation, sandbox compute) and whether reported totals exclude expensive out‑of‑protocol attempts.
- Tooling disclosure: which components were provided by the vendor harness (summarizers, caching, state stores), and whether the model was allowed persistent execution (e.g., PRO‑LONG) or external code execution.
Prioritized, practical checklist for executives
- Measure model+tooling, not model alone. Run N=10 runs of your workflow both under the vendor harness and your scaffold. Compare median completion time, token consumption and failure modes.
- Benchmark the actual workflow you care about. Aggregator leaderboards are noisy. Prioritize repeatable tests that mirror your inputs, data formats and latency requirements.
- Simulate end‑to‑end cost. Include per‑token price, expected token reduction, retries and human evaluation in your cost model. A higher per‑token price can still be cheaper per completed task if tokens drop enough.
- Capture emergent scratchpads. Log model‑generated internal representations. If the model invents a shorthand DSL that helps your pipeline, convert those patterns into reusable templates and guardrails.
- Insist on reproducibility and clear boundaries. Require vendors to disclose harness features and to provide a path to replicate high‑value runs in your environment before production deployment.
Unanswered questions worth tracking
- Will Astra’s internal symbolic modeling generalize beyond tightly constrained game worlds into messy, real‑world tasks?
- How much of reported performance depends on vendor harness engineering versus intrinsic model improvements, and can purchasers reproduce vendor harness benefits independently?
- What baseline should count: model alone, model+tools, or model+vendor tooling? Benchmarkers and procurement teams need a shared vocabulary.
- How should the community weight expensive, out‑of‑protocol runs (the ones that consumed hundreds of thousands of dollars) when assessing frontier problem solving?
Key questions and short answers
- Does Astra beat other models across the board?
No. Epoch AI’s ECI places Astra first with 169 across the aggregator’s reported set, while Artificial Analysis’s AA Index rates Astra at 61, tied with Sol and behind Claude Fable 5.1 at 66. Different benchmark mixes, harness setups and weighting explain the divergent rankings.
- Is Astra proof of AGI?
No. ARC Prize co‑founder François Chollet and ARC’s public materials emphasize that benchmark success is not proof of AGI; ARC‑AGI‑3 is a useful test of short‑horizon causal reasoning, not a finish line.
- Why did Astra do so well on ARC‑AGI‑3?
Reportedly because Astra produced dense, compact symbolic world models and, in runs that allowed it, used a sandboxed code executor (PRO‑LONG) to build and reuse libraries. Vendor harness optimizations (state preservation, summarizers) also materially improved token efficiency in some runs.
- Are the FrontierMath proofs credible?
Epoch AI reports two Lean‑verified solutions under the FrontierMath $300‑budget protocol; Lean verification means the proofs passed the Lean theorem prover’s machine checks, which is a strong indicator of formal correctness for those specific formalizations.
- How should my company treat vendor claims and published scores?
Treat them as conditional. Ask which harness and tooling were used, request per‑run artifacts, and reproduce critical runs in your environment before committing to production. Vendor tooling is part of the product you’re buying; evaluate it explicitly.
Final thought
GPT‑6 Astra is not a single, tidy result you can paste into a procurement form. It is a set of capabilities, internal symbolic representations, agentic code use, compute‑step efficiency, that perform well in specific contexts and are sensitive to test harnesses. For leaders the actionable response is simple: demand transparency, test the full model+tooling stack on your workflows, and budget for continuous evaluation. Progress is accelerating on particular axes. Planning that treats vendor tooling and reproducibility as first‑class concerns will pay off sooner than waiting for a single, decisive benchmark to declare victory.