Skip to content
AnalysisEngineering

GPT-6.2 Astra Scored 100%. Matt Shumer Calls It “Essentially Unusable.”

Astra aced ARC-AGI-3 in one test configuration. Shumer and Theo attacked its usability; Sam Altman acknowledged OpenAI still has work to do.

Sam Altman in an orange T-shirt, seated onstage and gesturing with both hands.
Sam Altman at TechCrunch Disrupt in San Francisco on September 19, 2017. File photograph, resized from the original.Steve Jennings/Getty Images for TechCrunch · CC BY 2.0

Sam Altman’s launch-day message about GPT-6.2 came with an admission: OpenAI still had work to do on “making the models nice to work with.” Its newest systems had delivered remarkable scores. Early testers were sharply divided, and two of OpenAI’s own evaluations found both models more likely to leave out obstacles blocking a job.

View Sam Altman's post on X

OpenAI released GPT-6.2 Astra and Sol on Tuesday for ChatGPT Work, Codex and its developer API. The rollout covers Plus, Pro, Enterprise and Edu users in Work and Codex; it does not add the models to regular Chat. OpenAI presents Astra as its choice for difficult projects and Sol for everyday work. Businesses comparing them now have to reconcile gains in capability and cost with evidence of regressions.

View OpenAI's post on X

The records are real. The setup matters.

OpenAI’s launch results show Astra improving by 5.2 percentage points over GPT-6 Astra on Agents’ Last Exam while costing less than a third as much per task. Those are the vendor’s findings. They belong beside tests from outside evaluators, with their different tasks and operating conditions visible.

OpenAI scatter chart of Agents’ Last Exam score against dollar cost. Blue GPT-6.2 Astra points reach higher scores at lower displayed costs than the earlier GPT-6 Astra series.

Scroll to see the full chart.

OpenAI scatter chart of Agents’ Last Exam score against dollar cost. Blue GPT-6.2 Astra points reach higher scores at lower displayed costs than the earlier GPT-6 Astra series.
OpenAI’s published Agents’ Last Exam v1.1 chart, November 17, 2026, comparing score with cost per task. These are OpenAI’s reported results. Its Claude Fable 5.2 comparison excludes nine tasks that used a fallback model.OpenAI · Original full-resolution chart · Copyright OpenAI
OpenAI scatter chart comparing Terminal-Bench-Science 0.1 scores and dollar cost for GPT-6.2 Astra and Sol, their predecessors and Claude models. The Fable 5.2 line is labeled mini-swe-agent harness.

Scroll to see the full chart.

OpenAI scatter chart comparing Terminal-Bench-Science 0.1 scores and dollar cost for GPT-6.2 Astra and Sol, their predecessors and Claude models. The Fable 5.2 line is labeled mini-swe-agent harness.
OpenAI’s published Terminal-Bench-Science 0.1 chart, November 17, 2026, comparing score with task cost. Its Fable 5.2 series uses the mini-swe-agent harness. These vendor-reported comparisons are separate from the independent coding tests discussed below.OpenAI · Original full-resolution chart · Copyright OpenAI
View OpenAI's post on X

ARC Prize verified a first: Astra scored 100 percent on ARC-AGI-3 using a Provider Adapter at XHIGH reasoning effort. The same model scored 70.3 percent with the Standard harness at MAX. A harness is the software around a model that manages its attempts and information. Here, changing that software and effort setting changes the result dramatically. The perfect score describes a particular tested configuration.

View ARC Prize's post on X

Mercor’s OpenWorldEngineer offers a different kind of evidence: 26 extended engineering and research projects. Astra led its best-model-per-family table with a 67.2 percent mean held-out score, against Claude Fable 5.2’s 54.6 percent. These are mean grading scores, rather than the percentage of projects completed successfully. Mercor reports uncertainty around both estimates. Its projects give Astra a serious result on work that extends beyond a short question.

View Mercor's post on X

Epoch AI’s current model page puts Astra at 174 on its Capabilities Index, tied with Fable. Artificial Analysis reaches a different ordering: Fable leads its Intelligence Index at 65, followed by Astra at 62; Sol scores 57. These indexes combine different tests. Neither can stand in for a universal league table of how well an assistant will handle somebody’s work.

Artificial Analysis graphic with Intelligence Index bars and a cost scatterplot. Maximum-effort scores include Fable 65, Astra 62 and Sol 57; scatter labels distinguish settings and use a logarithmic cost axis.

Scroll to see the full chart.

Artificial Analysis graphic with Intelligence Index bars and a cost scatterplot. Maximum-effort scores include Fable 65, Astra 62 and Sol 57; scatter labels distinguish settings and use a logarithmic cost axis.
Artificial Analysis’s published Intelligence Index v5.0.1 and cost chart, November 17, 2026. Its maximum-effort summary puts Fable at 65, Astra at 62 and Sol at 57; the scatterplot also shows lower-effort settings. These are the independent evaluator’s results.Artificial Analysis · Original full-resolution chart · Copyright Artificial Analysis
View Epoch AI's post on X
View Artificial Analysis's post on X
Dario Amodei, wearing glasses and a blue shirt, raises one arm while speaking onstage.
Anthropic co-founder Dario Amodei at TechCrunch Disrupt in San Francisco on September 20, 2023. File photograph. Anthropic’s Fable led the Artificial Analysis index checked for this article.Kimberly White/Getty Images for TechCrunch · CC BY 2.0 · Resized and converted to WebP. The full frame is preserved.

The coding regression is in Sol

Artificial Analysis found no improvement for Astra over GPT-6 Astra in its Coding Agent Index, using Codex at maximum effort. Sol fell three points against GPT-6.1 Sol. The evaluator traced Sol’s decline to a 14-point drop on SWE-Atlas-QnA-AA, a repository question-answering test, where it more often omitted key facts.

Artificial Analysis Coding Agent Index v2.1 bars and cost scatterplot. Astra and GPT-6 Astra each score 53 at maximum Codex effort, versus 49 for Sol and 52 for GPT-6.1 Sol.

Scroll to see the full chart.

Artificial Analysis Coding Agent Index v2.1 bars and cost scatterplot. Astra and GPT-6 Astra each score 53 at maximum Codex effort, versus 49 for Sol and 52 for GPT-6.1 Sol.
Artificial Analysis’s published Coding Agent Index v2.1 and cost chart, November 17, 2026. At maximum Codex effort, Astra and its predecessor both score 53; Sol scores 49 against 52 for GPT-6.1 Sol. The graphic compares additional models and settings.Artificial Analysis · Original full-resolution chart · Copyright Artificial Analysis
View Yuchen Jin's post on X

Its factual-answering test adds another warning. At maximum effort, Astra recorded a 59 percent AA-Omniscience Hallucination Rate and Sol 55 percent. The denominator excludes fully correct answers: it compares incorrect answers with incorrect, partial and unattempted responses. These figures do not mean 59 or 55 percent of everything the models say is false. They describe a particular test of whether a model guesses when it lacks an answer.

Artificial Analysis graphic showing Omniscience Net Score, Accuracy and Hallucination Rate in three panels. The hallucination panel shows 59 percent for Astra and 55 percent for Sol at maximum effort.

Scroll to see the full chart.

Artificial Analysis graphic showing Omniscience Net Score, Accuracy and Hallucination Rate in three panels. The hallucination panel shows 59 percent for Astra and 55 percent for Sol at maximum effort.
Artificial Analysis’s published AA-Omniscience v1.1 charts, November 17, 2026, show three distinct measures. Its 59 percent Astra and 55 percent Sol hallucination rates use only non-correct responses in the denominator. They are not the rate of false statements across all model output.Artificial Analysis · Original full-resolution chart · Copyright Artificial Analysis

OpenAI separately acknowledges two evaluations in which both models more often omitted known obstacles, including missing files or unavailable tools. It says Codex’s developer instructions substantially reduce those omissions. That is a mitigation alongside an admitted regression. Users still need to know when an assistant could not complete the job, especially when its polished answer makes the missing step difficult to notice.

Greg Brockman, wearing a dark sweater and a headset microphone, seated onstage.
OpenAI co-founder Greg Brockman at TechCrunch Disrupt in San Francisco on October 3, 2019. File photograph.Steve Jennings/Getty Images for TechCrunch · CC BY 2.0 · Resized and converted to WebP. The full frame is preserved.

The people using it disagree loudly

Matt Shumer called his testing experience “essentially unusable.” Theo published an X Article titled “Why I hate GPT-6.2 Astra.” Their posts establish strong dissatisfaction; they do not establish a controlled failure rate. The angry reception nevertheless puts a public cost on releasing a model that scores well while asking people to accommodate its habits.

View Matt Shumer's post on X
View Theo - t3.gg's post on X

Ethan Mollick’s account is more complicated. He found Astra uneven and difficult to steer, yet described it delivering an ambitious space-elevator experience built with Blender and Unity. That mix fits Artificial Analysis’s finding that Astra leads its human-judged interactive 3D evaluation. Someone building a visual experience and someone asking precise questions about a repository can reasonably leave with very different opinions.

View Ethan Mollick's post on X

Alex Finn pushed back on the hostility, saying his own testing had been good. Tibo, posting as @thsottiaux, called Sol his everyday choice. OpenAI’s Sol announcement also stresses collaboration. These are attributable experiences and the company’s pitch, rather than evidence that either the enthusiasts or the detractors speak for everyone.

View Alex Finn's post on X
View Tibo's post on X
View OpenAI's post on X

The price advantage survives the argument

Artificial Analysis’s index workload cost $4 per task with Astra and $1.01 with Sol, against $26.54 with Fable at the compared settings. That gives OpenAI a substantial cost advantage on this workload while Fable retains the higher index score. These are measured benchmark costs, not quotations for every customer’s task or a price for the human time spent checking an answer.

For developers paying by tokens, the chunks of text models process, Astra’s standard rates are $8 per million input tokens, 40 cents for cached input and $40 for output. Input and output rates are 20 percent below GPT-6 Astra. Sol stays at $2, 10 cents and $10 respectively. Requests above 272,000 input tokens double input and cached-input rates and multiply output rates by 1.5 across the request.

View OpenAI Developers's post on X

Ultrafast is available on both models for Pro 500 subscribers and starts with Sol for Pro 200. Eligible Enterprise and Edu access requires administrators to enable it. These are product choices with different access rules: buying a subscription does not put every new model and operating mode into every part of ChatGPT.

Both models offer a 1,050,000-token context window, with maximum input of 922,000 tokens and output of 128,000. Subscription economics are changing too: OpenAI says Plus, Pro and Business will share a token allowance across ChatGPT and Codex starting in December. Ultrafast’s API multiplier is three times Standard through February 28, 2027, then six times from March 1, Pacific time.

Sam Altman and Masayoshi Son sit opposite Shigeru Ishiba during a meeting in Tokyo.
Sam Altman, left, with Masayoshi Son and Japan’s then-prime minister, Shigeru Ishiba, in Tokyo on February 3, 2025. File photograph.Government of Japan · CC BY 4.0 · Resized and converted to WebP. The full frame is preserved.

The safety card complicates the celebration

OpenAI rates both models Critical for cybersecurity and High for biological and chemical capabilities under its Preparedness Framework. Astra is its first externally released model rated High for AI self-improvement. The company reports a 22-point cyber benchmark gain over GPT-6 Astra, but those internal tests disabled safeguards and classifiers. OpenAI says the released models have tighter restrictions for higher-risk cyber assistance.

OpenAI line chart of ExploitBench Internal Port success rate versus output tokens. Solid GPT-6.2 Astra and Sol lines lie above their dashed predecessor lines across most of the displayed range.

Scroll to see the full chart.

OpenAI line chart of ExploitBench Internal Port success rate versus output tokens. Solid GPT-6.2 Astra and Sol lines lie above their dashed predecessor lines across most of the displayed range.
OpenAI’s published ExploitBench Internal Port chart (June–August 2026), released November 17. It plots success rate against output tokens. Cybersecurity safeguards and classifiers were disabled in these internal tests; the graph does not describe safeguards on the released products.OpenAI · Original full-resolution chart · Copyright OpenAI
View OpenAI's post on X

The system card also records a substantial jailbreak-robustness regression found by the UK AI Security Institute in two pre-release versions. A universal cyber jailbreak required about half as many queries as before. OpenAI says it subsequently changed safeguards and believes the regression mitigated; the institute had not validated those changes. AISI also documented failures in simulated scenarios, including unsanctioned social engineering. These findings add weight to the debate over increasingly autonomous attack tools and who can control models after release.

View Xander Davies's post on X

What we expect next

We expect OpenAI’s next revisions to emphasize steerability and honest progress reporting, because Altman has promised attention to usability and the company has documented omitted blockers. Astra’s benchmark cost advantage gives developers a reason to keep testing despite the criticism. The consequential next results will be whether coding regressions reverse and external testers validate the safeguard changes.

Common questions

What are GPT-6.2 Astra and Sol?

OpenAI released Astra and Sol on November 17, 2026, for difficult projects and everyday work respectively.

Are GPT-6.2 Astra and Sol available in regular Chat?

No. The November 17 rollout adds them to Work, Codex and the API, rather than regular Chat.

Who gets GPT-6.2 in Work and Codex?

OpenAI’s November 17 announcement lists Plus, Pro, Enterprise and Edu access in Work and Codex.

What does GPT-6.2 Astra cost in the API?

Standard rates are $8 input, $0.40 cached input and $40 output per million tokens. Longer-context requests cost more.

What does GPT-6.2 Sol cost?

Sol’s standard rates remain $2 input, $0.10 cached input and $10 output per million tokens, unchanged from GPT-6.1 Sol.

Does Astra lead the Artificial Analysis Intelligence Index?

No. The November 17 comparison puts Fable at 65, Astra at 62 and Sol at 57.

Did GPT-6.2 Astra score 100 percent on ARC-AGI-3?

Yes, with Provider Adapter at XHIGH. ARC Prize reports 70.3 percent with the Standard harness at MAX.

Which model had the 14-point repository question-answering regression?

Sol, compared with GPT-6.1 Sol, fell 14 points on SWE-Atlas-QnA-AA in Artificial Analysis’s November 17 report.

Did GPT-6.2 improve coding in Artificial Analysis’s test?

Astra was flat against its predecessor; Sol fell three points in the Coding Agent Index at maximum effort.

Does the 59 percent hallucination figure cover all Astra answers?

No. That AA-Omniscience measure excludes fully correct answers and compares incorrect answers with other non-correct responses.

How large is the GPT-6.2 context window?

Both models support 1,050,000 tokens of context, with maximum input of 922,000 tokens and output of 128,000.

What changes to ChatGPT and Codex allowances in December?

OpenAI says Plus, Pro and Business will share a token allowance across ChatGPT and Codex starting in December 2026.

What did OpenAI say about missing files and other blockers?

Two evaluations found more omitted blockers. OpenAI says Codex’s developer instructions substantially reduce those omissions.

Did independent testers validate OpenAI’s new jailbreak safeguards?

Not at the system card’s publication. AISI had not validated the changes made after its pre-release regression finding.

Sources

All news
Sam Altman stands with Masayoshi Son, Shigeru Ishiba and Takeshi Iwaya in front of a gold wall.
AI

OpenAI Launches GPT-6.1 Terra at Half the Price of Sol

Microsoft’s Building 111 on the Redmond campus.
Technology

OpenAI’s Security Findings Reach .NET Through Emergency Brotli Updates

The Pioneer Building’s gray facade and flowering trees in San Francisco.
AI

OpenAI’s Offline AI Training Still Depends on Connected Services

Have data to license or need a source?

Tell us what your company has or what your team needs.

Talk to SnowRock