🔍 Read the full analysis: FrontierCode: How To Prepare AI-Generated Code For A Maintainer on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, which scores patches on more than whether tests pass. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts; the results show that human review remains necessary under this benchmark.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that judges whether a patch meets maintainer standards, not just whether it passes tests. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts, indicating that even the top result falls short of dependable, unattended code changes under this evaluation.
FrontierCode’s results were added to Artificial Analysis’s evaluations on October 12, 2026. The benchmark’s reported score combines blocking requirements with further quality checks: fail any blocking check and an attempt receives zero, while successful attempts can score from 70% to 100%, depending on quality. The separate pass-rate figure records how often a model cleared all blocking checks.
After Opus, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%. The top nine models, from six labs, scored between 43.2% and 52.2%. Xiaomi’s MiMo-V2.6-Pro led the open-weight group at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. At the lower end of the reported results, Grok 4.7 scored 33.8% and Nemotron 3 Ultra 10.5%.
Artificial Analysis ran models through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. Its evaluation used the benchmark’s 100-task Main set. Nine of those tasks were not solved by any model on any attempt, according to the source report. Costs varied widely: GPT-6 Luna cost $0.09 per task, while Opus cost $10.15. The source identifies four models on the cost frontier, where no other listed model is both cheaper and higher-scoring.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer Checks Change the Ranking
FrontierCode targets a gap between code that passes tests and code a maintainer would be willing to merge. Its checks include hidden tests, commands for regressions and cleanliness, restrictions on files touched and diff size, and an LLM review against maintainers’ criteria. It also checks whether an agent’s own tests fail on the original code; tests that pass both before and after a change may not establish that the patch works.
That distinction matters to teams considering AI agents for repository work. A patch that meets visible tests can still introduce regressions, make unnecessary changes or provide weak tests, leaving maintainers with extra review work. The results do not show that AI coding tools cannot be useful. They do show that, on this benchmark, passing tests alone is not a reliable proxy for a merge-ready change, and human review remains an important safeguard.
The score-cost trade-off is also relevant for teams comparing models. Artificial Analysis reports that moving from GPT-6.1 Sol at $0.81 per task to Opus at $10.15 buys 3.1 score points. Those prices and scores describe this evaluation setup, not a guarantee of performance or cost in other repositories and workflows.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Repository Work
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against agents looking up solutions online. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The full benchmark contains 150 tasks; Artificial Analysis ran its 100-task Main set.
The tasks are private, according to the source, to reduce the chance models have encountered their solutions in training. A trajectory scanner can assign a zero to attempts flagged for solution lookup. Other penalties address overly large diffs, edits outside relevant files and unhelpful tests. Artificial Analysis used GPT-6 Luna (medium) for the review step.
The results also diverge from a general intelligence ranking. In the source report’s comparison, Claude Sonnet 5.5 ranked second among the listed models on the Intelligence Index but fifth on FrontierCode. Grok 4.7 ranked 18th on FrontierCode, while DeepSeek V4.1 Flash scored more strongly there than its Intelligence Index position might suggest. That comparison supports treating task-specific coding evaluations as distinct from broad model rankings.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Models and Limits Still Missing
The results are one benchmark run, not a measure of every coding task or production repository. The source does not establish how scores translate to real-world acceptance rates, how performance varies across teams’ review standards, or whether results would hold under different agent harnesses and model settings. The reported cost figures are tied to the tested setup and should not be treated as universal operating costs.
Several models had no FrontierCode result as of October 13, according to the source: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every prominent model. It is also not clear from the source report when those results might be added.
As an affiliate, we earn on qualifying purchases.
Watch for More Model Results
The next development to watch is whether Artificial Analysis adds results for models missing from the October 13 listing, and whether subsequent evaluations change the ranking. For developers, the immediate practical step is to judge coding systems against the checks their own maintainers use, including tests, scope limits and review quality, rather than relying on a single benchmark score.
FrontierCode’s published results offer a comparison under shared conditions; they do not remove the need to inspect individual patches. Teams evaluating agents will still need to decide how to validate changes, control file and diff scope, and assign human review before merging.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates repository patches using tests, scope and cleanliness checks, and a review against maintainers’ criteria.
Which model led the reported results?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, with a 52.2% score. It cleared all blocking checks in 57.7% of attempts.
Does a FrontierCode score mean an AI patch is ready to merge?
No. The score summarizes performance on the benchmark’s tasks and checks. It does not certify that a particular patch is safe or suitable for a real repository; maintainers still need to review changes.
Why do the reported score and pass rate differ?
The pass rate measures attempts that cleared every blocking check. The benchmark score also reflects quality checks: failed blocking checks receive zero, while successful attempts can score from 70% to 100%.
Were all leading AI models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode results as of October 13, 2026.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
