Artificial Analysis v5 Benchmark Launches; DeepSeek V4.1 Flash Tops Open-Source Coding

IT Home reported on October 29 that Artificial Analysis released the Intelligence Index v5 yesterday, marking the most significant benchmark restructuring by the organization since the launch of v4 in January 2026.

The most direct change is that 60% of the evaluation weight is now based on private test sets or hidden answers for the first time. This aims to compress the space for "contamination" of public question banks by model training data and reduce "padded" test items designed solely for leaderboard chasing by various vendors.

The index still consists of nine evaluations, with Knowledge Work, Coding, Science, and General Capability each accounting for 25%. Within the 25% weight allocated to Coding, FrontierCode 2.0—released by Cognition on the same day—accounts for 15%, while Terminal-Bench 4.0 accounts for 10%.

FrontierCode 2.0, which warrants the most attention, has completely overhauled its scoring logic. All 132 tasks are drawn from real-world open-source repositories. Code changes submitted by models are no longer judged solely by whether unit tests pass; instead, they are scored by repository maintainers based on their willingness to merge the pull request (PR). Any failure in a blocking check results in a zero score.

When Cognition launched the initial version of FrontierCode in June, it emphasized that traditional benchmarks only check if code runs successfully, whereas merge decisions involve dimensions not fully covered by tests, such as whether the scope of changes is restrained, whether the style aligns with the codebase, and whether regression risks are controllable. By upgrading this benchmark to 2.0 and incorporating it into the main index, v5 effectively shifts the center of coding capability evaluation from "runnability" to "mergeability."

Under this metric, DeepSeek V4.1-Flash 1017 (max) recorded a score of 28% on FrontierCode 2.0, making it the highest-scoring model among all released weights.

Kimi K3.1 Pro follows closely with 25.5%. However, according to Moonshot AI's official blog, its full weights will not be released until November 4. Artificial Analysis noted in its v5 report that Kimi K3.1 Pro's "weights are not yet released."

Google Gemini 4 Argon (high) scored 25.7%, and GPT-6 Luna scored 25.2%, both lower than DeepSeek V4.1 Flash. It should be noted that these scores are within the single FrontierCode 2.0 category. In terms of the overall Intelligence Index v5 composite score, DeepSeek V4.1 Flash and GLM-5.3 both scored 32 points, trailing the leader Claude Opus 5.5 (58 points) by 26 points, and falling below Claude Haiku 5.5's 36 points.

Artificial Analysis pointed out that v5 truly exacerbates the overall gap between open-source and closed-source frontier models. In v4.3, the gap between the strongest open-source model and the leader was 3 points. After v5 replaced most public questions with private ones and significantly increased the weight of agentic practical tasks, this gap widened to 12 points. Kimi K3.1 Pro, with 46 points, is nominally the strongest open-weight model, but its weights have not been released. Among models with released weights, 32 points is currently the ceiling.

The disparity is most glaring in two categories: HLE-Diamond and Terminal-Bench-Science. Kimi K3.1 Pro scored 37% and 34% respectively, while Claude Opus 5.5 scored 68% on HLE-Diamond, and GPT-6 Astra scored 63% on Terminal-Bench-Science.

v5 did not make open-source models "worse overnight"; rather, it changed what is being measured. When questions are no longer public and scoring no longer relies solely on pass rates, scores optimized through public question banks are systematically stripped of inflation.

Another variable amplified by v5 is the cost advantage associated with "effort levels" (thinking tiers). For Claude Opus 5.5, moving from the low-effort tier to the max tier spans a 23-point difference, with the cost per task rising from $0.98 to $10.04—a roughly tenfold increase.

The performance gap between effort tiers for Claude Sonnet 5.5 has widened from 20 points in v4.3 to 29 points, while the span for GPT-6.1 Sol has doubled to 20 points. This means that the difference in performance between a model’s “cost-saving mode” and its “full-power mode” is now so significant that it no longer feels like the same model.

Among the top ten scoring models, GPT-6.1 Terra has the lowest cost per task at $0.96, whereas Claude Fable 5.1 has the highest at $13.04—a difference of more than 13 times.

Launched alongside v5, the Progress Index v1 addresses a different challenge: how to compare models from different eras on a consistent scale when benchmarks themselves change every few months. The Progress Index maps major language models released since November 2022 onto a continuous capability scale, drawing methodological inspiration from Epoch AI’s Epoch Capabilities Index. While the Intelligence Index remains the primary metric for measuring the “current frontier,” the Progress Index is used to observe “how capabilities evolve over time.” These two indices answer fundamentally different questions.

In practice, no single model leads across all nine evaluations in v5: Claude Opus 5.5 tops three categories—FrontierCode 2.0, HLE-Diamond, and AA-Omniscience; GPT-6 Astra leads in Terminal-Bench-Science and the difficult subset of CritPt v2; and Kimi K3.1 Pro takes first place in Terminal-Bench 4.0 with a score of 67%. The dispersion of individual category advantages is arguably more noteworthy than the overall ranking by total score.