Today for AI
HOT RADAR
llmHEAT 44.5°

AI CURATED HOT STORY · 10/28/2026

Artificial Analysis Launches v5 Intelligence Index to Combat Benchmark Contamination

7 reports archived5 independent sourcesupdated 10/29/2026, 01:06:21
Synthesis & Latest Updates
5 Sources Cross-Validated

Artificial Analysis has launched a major upgrade to its Intelligence Index, version v5, shifting 60% of evaluation weight to private test sets or held-out answers. This move addresses benchmark saturation and data contamination, introducing challenging agentic benchmarks like FrontierCode 2.0 and Terminal-Bench-Science to enhance robustness and real-world applicability.

Cross-referenced reports confirm that v5 integrates nine specific datasets covering reasoning, knowledge, math, and coding, while explicitly excluding image and multilingual tests. The simultaneous release of Progress Index v1 provides a continuous scale for tracking long-term model evolution. Under the new standards, some open-source models like Kimi K3.1 Pro saw significant ranking drops, widening the gap between open-source and top closed-source models from 3 to 12 points, revealing potential overestimation of open-source generalization in previous benchmarks.

This update significantly impacts the AI landscape by curbing 'benchmaxxing' and forcing developers to focus on true generalization rather than leaderboard gaming. The widened gap challenges optimistic predictions about open-source parity with closed-source models, offering a more objective and rigorous standard for capability comparison.

LATEST/Latest data shows the gap between open-source and top closed-source models widened to 12 points under v5's strict evaluation, with rankings for models like Kimi K3.1 Pro dropping significantly, highlighting the overestimation of open-source generalization in previous benchmarks.

Hourly heat curve

37 fully observed hours
Now
44.5
Peak
82.502:00
24h change
-40%

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 7 reports · Latest first
  1. IT之家 · 智能时代T2·78 pts

    Artificial Analysis v5 Benchmark Launches; DeepSeek V4.1 Flash Tops Open-Source Coding

    • Intelligence Index v5 allocates 60% weight to private/hidden test sets, significantly reducing space for training data contamination in public benchmarks.
    • FrontierCode 2.0 moves beyond unit test pass/fail metrics, scoring based on whether open-source repo maintainers would merge the PR, emphasizing engineering aspects like style and regression risk.
    • DeepSeek V4.1 Flash scored 28% on FrontierCode 2.0, the highest among open-source models, demonstrating superior code quality in real-world engineering scenarios.
  2. The DecoderT2·68 pts

    Artificial Analysis Index Update Widens Open vs Closed Model Gap to 12 Points

    • Intelligence Index v5 allocates 60% weight to private test sets to prevent models from over-optimizing for public benchmarks (Benchmaxxing).
    • Under the new standards, Kimi K3.1 Pro dropped from 4th to 9th place, and Grok 4.7 lost 12 points, indicating weaknesses in generalization for some open models.
    • The composite score gap between open models and top closed models widened significantly to 12 points, challenging previous optimistic views that open source was catching up.
  4. Hacker News AIT2·68 pts

    Artificial Analysis Releases Intelligence Index v5.0 with Nine Benchmarks

    • Version 5.0 integrates 9 specific evaluation datasets covering reasoning, knowledge, math, and programming
    • The index is a text-based English suite; image inputs, speech inputs, and multilingual performance are benchmarked separately
    • Officially claimed to offer a more useful synthesis for comparing overall language model intelligence than existing metrics
  5. Hacker News AIT2·78 pts

    Artificial Analysis Launches Intelligence Index v5 with Private Benchmarks and Progress Tracking

    • 60% of Intelligence Index v5 is based on private test sets to address score inflation caused by public benchmark contamination
    • Two new agentic benchmarks added: FrontierCode 2.0 (real-world code merging) and Terminal-Bench-Science (terminal-based research tasks)
    • Launch of Progress Index v1 provides a single continuous scale for long-term tracking of model capability evolution
  6. Artificial Analysis (@ArtificialAnlys)T1·78 pts

    Artificial Analysis Launches Intelligence Index v5 with Private Benchmarks

    • 60% of Intelligence Index v5 is based on private test sets or held-out answers, significantly reducing data contamination risks
    • Two key benchmarks added: FrontierCode 2.0 (code merge standard) and Terminal-Bench-Science (research tasks)
    • Progress Index v1 launched simultaneously, offering a single continuous scale to track long-term model capability evolution
  7. Artificial AnalysisT1·78 pts

    Artificial Analysis Launches Intelligence Index v5 with Private Datasets and New Code Benchmarks

    • 60% of v5 evaluations are based on private test sets or held-out answers, significantly reducing data contamination risks
    • Two new benchmarks introduced: FrontierCode 2.0 (real-world coding) and Terminal-Bench-Science (agentic scientific tasks)
    • Launch of Progress Index v1 provides a single continuous scale to track long-term model capability evolution