Amazon Paper Finds Rule-Following AI Benchmarks Riddled with Answer-Key and Scorer Bugs

1 reports archived1 independent sourcesupdated 11/9/2026, 05:24:38

Synthesis & Latest Updates

1 Sources Cross-Validated

A new Amazon paper audited nine public rule-following benchmarks such as RuleArena and LegalBench and found that their reference answers and grading code contain errors, causing correct model responses to be marked wrong. In LegalBench's tax scorer, decimal points were stripped, turning $259,487.13 into 25,948,713; after the fix, three models' accepted answers rose from 8-9 out of 96 to 87-91. The authors urge checking scorer code and hand-reviewing failed answers before trusting low scores.

LATEST/A new Amazon paper audited nine public rule-following benchmarks such as RuleArena and LegalBench and found that their reference answers and grading code contain errors, causing correct model responses to be marked wrong. In LegalBench's tax scorer, decimal points were stripped, turning $259,487.13 into 25,948,713; after the fix, three models' accepted answers rose from 8-9 out of 96 to 87-91. The authors urge checking scorer code and hand-reviewing failed answers before trusting low scores.

Hourly heat curve

55 fully observed hours

Now

0.0

Peak

17.106:00

24h change

-100%

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first

Rohan Paul (@rohanpaul_ai)T2Nov 9, 05:24·71 pts
Amazon Paper Finds Rule-Following AI Benchmarks Riddled with Answer-Key and Scorer Bugs
- Audited nine public rule-following benchmarks and found errors in answer keys and grading code
- LegalBench's tax scorer stripped decimal points; fixing it raised accepted answers from 8-9/96 to 87-91/96
- Low scores may reflect benchmark bugs rather than model ability; read scorer code and hand-check failures
View Related Reports & Analysis Original Source