Post
Building Smarter AI Benchmarks
June 19, 2025· 4 min read
Originally published on LinkedIn (June 19, 2025). Republished here with light copy-editing.
As someone who has spent considerable time analyzing how we evaluate AI reasoning capabilities, I want to share insights from a fascinating academic debate that is reshaping how we think about AI benchmarks. The exchange between two recent papers, The Illusion of Thinking by Apple and The Illusion of the Illusion of Thinking, reveals critical flaws in how we currently test AI systems and points toward building much smarter evaluation methods.
The problem: when “failure” is not really failure
The first paper tested advanced reasoning models (like OpenAI’s o3 and Anthropic’s Claude) on puzzle-solving tasks, finding what appeared to be dramatic failures at higher complexity levels. Models that could handle simple puzzles completely collapsed when faced with harder versions. This seemed to confirm fears about fundamental limits in AI reasoning.
But here is where it gets interesting: the second paper showed these “failures” were largely artifacts of flawed testing methods, not genuine reasoning limitations.
The evaluation problems were striking
Models were forced to enumerate every single move in complex puzzles, often hitting output length limits. Some test cases were literally impossible to solve (like asking for solutions to mathematically unsolvable problems). The scoring system could not distinguish between “can’t solve” and “chooses not to write 1,000+ steps.”
When researchers redesigned the test, allowing models to provide algorithms instead of exhaustive step lists, the supposed reasoning collapse disappeared. Models that had “failed” suddenly produced perfect solutions.
What this means for building better benchmarks
This debate highlights four critical principles for smarter AI evaluation.
1. Separate reasoning from output constraints
Instead of requiring models to list every step of a 100-move solution, let them demonstrate understanding through algorithms, formulas, or strategic summaries. A model that can write correct code to solve Tower of Hanoi clearly understands the problem. We should not penalize it for not wanting to print 2^15 moves.
2. Verify your tests are actually solvable
Penalizing models for recognizing impossible scenarios is backwards. If a model correctly identifies that a puzzle has no solution, that is sophisticated reasoning, not failure. Our benchmarks should reward this kind of analytical thinking.
3. Use meaningful complexity measures
Length does not equal difficulty. A 100-step repetitive task might be easier than a 10-step optimization problem. We need complexity metrics that reflect true computational challenges: search space size, optimization requirements, combinatorial explosion, not just solution length.
4. Allow diverse solution formats
Smart humans solve problems in different ways. Some sketch algorithms, others work through examples, still others provide mathematical proofs. Our benchmarks should accept multiple valid approaches to demonstrating understanding.
The bigger picture: evaluation design shapes AI progress
This is not just an academic debate. It has real implications for AI development. When we misdiagnose AI capabilities, we misdirect research efforts. If we think models cannot reason when they actually cannot format outputs properly, we will waste time on the wrong problems.
The exchange between these papers shows how easily we can create “illusions” of AI failure through poor benchmark design. But it also shows the path forward: more thoughtful evaluation that truly measures reasoning rather than compliance with arbitrary output requirements.
Building the next generation of AI benchmarks
As we develop more sophisticated AI systems, we need equally sophisticated ways to evaluate them. This means:
- Flexibility over rigidity: tests that adapt to different problem-solving approaches
- Understanding over performance: measures that capture genuine comprehension, not just correct answers
- Practical relevance: evaluations that reflect real-world reasoning challenges
- Transparent limitations: clear acknowledgment of what our tests can and cannot measure
The goal is not to make AI look better than it is. It is to understand what AI can actually do, so we can build on genuine strengths and address real limitations.
This research reminds us that in the race to build smarter AI, we also need to build smarter ways to measure intelligence. The quality of our evaluations directly impacts the quality of our progress. When we get measurement right, we can finally see clearly what needs to be improved, and what is already working better than we thought.
The future of AI reasoning depends not just on better models, but on better ways to understand what those models can really do.