Announcing ARC-AGI-3
The only unsaturated agentic intelligence benchmark in the world
Humans score 100%, AI <1%
This human-AI gap demonstrates we do not yet have AGI
Most benchmarks test what models already know, ARC-AGI-3 tests how they learn
NVIDIA Nemotron 3.5 Super VL from @NVIDIAAI on ARC-AGI (Verified):
- ARC-AGI-2: 1.94%, $0.02/task
- ARC-AGI-1: 22.67%, $0.03/task
Nemotron 3.5 Super VL is the first NVIDIA model we've evaluated on ARC-AGI.
DeepSeek V4.1 Flash 1017 from @deepseek_ai on ARC-AGI (Verified):
- ARC-AGI-2: 78.1%, $0.16/task
- ARC-AGI-1: 94.5%, $0.06/task
The best ARC-AGI-2 scores for DeepSeek's Flash models rose from 61.4% (V4 Flash 0731) to 72.9% (V4.1 Flash) to 78.1% (V4.1 Flash 1017).
Beam 1.0 from @reflection_ai on ARC-AGI (Verified):
- ARC-AGI-2: 23.5%, $0.61/task
- ARC-AGI-1: 74.0%, $0.31/task
Beam’s best scores came at xhigh reasoning, but high was close on ARC-AGI-2: 20.0% versus 23.5%, at about 43% lower cost.