The high costs challenge the notion that scaling AI models with more processing power and data is sustainable, as the gains achieved through improved reasoning rather than scaling alone still incur significant expenses. François Chollet, the benchmark's creator, notes that while o3 approaches human-level performance, it is not yet economical, with human task-solving costing about $5 per task. Despite these challenges, there is optimism that cost-performance will improve over time. The public release of a "mini" version of o3 is expected in January.
Key takeaways:
- OpenAI's o3 model is its most powerful AI yet, but it is extremely costly to run, with high-compute tasks costing over $1,000 each.
- The o3 model scored 87.5% on the ARC-AGI benchmark, significantly outperforming its predecessor, o1, which scored 32%.
- Despite its high performance, the o3 model's costs raise concerns about the sustainability of improving AI through scaling alone.
- Chollet suggests that while o3 approaches human-level performance, it is not yet economical, though cost-performance may improve in the future.