In December, OpenAI introduced its o3 “reasoning” AI model, collaborating with the creators of ARC-AGI, a benchmark assessment for evaluating advanced AI capabilities, to highlight the model’s potential. Recently, revisions have been made to these results, revealing that they are somewhat less impressive than initially thought.
The Arc Prize Foundation, responsible for maintaining and managing ARC-AGI, updated its estimation of the computing expenses associated with the o3 model. Initially, the best-performing configuration, o3 high, was estimated to cost approximately $3,000 per problem solved using ARC-AGI. This has now been revised to potentially around $30,000 per task.
This update is significant, demonstrating the potential high costs associated with sophisticated AI models for specific tasks, especially during their early stages. Although OpenAI has not yet priced or released the o3 model, the Arc Prize Foundation considers the pricing of OpenAI’s o1-pro model as a reasonable comparison.
The o1-pro model is currently OpenAI’s most expensive offering. Mike Knoop, co-founder of the Arc Prize Foundation, mentioned to TechCrunch that o1-pro is likely a closer comparison for the true cost of o3 due to the amount of test-time compute used. Despite this, there remains uncertainty until official pricing is announced, thus o3 remains labeled as a preview on their leaderboard.
Given the computing resources reportedly employed by the o3 high configuration, a high price point would not be surprising. The Arc Prize Foundation has indicated that o3 high used 172 times more computing power compared to o3 low, its lowest-computing counterpart, to handle ARC-AGI tasks.
There have also been ongoing rumors about potentially high-priced plans OpenAI may be considering for enterprise clients. Early in March, reports emerged that OpenAI might charge up to $20,000 monthly for specialized AI “agents” such as a software developer agent.
Some may contend that even OpenAI’s most expensive models are still more cost-effective compared to traditional human contractors or employees. However, AI researcher Toby Ord highlighted in a post that these models may not always be as efficient, citing that o3 high required 1,024 attempts per task in ARC-AGI to achieve its top performance.
