OpenAI's newest model, Orion, has not shown significant improvement over its previous model, challenging the assumption that language models will continue to improve with more data and computing power. In response, the industry is shifting its focus to improving models after their initial training. Some CEOs, like Mark Zuckerberg of Meta Platforms, believe there is still potential to build products on top of the current technology, even if it doesn't improve.
Key takeaways:
- AI companies like OpenAI are developing training techniques that use more human-like ways for algorithms to "think", which could reshape the AI arms race and have implications for the types of resources that AI companies demand.
- Despite the previous belief that "scaling up" current models through adding more data and computing power will consistently lead to improved AI models, prominent AI scientists are now speaking out on the limitations of this philosophy.
- Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, stated that results from scaling up pre-training have plateaued, challenging the assumption that larger language models (LLMs) would continue to improve with more data and computing power.
- In response to the recent challenge to training-based scaling laws posed by slowing GPT improvements, the industry appears to be shifting its effort to improving models after their initial training, potentially yielding a different type of scaling law.