A significant part of the problem is the lack of new, quality textual data for training new LLMs. Experts suggest that model makers may have already exhausted the most easily accessible data from the public Internet and published books. The challenge now is to find new and effective ways to scale AI models beyond the current plateau.
Key takeaways:
- AI industry watchers are concerned that the capabilities of large language models (LLMs) may be hitting a plateau, with OpenAI's upcoming model, Orion, showing a smaller performance jump than previous models.
- Unnamed OpenAI researchers have reported that Orion is not reliably better than its predecessor on certain tasks.
- OpenAI co-founder Ilya Sutskever has suggested that the era of scaling, where additional computing resources and training data could lead to improvements, may be over, and the focus should now be on finding the 'next thing'.
- Experts and insiders believe a significant part of the training problem for new LLMs is a lack of new, quality textual data to train on, with the easiest-to-access data from the public Internet and published books potentially already exhausted.