Yes, AI Models Can Get Worse over Time

The performance of OpenAI's text-generating AI, GPT-4, has been found to vary over time, with a significant drop in its ability to identify prime numbers from 97.6% accuracy in March to just 2.4% in June, according to a study by researchers at Stanford University and the University of California, Berkeley. The study also found that the AI's responses became less verbose and it developed new quirks, such as appending accurate but potentially disruptive descriptions to snippets of computer code. However, it also became safer, filtering out more questions and providing fewer potentially offensive responses.

The changes in GPT-4's performance are thought to be due to the fine-tuning process used by developers, which involves introducing new information to hone the system's performance. This process can have unintended consequences, as changes made with one outcome in mind can have ripple effects elsewhere. For instance, efforts to make the AI less prone to offering offensive or dangerous answers may have inadvertently reduced its chattiness on certain topics. However, it's also possible that the AI's ability to identify prime numbers didn't actually change, but was affected by changes in the data used for fine-tuning.

Key takeaways:

The performance of AI models like GPT-4 can change over time, with some tasks showing a decrease in performance, a phenomenon known as 'model drift'.
These changes can be problematic for developers and researchers who rely on the AI's consistent behavior for their work.
The behavior of AI models can be influenced by two main factors: the parameters that define the model and the training data used to refine it. Changes to these can have unintended consequences.
While AI models can mimic reasoning, they do not actually perform reasoning in the way humans do, instead relying on patterns and relationships in the data they are fed.

Yes, AI Models Can Get Worse over Time

Key takeaways:

Comments (0)

Newsletter