The research also explored the practical implications of these hallucinations by uploading a fake Python package, which surprisingly received over 30,000 authentic downloads in three months. The article concludes with recommendations for users to exercise caution when relying on LLMs and to thoroughly verify any unfamiliar software packages before integrating them into a production environment.
Key takeaways:
- The research conducted six months ago revealed a new attack technique called AI Package Hallucination, which uses LLM tools to spread non-existent malicious packages based on model outputs.
- The recent research aimed to scale up the previous one, with more questions asked, more languages checked, and more models tested. The research found that for almost 30% of the questions, the models recommended at least one hallucinated package.
- The research also explored the possibility of cross-model hallucinations, where the same hallucinated package appears in different models, and tested the attack effectiveness in the wild.
- The researcher recommends caution when relying on Large Language Models (LLMs) and when using Open Source Software (OSS), advising thorough cross-verification and comprehensive security scans before integrating any unfamiliar package into a production environment.