The study also found that an automated process of trying numerous variations of prompts and tweaking the language based on how much it improved the chatbots’ accuracy was more effective than the researchers' attempts to frame questions with positive thinking. The researchers suggested that these chatbots, trained on billions of lines of text from the real world, might have picked up on human tendencies to give more accurate responses when pressured or encouraged. However, they admitted that they don't fully understand how AI language models work.
Key takeaways:
- Research from software company VMware shows that chatbots perform better on math questions when prompted to pretend they're on Star Trek.
- The study used an automated process to try numerous variations of prompts and tweak the language based on how much it improved the chatbots’ accuracy.
- The most effective prompts exhibited a degree of peculiarity, with one of the models yielding the most accurate answers when asked to start its response with the phrase “Captain’s Log, Stardate [insert date here]:.”
- The researchers have no idea why Star Trek references improved the AI’s performance, highlighting the fact that even the people who build and study AI language models don’t fully understand how they work.