The World Economic Forum predicts a significant growth in affordable devices that can run GenAI models offline, including PCs, smartphones, and IoT devices. Offline models are more private, lower latency, and more cost-effective than cloud-hosted models. However, the democratization of tools to run and train models could potentially open the door to malicious actors. Despite this, proponents of apps like Chat with RTX argue that the benefits outweigh the harms.
Key takeaways:
- Nvidia is releasing a tool called Chat with RTX that allows owners of GeForce RTX 30 Series and 40 Series cards to run an AI-powered chatbot offline on a Windows PC.
- The tool can be customized to connect to documents, files, and notes for querying, and it supports text, PDF, .doc, .docx, and .xml formats.
- Chat with RTX has limitations including not remembering context and the relevance of responses being affected by various factors such as question phrasing and the size of the fine-tuning dataset.
- The World Economic Forum predicts a growth in affordable devices that can run GenAI models offline due to benefits such as increased privacy, lower latency, and cost-effectiveness compared to cloud-hosted models.