Sign up to save tools and stay up to date with the latest in AI
bg
bg
1

Building a fully local LLM voice assistant to control my smart home

Jan 13, 2024 - johnthenerd.com
The author discusses their dissatisfaction with the limitations of Siri and Google Assistant, and their decision to create a custom, local, and sassy AI assistant. They used a Protectli Vault VP2420 for the firewall, a TRENDnet TEG-3102WS managed switch, two RTX 4060Ti’s, a Minisforum UM690 to run HomeAssistant, and a vLLM for the inference engine. They chose the Mixtral model from Mistral AI due to its VRAM vs performance trade-off. They also downloaded a custom GlaDOS voice model from HuggingFace and created a custom integration to control their devices.

The author faced several challenges, such as the inability of the Mixtral model to process system prompts and the lack of support for OpenAI’s function calling APIs. They solved these issues by modifying the chat template and using JSON to execute commands. They also made several modifications to the initial prompt to discourage the AI from taking actions unless explicitly asked to. The author concludes that their custom assistant works almost perfectly, despite being a bit slow due to the memory bus limitations of their 4060Ti’s.

Key takeaways:

  • The author wanted to create a customized, local, and more functional smart home assistant than Siri or Google Assistant, and decided to build one using a variety of hardware and software tools.
  • The author used a Protectli Vault VP2420 for firewall, a TRENDnet TEG-3102WS managed switch, two RTX 4060Ti’s for running the system, a Minisforum UM690 to run HomeAssistant, and vLLM for the inference engine.
  • The author encountered several issues during the process, including the inability of the OpenAI integration to control devices and the lack of support for OpenAI’s function calling APIs by vLLM. These were solved by creating a custom integration and using JSON to execute commands.
  • The final product is a smart home assistant that can be controlled using a chatbot interface, and can perform a variety of tasks in the home, from controlling lights to making coffee, all while maintaining a sassy and sarcastic tone.
View Full Article

Comments (0)

Be the first to comment!