The author faced several challenges, such as the inability of the Mixtral model to process system prompts and the lack of support for OpenAI’s function calling APIs. They solved these issues by modifying the chat template and using JSON to execute commands. They also made several modifications to the initial prompt to discourage the AI from taking actions unless explicitly asked to. The author concludes that their custom assistant works almost perfectly, despite being a bit slow due to the memory bus limitations of their 4060Ti’s.
Key takeaways:
- The author wanted to create a customized, local, and more functional smart home assistant than Siri or Google Assistant, and decided to build one using a variety of hardware and software tools.
- The author used a Protectli Vault VP2420 for firewall, a TRENDnet TEG-3102WS managed switch, two RTX 4060Ti’s for running the system, a Minisforum UM690 to run HomeAssistant, and vLLM for the inference engine.
- The author encountered several issues during the process, including the inability of the OpenAI integration to control devices and the lack of support for OpenAI’s function calling APIs by vLLM. These were solved by creating a custom integration and using JSON to execute commands.
- The final product is a smart home assistant that can be controlled using a chatbot interface, and can perform a variety of tasks in the home, from controlling lights to making coffee, all while maintaining a sassy and sarcastic tone.