GitHub - amaiya/onprem: A tool for running on-premises large language models with non-public data

The article introduces OnPrem.LLM, a Python package designed to run large language models (LLMs) on non-public or sensitive data, especially in environments with no internet connectivity. The tool is inspired by theprivateGPT GitHub repo and Simon Willison’s LLM command-line utility. It allows users to integrate local LLMs into practical applications. The tool supports models in GGML format and is expected to transition to the newer GGUF format in future versions.

The article provides a detailed guide on how to install and use OnPrem.LLM. It explains how to set up the tool, send prompts to the LLM, generate text to code, and speed up inference using a GPU. It also provides solutions to common issues such as SSL errors when downloading the model behind a corporate firewall. The tool can be used with any model by supplying the URL to the LLM constructor.

Key takeaways:

OnPrem.LLM is a Python package that allows running large language models on non-public data and on machines with no internet connectivity, such as behind corporate firewalls.
The package supports various functionalities including sending prompts to the LLM to solve problems, answering questions about documents, and generating code from text.
OnPrem.LLM currently supports models in GGML format, but future versions will transition to the newer GGUF format.
Users can speed up inference by using a GPU, and the package provides detailed instructions on how to install and use the necessary libraries for this purpose.

GitHub - amaiya/onprem: A tool for running on-premises large language models with non-public data

Key takeaways:

Comments (0)

Newsletter