Ask HN: Are you training and running custom LLMs and how are you doing it?

The author is researching methods and projects for training and running Language Models (LLMs) locally. They have listed several engines and APIs for this purpose, including vllm, ollama, llama.cpp, llama-cpp-python, llm-engine, Lamini, GPT4All, SkyPilot, HuggingFace Transformers, and RAGStack. They also mention a user interface tool by Simon for LLMs.

In addition, the author discusses various quantization bits like AutoGPTQ, QLoRA, bitsandbytes, and SkyPilot QLoRA. They also provide links to two video guides on the topic and a list of references for further reading. The references include links to the GitHub repositories of the mentioned projects and tools, a link to the Lamini website, and a link to Simon's tool.

Key takeaways:

The author has been researching methods and projects for training and running Language Models (LLMs) locally and is interested in what others have been using, including Pytorch/Transformers.
Several engines/APIs for LLMs are mentioned, including vllm, ollama, llama.cpp, llama-cpp-python, llm-engine, Lamini, GPT4All, SkyPilot, HuggingFace Transformers, and RAGStack.
There are also tools and packages for quantization of LLMs, including AutoGPTQ, QLoRA, bitsandbytes, and SkyPilot QLoRA.
Simon's `llm` tool is mentioned as a UI/Interface for LLMs, and there are video guides available for further learning.

Ask HN: Are you training and running custom LLMs and how are you doing it?

Key takeaways:

Comments (0)

Newsletter