TomBot2000: automatically finding related posts using LLMs
Dec 08, 2023 - tomhazledine.com
The author discusses how they used embeddings and GPT4 to generate related posts for their blog. Embeddings, derived from Large Language Model (LLM) technology, are used for ‘natural language search’, determining how similar two pieces of text are. This is useful for search applications and generating related blog posts. The author provides a step-by-step guide to their automated process for generating related posts. They extract relevant content from all markdown files in their directory, generate a single post embedding with OpenAI's API, find the top two most similar posts, compare these summaries, and writes this data back into the original markdown files.
The author encountered several hurdles along the way, including API rate limits, token limits, pricing, prompt engineering, inconsistency and hallucination in the output, deletion, and node memory. The author overcame these hurdles to use and optimize the script for their blog workflow. The author has open-sourced the script and published it to NPM for others to use.
Key takeaways:
The author discusses the use of embeddings, derived from Large Language Model (LLM) technology, for "natural language search" and generating related blog posts.
A step-by-step guide is provided on how to automate the process of generating related posts using OpenAI's API and a node script.
Several challenges were encountered during the development of the script, including API rate limits, token limits, pricing, prompt engineering, inconsistency and hallucination in the output, deletion, and node memory.
The author has open-sourced the script and published it on NPM, allowing others to implement a similar process for their own sites.
