How are generative AI companies monitoring their systems in production?
Sep 20, 2023 - news.ycombinator.com
The markdown data is a comment from a user named 'marcklingen' who shared their experience of struggling with building LLM-based products. To address the challenges, they developed and open-sourced an observability/monitoring tool. This tool is used by many to monitor RAG and agents in production, run custom evaluations on the production traces, and track metrics differences across releases or customers.
The user also offered to provide assistance to anyone looking to solve specific problems related to this topic. They provided a link to their open-sourced tool hosted on GitHub for anyone interested in using or learning more about it.
Key takeaways:
The user marcklingen shared their experience of struggling with building LLM-based products.
They have open-sourced their observability/monitoring tool which is used to track RAG and agents in production, run custom evaluations on the production traces, and track metrics across different releases or customers.
The tool is particularly focused on hallucination.
Marcklingen is open to providing help on specific issues related to their tool and invites direct messages for the same.
