Promptfoo

Promptfoo Overview

Promptfoo is an advanced tool designed to enhance the quality of Language Learning Model (LLM) outputs. It allows users to create a list of test cases, set up evaluation metrics, and select the best prompt and model. It can be integrated into existing test/CI workflows and is currently used by LLM apps serving over 10 million users.

Promptfoo Highlights

• Enables creation of a representative sample of user inputs to reduce subjectivity when tuning prompts.
• Offers built-in metrics, LLM-graded evals, and the ability to define custom metrics for comprehensive evaluation.
• Allows comparison of prompts and model outputs side-by-side for optimal selection.