GitHub - habedi/cogitator: A Python toolkit for chain-of-thought prompting 🐍
May 15, 2025 - github.com
Cogitator is a Python toolkit designed to facilitate experimentation with chain-of-thought (CoT) prompting methods in large language models (LLMs). CoT prompting enhances LLM performance on complex tasks by guiding models to generate intermediate reasoning steps, thus improving interpretability by providing insights into the model's reasoning process. The toolkit supports popular CoT strategies and frameworks, including Self-Consistency CoT, Automatic CoT, Least-to-Most Prompting, Tree of Thoughts, Graph of Thoughts, and Clustered Distance-Weighted CoT. It offers a unified sync/async API, supports OpenAI and Ollama as LLM providers, and includes a customizable benchmarking framework.
Installation instructions are provided for setting up Cogitator, including examples of using the Self-Consistency CoT strategy with Ollama. The toolkit also features a documentation section, a benchmarking framework for evaluating CoT strategies on datasets like GSM8K and StrategyQA, and guidelines for contributing to the project. Cogitator is licensed under the MIT License, and users are encouraged to cite the project in research using the provided citation information.
Key takeaways:
Cogitator is a Python toolkit designed to enhance large language models (LLMs) using chain-of-thought (CoT) prompting methods.
The toolkit supports both synchronous and asynchronous APIs, and works with OpenAI and Ollama as LLM providers.
It includes implementations of popular CoT strategies such as Self-Consistency, Automatic CoT, and Tree of Thoughts.
Cogitator offers a customizable benchmarking framework to evaluate CoT strategies on datasets like GSM8K and StrategyQA.
