Sign up to save tools and stay up to date with the latest in AI
bg
bg
1

Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience

Jun 02, 2025 - news.ycombinator.com
The article describes a system designed to enhance the problem-solving capabilities of large language models (LLMs) by allowing them to learn and refine strategies over time, inspired by Andrej Karpathy's concept of a "third paradigm" for LLM learning. Instead of relying on static system prompts, the system builds a database of effective strategies for various problem types. When faced with a new problem, the LLM selects and applies the most relevant strategies, evaluates their effectiveness, and refines them as needed. This approach has shown improvements in performance on math benchmarks, such as an 8.6% increase on Arena Hard and a 6.67% increase on AIME24. The system is implemented as an open-source plugin for optillm, compatible with any OpenAI-compatible API, and operates in two modes: inference-only and learning mode.

The system aims to bridge the gap between sophisticated system prompts used in production AI and the basic prompts typically employed by most developers. By allowing the model to improve its problem-solving skills based on the types of problems it encounters, the approach offers a transparent and editable way to enhance LLM performance without the need for pretraining or fine-tuning. The strategies are stored in human-readable JSON format, ensuring transparency and enabling users to understand the system's learning process. The article invites feedback and discussion on this novel approach to LLM learning.

Key takeaways:

  • System allows LLMs to automatically learn and improve problem-solving strategies over time by building a database of effective strategies for different problem types.
  • Strategies are stored as human-readable JSON, enabling inspection and editing, and the system has shown improvements on math benchmarks like Arena Hard and AIME24.
  • Implementation is an open-source plugin for optillm, compatible with any OpenAI-compatible API, featuring inference-only and learning modes.
  • Approach bridges the gap between sophisticated system prompts used in production AI and basic prompts, inspired by Andrej Karpathy's "third paradigm" for LLM learning.
View Full Article

Comments (0)

Be the first to comment!