The article also highlights the challenges faced in the process, such as handling indentation issues in Python and the limitations of Large Language Models. Despite these challenges, SuperCoder 2.0 managed to solve 101 out of 300 instances in the SWE-Bench Lite dataset. The system performed particularly well with the Django repository, solving 46 instances. The team at SuperAGI plans to improve file and method localization and address other identified bottlenecks for further development.
Key takeaways:
- SuperCoder 2.0, a multi-agent system leveraging GPT-4o and Sonnet-3.5, has achieved a 34% success rate in SWE-bench Lite, ranking #4 globally and #1 among all open-source coding systems.
- The system uses a two-tiered approach to code search and generation, first identifying relevant sections of the codebase and then creating a patch or set of modifications to address the problem.
- SuperCoder 2.0 managed to solve 101 out of 300 instances in the SWE-Bench Lite dataset, demonstrating its ability to solve coding problems across diverse repositories.
- Despite its success, the team behind SuperCoder 2.0 acknowledges the need for further improvement in file and method localization, and is exploring the use of Repo Map and other strategies to enhance its performance.