Dao's move to Together marks his first exposure to the open-source community, which he says differs from academia in its focus on practicality and cost-effectiveness. He believes the future of AI will be shaped by open-source models that many players can contribute to and improve. However, transitioning from academia to open-source requires a shift to a more production-level mindset, a challenge that Dao and other AI researchers will continue to face as their work is quickly adapted and iterated on in the open-source community.
Key takeaways:
- Tri Dao, the creator of FlashAttention, a technique used in language model development, has joined Together, a startup focused on building open source language models. He will serve as the chief scientist.
- FlashAttention, which increases the amount of information that can go into a context window for a large language model, has been adopted by some developers and could help make large language models more practical for complex tasks.
- Together, which raised $20 million in a funding round earlier this year, is built around open source methodologies, contrasting with closed-source model developers like OpenAI.
- Dao believes the future of AI models will be a variety of open source models that are accessible and can be improved by many contributors, rather than a few companies offering proprietary AI models.