The open-source AI community has expressed admiration and excitement about Llama 2 Long, as it validates Meta’s open-source approach towards generative AI. It also indicates that open source can compete with the closed source, “pay to play” models offered by well-funded startups. The researchers used reinforcement learning from human feedback and synthetic data generated by Llama 2 chat to improve its performance in common tasks including coding, math, language understanding, common sense reasoning, and answering a human user’s prompted questions.
Key takeaways:
- Meta Platforms has introduced a new AI model, Llama 2 Long, which outperforms some of the leading competition in generating responses to long user prompts, including OpenAI’s GPT-3.5 Turbo and Claude 2.
- The new model is based on Meta’s open source Llama 2, but has undergone “continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled."
- The researchers made a crucial modification to the Rotary Positional Embedding (RoPE) encoding, which enabled them to ensure more “distant tokens,” those occurring more rarely or with fewer other relationships to other pieces of information, were still included in the model’s knowledge base.
- The open-source AI community has been expressing their admiration and excitement about Llama 2 Long since the paper’s release, indicating that open source can compete with the closed source, “pay to play” models offered by well-funded startups.