JEST works by selecting complementary batches of data to maximize an AI model's learnability, rather than selecting individual examples. This method, known as "multimodal contrastive learning," identifies dependencies between data points, improving the speed and efficiency of AI training while requiring less computing power. The technique focuses on high-quality, well-curated datasets, optimizing training efficiency and showing solid performance gains across various benchmarks.
Key takeaways:
- Google's DeepMind researchers have developed a new method called JEST (multimodal contrastive learning with joint example selection) that accelerates AI training and reduces the computational resources and time needed.
- The AI industry is known for its high energy consumption, but JEST could significantly reduce the number of iterations and computational power needed, potentially lowering overall energy consumption.
- JEST works by selecting complementary batches of data to maximize the AI model's learnability, improving the speed and efficiency of AI training while requiring less computing power.
- The researchers found that the JEST algorithm quickly discovered highly learnable sub-batches, accelerating the training process by focusing on specific pieces of data that “match” together, a technique referred to as 'data quality bootstrapping'.