The research involved testing the compression capabilities of different sized language models on three different 1GB datasets: text, images, and audio. The findings showed that despite being trained only on text, the foundation models compressed all modalities better than methods specialized for each domain. However, there are inherent trade-offs between model scale, datasets, and compression performance. The study also suggests that these models could have practical applications for compressing images, video, and more, offering new insights into model generalization, failure modes, tokenization, and other aspects of deep learning.
Key takeaways:
- Large language models like GPT-3 have excellent general-purpose compression capabilities, able to compress text, images, and audio down to very small sizes.
- Their compression ability indicates a deep understanding of the structure and patterns in data, demonstrating they have learned general abilities beyond just processing language.
- Despite being trained only on text, these models compressed all modalities better than methods specialized for each domain.
- Their exceptional compression capabilities could have practical applications for compressing images, video and more, offering new insights into model generalization, failure modes, tokenization, and other aspects of deep learning.