The author believes that RAG (Retrieval-Augmented Generation) is a temporary solution until infinite context can be virtually figured out. They compare the context of Language Models (LLM) to cache levels, where the first level is small but fast (similar to working memory), and subsequent levels are larger but slower. They view RAG as a flawed version of attention mechanisms, used to focus on relevant documents.
However, the author points out that RAG systems are not designed to minimize loss, but rather operate on a similarity score. They caution that this is their personal opinion and they could potentially be incorrect.
Key takeaways:
The author believes RAG is a temporary solution until virtually infinite context is figured out.
They compare LLM context to cache levels, with varying sizes and speeds.
RAG is seen as a poor version of attention mechanisms, used to focus on relevant documents.
The author criticizes RAG systems for not being trained to minimize loss, but rather to score similarities.
