> The 10M context ability wipes out most RAG stack complexity immediately.
Large contexts are not always better. For starters, it takes longer to process. But secondly, even with RAG and the large context of GPT4 Turbo, providing it a more relevant and accurate context always yields better output.
What you get with RAG is faster response times and more accurate answers by pre-filtering out the noise.
An actually useful RAG would be to convert text to Q&A and use Q's embeddings as an index. Large context can make use of in-context learning to make better Q&A.
We also embed the actual text, though, because I found that only doing the questions resulted in inferior performance.
1. Get text from page/section/chunk
2. Generate possible questions related to the page/section/chunk
3. Generate an embedding using { each possible question + page/section/chunk }
4. Incoming question targets the embedding and matches against { question + source }
btw, 10M tokens is 78 times more context window than the newest GPT-4-turbo (128K). In a way, you don't need 78 GPT-4 API calls, only one batch call to Gemini 1.5.
People also seem to forget that the average is 1b words that are read by people in their entire LIFETIME, and at 10m, with nearly 100% recall thats pretty damn amazing, i'm pretty sure I don't have perfect recall of 10m words myself lol
It can also be a good alternative for fine-tuning.
And the use case of a code base is a good example: if the ai understands the whole context, it can do basically everything.
Let me pay 5€ for a android app rewritten into iOS.
For any use case where you want contextual results, you need to be able to either filter the search scope or use RAG to pre-define the acceptable corpus.
Unless you can get nearly perfect recall with millions of tokens, which is the claim made here.