I think so. Mostly:

- Hosted solutions charge you for tokens. More tokens, more money. Keeping money in your pocket: generally recognized as rad.

- 1M tokens wouldn't hold the entire codebase I have open in my other window.

I'm still not sold on recall at such large context window sizes. It's easy for an LLM to find a needle in a haystack, but in most RAG use-cases it's like finding a needle in a stack of needles, and the benchmarks don't really reflect that. There's also the speed and cost implications of dumping millions of tokens into a prompt - it's prohibitively slow and expensive right now.

It's still much cheaper to run RAG in production (at least if you are using closed models). I'd love to use the entire context of GPT4, but if I do that in production it'll cost much more than using some RAG-dependent implementation.

The LLM is always out of date. RAG is here to stay.

correct me if i'm wrong but you still have to get the context populated. It's still the RAG pattern you just put more data in the prompt than before.

yes - private data, real-time data, curated data, citations with no hallucinations, RAG on tabular data, RAG on video, RAG on hierarchical mixed data, RAG over a graph

Memory-constrained environments.