The problem is that context windows are very short. 200k-1M tokens or so. This means that the model needs to focus down on very specific information if possible. This is what makes tool using, reasoning, and agentic AI very powerful. The model can find the most relevant information it needs within its limited context and generate relevant answers to questions. The LLM pulls from web searches, documentation, long term memories in graph databases, and database queries to answer the questions using real information.