I will first do some pre-processing on the content and fetch the relevant pieces of content before giving it as a prompt to the API.
Amazing concept btw - would love to see more examples (like a chatbot for a more well-known site).
Better to question each document separately and then combine the answers into one last LLM round. Even so, there might be inter-document context that is lost - for example looking at one document that depends on details in another one. Large collections of documents should be loaded up in multiple passes, as the interpretation of a document can change when encountering information in another document. Adding one single piece of information to a collection of documents could slightly change the interpretation of everything, that's how humans incorporate new information.
One interesting application of document-collection based chat bots is finding inconsistencies and contradictions in the source text. If they can do that, they can correctly incorporate new information.
When the user asks, I try to get the relevant bits and answer the question based on that.