You could also compute embeddings for the questions (don’t have to be OpenAI embeddings), and reuse the answer if the question is sufficiently similar to a prevously asked question.
Wouldn't it be almost impossible to hit a duplicate when the users each form their own question?
Another issue I see is that these chat AIs usually have "history", so the question might be the same, but the context is different: the app might have received "when was he born", but in one context, the user talks about Obama and in another, she talks about Tom Brady.
If there are ways around these issues, I'd love to hear it, but it sounds like this will just increase costs via cache hardware costs and any dedup logic instead of saving money.
The embeddings approach would increase the likelyhood of finding the same question, even if phrased slightly differently.
Regarding context, that should be a part of the input for the embeddings.
I assume only a small percentage of users would put in the same prompt twice, and even then, why would they be upset at getting the same response?