We are working on a similar application and we have the same observation: external data is required to avoid hallucinations, especially if you go to less known places. It's absolutely the case with GPT-3.5 and often with GTP-4.
We will release our new content in the next few days. We are finally wondering about eating the cost of expensive TTS or going with a cheap option for okay results. Can I ask which option you used for TTS?