Take a look at Piper. It's the tts solution used by the open source home automation project HomeAssistant. Produces decent quality speech in a couple seconds on raspberry pi class hardware.