Hacker News Reader

Comment by youssefabdelm | Hacker News Reader

Nobody has done this well enough yet. What's required:

1. Transcribe your speech using Whisper (in that case you don't have to make an effort to speak clearly so long as you're in a relatively quiet room)

2. Get a TTS system that actually sounds good (e.g. Descript, Eleven Labs, etc.)

3. Have RAPID responses like a normal human conversation (mostly on OpenAI's side... so hopefully ChatGPT Plus fixes that)

View on HN

The bottleneck is currently TTS. The best option is probably Eleven Labs, but response times are unpredictable. GPT response times can be worked around by falling back to a faster model, but you can't do that with TTS because the voice needs to be consistent. It seems like current state of the art are diffusion models ala DALL-E, see e.g. [1] (the developer, James Betker now incidentally works for OpenAI). It's nontrivial to turn this into something that works in real-time without a decent budget, though.

Whisper (for transcription) is insanely fast and good.

1. https://github.com/neonbjb/tortoise-tts

The Eleven Labs guy told me they put out an update that made it more consistent, which maybe it is slightly better, but still unpredictable like you say. Maybe you could ask them about it again. I did suggest they make a higher queue priority available for higher level paid tiers.

rapid responses is basically the biggest problem, it is quite hard but also because you can't stream tokens into any TTS system and have it sound good. The more "complete" the corpus the better it seems to be at using the right sort of pausing and such. So it is more like a: LLM needs to be directly connected to a TTS type of issue, somehow.

If you want to have rapid responses, you basically want to have the first part of the response ready to go as soon as it is detected the user is no longer speaking. Were they asking a question? While the actual answer is being processed, start responding with "let me answer that for you..." or something like that.

This seems "well enough" to me! Faster responses sound like a BIG ask for a free service, and I imagine the best AI voice tech is highly proprietary.

Reply on news.ycombinator.com