Comment by woeirua | Hacker News Reader

Comment by woeirua | Hacker News Reader

Parent Full thread

> Or text messages that you can listen to in the voice of the people who sent them.

Like voice snippets? We already have them. People don't want to use them.

> Or the death of the audiobook business? Any book read in any voice you want.

Press F to doubt. Human voice inflection is hard to mimic because you have to have a contextual understanding of what is being said, and what has been said in the story up to that time frame. No TTS model is capable of that today, and probably not for a long time.

> Or maybe a form of extreme compression, voice is converted to text with Whisper, sent over the wire as text, and re-created with the same voice on the receiver.

I don't understand the value prop here. The number of cases where you have access to the necessary computational resources, but _not_ adequate bandwidth is so small.

The most likely use case is probably scamming. With a small snippet of someone's voice (for example, from answering a robo call) you can now synthesize a completely reasonable sounding phone call for conning people out of their money. By the way, this attack vector was _highly_ effective against the elderly even when the attackers voice sounded quite distinct from the individual that they were posing as. The elderly have no chance against this type of fraud. There is going to be big money in authenticating individuals so that phone calls, etc can happen between trusted parties.

> Human voice inflection is hard to mimic because you have to have a contextual understanding of what is being said

I think that's correct if we are talking about the author reading his book. But if we are using another person, they have no way to learn the correct vocalization other than using the text. And that's the same input data that the AI is dealing with, so it should be possible for them to be as good as non-authors reading the book.

My intuition is that in the case of audiobook narrators who adopt different voices for different characters, VALL-E would struggle to identify which character is speaking and thus produce the correct voice for a given portion of, say, text in quotation marks. In some bits of prose it can be difficult for a human reader to determine who is speaking, for that matter, but this is a classic weakness with the current crop of AI tools, and stems from a lack of actual understanding of the text.

I disagree. In the case of audiobooks, a language model might be able to store context of the story similar to a reader, but a good reader/voice actor will be able to add emphasis or inflection to enhance the story - and decide to do so based upon the story itself.

The AI won’t have understanding of the story itself to know what is appropriate or not.

Also, you say only the author could read the story in a certain way. This kind of implies that all audiobook recordings are just 1 long take, with no direction. I am not an expert, but I would assume there’s some creative direction involved.

> but a good reader/voice actor will be able to add emphasis or inflection to enhance the story - and decide to do so based upon the story itself

But why do you think this AI (VALL-E) isn't doing the same? The training procedure implies that it must follow correct intonation in order to achieve lower test loss. Also, if there is indeed something specific about audiobooks that is harder to replicate, then we can just fine-tune these models on audiobook. Accuracy will jump considerably.

Because the intonation can depend on the events of the story, which it can’t be trained on.

I'm thinking like when you're working out or in the car, when you get a text message it's read in the Siri voice, but it'd be nifty if it was read in the voice of the person who sent the message.

Also I believe ChatGPT and other models could be trained to understand where inflection should go. It already uses the chat history as input for it's next output.

Reply on news.ycombinator.com