I think that's correct if we are talking about the author reading his book. But if we are using another person, they have no way to learn the correct vocalization other than using the text. And that's the same input data that the AI is dealing with, so it should be possible for them to be as good as non-authors reading the book.
The AI won’t have understanding of the story itself to know what is appropriate or not.
Also, you say only the author could read the story in a certain way. This kind of implies that all audiobook recordings are just 1 long take, with no direction. I am not an expert, but I would assume there’s some creative direction involved.
But why do you think this AI (VALL-E) isn't doing the same? The training procedure implies that it must follow correct intonation in order to achieve lower test loss. Also, if there is indeed something specific about audiobooks that is harder to replicate, then we can just fine-tune these models on audiobook. Accuracy will jump considerably.