It's so painful when there is a book I am really excited to listen to and find that I just can't. Maybe I just mentally drop out mid-sentence or the words float past without carrying any meaning and it can be difficult to pin down why.
One aspect is the performance. An accent or an intonation can cause a nails on blackboard type revulsion. A morose, monotone or bored voice, ugh? For example, I just can't do any Scott Brick performance (which is pretty annoying given how prolific he is) because he trails off every sentence in this weird pitch change that doesn't match how any normal person speaks. It causes me this distressing mental dissonance because it's so disconnected to the passage being read. I suspect it's a result of him just reading out words blind and not talking like a human trying to communicate specific meanings and feelings.
This is also the problem with robot voices - the difference between a joke, a question, an assertion, a doubt, sarcasm or self-delusion can all be in the intonation. With text you might pause and revoice a passage to catch a meaning but the conveyor belt of audio means the next words are upon you giving no chance for reflection. The place where I can handle a robot voice is when using it with the text in front of me. I quite like it as a pace setter. I still have my own internal voice giving flavour but gain some reading endurance when tired or the material is boring/dry and I keep mentally wandering off.
The other aspect is the style of writing - some writing is delightful to read but awful to listen to. The parameters includes everything from sentence structure to vocabulary. A simple example is lists: perfect for quickly imbibing visually but can be maddening to hear read out. Same with numbers, or parentheticals or footnotes or... so I generally can't listen to anything technical.
The most listenable things for me is anything in a conversational style. However, even dialogue dominated works can be torture when they use certain patterns of "he said"/"she said" directions that are effectively punctuation and silent when we read but overwhelming when voiced. A first-person diary style often fits with being spoken without changes. There is a probably an AI opportunity to rewrite books to more resemble radio plays that work better as voice performances.
IMHO voicing technical works needs some brand new ideas. Real life conference talks that just narrate their paper are torture to sit through. Successfully engaging speakers use personality, humour, anecdotes and structure talks playfully with setup/reveals, hypotheticals, pauses for thought and have all sorts of rhetorical tricks to bring drama to the driest content.