I'm skeptical of the retention of information when it comes to audio books because of this very reason. Do you find you have the same sort of recall when it comes to dicsussing what you've read with audio books?
When I first started listening to audiobooks I did find that my mind would wander, but I've discovered that if I play the audiobooks at a slightly faster speed, I have to pay attention more and so my mind doesn't wander so easily.
The first audiobooks I tried listening to were books I had already read myself. Just to see how it would be if someone else narrated them. Despite having a feel for what the story was already. I'd lose track of it constantly. Akin to reading the same line over and over when tired. Which is arguably harder to rewind in audio form than in text.
Watching shortish videos on youtube or some livestream is fine for background noise. But to retain much from what was originally a 300 page novel or whatever. Having large parts of just be remembered as static feels really bad.