I assume you don't find that; is that because you very rarely feel the need to stop and think, or because it turns out that doing so isn't significantly impeded by having an audiobook playing?
(More generally I'd expect that sort of book to have easier and harder bits and I'd likely want to read them at different speeds. But I could believe that the advantages of doing this are enough to make up for it being harder to do that.)
Disclaimer: I basically never listen to audiobooks, so my guesses at what might or might not work well for me are (1) sheer guesswork and (2) not very interesting except in so far as other people have the same sort of experience I'm guessing I would.