This is quite different from tv and film where you're just watching and not using your mind.
If I were to sit down in my book corner chair, putting on an audiobook and leaning back, I can imagine I'd have the same experience as you, but that is when I would otherwise read a book the traditional way. Perhaps this will change when I get older and wish to rest my eyes and arms while taking in a book.
Now, I use audiobooks because I can then take the dog for a walk and do other chores while listening to them. For me, it's a good way to get my mind working while my body is too. Plus, you can speed up the narration to some multiple so you're at the same pace as if you were reading anyway.
For me, it's no different than if I were a cigar roller with someone reading out a book to the lot of us.
Is it exactly the same? No, of course not. But if the alternatives to doing chores is 1) doing them with no auditory enjoyment 2) doing them with some podcast/radio station blaring topical news 3) doing them with a classic book
then, I'm going for 3. It's just the best use of time
Put another way, reading involves:
1. Parsing and interpreting the words on the page.
2. Synthesizing the information in our heads (scenes, arguments, etc.)
3. Interpreting the synthesis (Does it work? Is there subtle or implied information in the synthesis? What comes next?)
My argument is that audiobooks drastically alter step 1.
Let's not denigrate any forms of media though. They all have their unique benefits. You can use your mind or not during the consumption of any of them.