I agree about unedited podcasts where people just talk to each other, but there are excellent podcasts that are closer to documentaries or audio books (Hardcore History, for example). They're pre-written, heavily edited, and performed well.
And that's just fine, except I don't have any interest in sitting and listening to such things. Other people are totally welcome to enjoy them; I just spend my free time on other things.
Generally, you don't just sit and listen. You do it while doing something else that is non-audio, like commuting or exercising or cleaning or cooking or...
I don't multitask. I follow the philosophy of committing the entirety of my attention to one task at a time.