What is it about the conversation that you find frustrating when listening to those types of podcasts?
I like some conversational podcasts like After On where they tend to stick to discussing the topic at hand but I couldn't get into Stuff You Should Know or Joe Rogan as there was a lot of general chat in there or not high enough information density.
I don't want to hear 5 minutes of why somebody thinks something is worth doing a podcast on: just tell me the interesting stuff and I'll judge for myself.