But this is exactly why we should cherish podcasts! They're one of the last vestiges of the old, weird, free Internet that we have left.
Podcasts (mostly) sound bad because the people making them are (mostly) amateurs, which means they don't know the ins and outs of audio recording. They just hooked up a crappy USB mic to their laptop and started podcasting. And once they did that, there was no central corporate gatekeeper who could turn them off and tell them to come back when they knew what they were doing.
All of which, yes, results in a lot of crappy podcasts. But consider the alternative. Do we really want to see one more wide open indie publishing channel turn into bland, pre-digested corporate mush? Is the problem with podcasting that it's not enough like Facebook? Because that's the alternative: one more medium where all the weirdness has been wrung out, except for those random bits of weirdness that happen to tickle an algorithm that makes some corporate overseer another nickel.
Maybe that is what you want, I dunno. If so, don't worry! Now that people with money have taken notice of podcasts, there are plenty of would-be corporate overseers lining up to give it to you as we speak.