For example, if you listen to NPR interviews, you'll notice that questions and answers are articulate and succinct, with very few "ums" and "ahs " and such. This is because an editor removes most of the extra sounds, lengthy pauses, stammering, intakes of breath, etc.
There is also capturing the audio in decent quality. Having good microphones and equipment is just the first step. If recording indoors, one must account for room noise and echo. If outdoors, one must consider wind and background noise. Experienced audio engineers will be sure to record 1+ minutes of the ambient sound at the recording location, so they can use this if they need to insert any pauses. If recording multiple people, one must ensure they are all captured at a similar volume, especially if the recordings are all mixed together at capture time (if captured on separate mics/tracks this is easier to deal with). Some people also have a lot of "mouth noise" that is amplified by the mic, so you can hear clicks and squirts of saliva as they speak. It can be kinda gross, and there are tricks to working around this.
Editing the audio for content is also a substantial task. It takes skill to know what to leave in and what to remove, particularly if one wishes to be true to the speakers' intentions. Real conversation, if you listen to it, is messy. Lots of thought-switching mid-sentence, lengthy digressions, and things like that. Those are hard to listen to in a podcast, so a podcast that wishes to remain successful will need a skilled editor to smooth it all out. Often a 2 hour interview will become 25 minutes of usable audio. And it's especially tricky if recording with background noise, which will often betray the editing locations if one doesn't know how to mask them (usually with aforementioned ambient recording).
"Sound design" is another facet; if your podcast has music, audio clips, sound effects, etc., somebody needs to put them in the right place, set the volume levels correctly (some art, some science), fade in/out, etc.
So, yes, proper audio engineering takes some skill and makes a big difference in production quality.
But, yeah, you get into professional production values and a lot of incremental gear and work gets involved--arguably for pretty incrementally diminishing returns. Whenever I'm involved in some way with video for some company event or web clip, I'm always a bit taken back by the amount of equipment that's set up to record a few minutes of talking head footage.