Structure is an editing problem, but there are lots of people who listen to podcasts sped up, or with silences removed. Apps like Overcast will do this for you.
Some podcasts, like Hardcore History, are worth the 5 hour runtime and they're so rarely published it doesn't really matter.