Those are all problems with a specific, objectively crappy container, as you've noted. There are many alternative containers and no reason the podcasting community couldn't come up with its own, such as by extending Matroska. This obviously creates separate problems, but I'd like to see the podcasting community start to shape its own destiny a bit rather than expect others to innovate for them or kludge older formats to do things they weren't supposed to do.
Why not a podcasting specific container that handles streaming and fixed cases, includes support for all the novel stuff podcasters want to do, and so on? Container development is one of the easier parts of multimedia, and there's too much thinking around a specific codec being tied to a specific container (just like your comment). Legacy support is of course the big elephant in the room, but I'd venture a well-founded guess that the average podcast listener is listening on a device that supports apps. (Maybe I'm wrong.)
I can't think of a single good container format. They're all rubbish. Even Matroska is arcane and very poorly documented, and suffers from the end goals and types of media it is used for being encoded in the design. Containers are a giant space in which to innovate and have been that way for years. Unfortunately, open source multimedia development is largely tied up by a certain group of people with a certain background and very specific use cases in mind.
That context from the podcast would have been awesome in this piece. Why wasn't it there?