I do remember someone mentioning at some point that most video codecs were never intended for realtime streaming. That was in a blurb for a new video codec intended for streaming though, iirc in the context of video conferencing. Might be related, I'm not sure.