Curious what exactly you tried - H.264 is a codec, and there's a bunch of ways of delivering it to the client (HLS, WebRTC, some have build WebSocket-based streaming, ...), and I'd expect that the main latency is hidden there, not in the decoding?
And I think you're right on the encoding latency. I believe that H.264 buffers a little bit before it makes a decision on how to compress the frames, where as MPEG1 doesn't? I could be completely wrong but my gut is telling me that MPEG1 is basically independent, slightly-modified JPEG frames.
Not that simple, but not that far off.
Basically MPEG1 has key-frames (“JPEG” frames) and (forward) delta-frames (diffs).
From my understanding H264 has many improvements, including reverse delta-frames.
My guess is that disabling those in the encoder will improve real-time capabilities.
I didn't realize that you could disable those features in the encoder. I'll have to look into x264 tunables that lower latency. Might be interesting.