Not self-hosting, but mux.com provides scalable infrastructure for video delivery. Gets extremely expensive though if you're talking about frequent livestreaming to a sizeable audience.

https://github.com/awesome-selfhosted/awesome-selfhosted#vid... may be helpful.