Lowering hls_fragment[1] to 1s should help out a little with the latency. This is what I've done in the past with a similar project as yours, Open Streaming Platform[2], with good results.
Since you mentioned WebRTC for streaming to provide even lower latency, this is what I believe Project Lightspeed has achieved[3]. Might be something you'd be interested in, if you hadn't already seen it.
[1] https://wiki.openstreamingplatform.com/Usage/Streaming#osp-n...
[2] https://wiki.openstreamingplatform.com/Usage/Streaming#osp-n...