I was able to build something that does all this, more or less, in a couple weeks. It works really well.

I wanted to be able to transcribe and diarize in realtime though, which is much harder. Didn't manage to make that happen.