moshi by Kyutai seems to have beaten your approach by about 500ms, and they're going to release open source.

https://www.youtube.com/live/hm2IJSKcYvo

hn discussion here: https://news.ycombinator.com/item?id=40866569