This sounds a lot like my own side project's goals, but I've made it work inside the browser using web-rtc. There's a (somewhat) usable prototype online.

https://www.hyperhyperspace.org