Having worked in VOIP for a good part of my career, there aren't actually that many hurdles left. The biggest one is IPV6. Once that is pervasive, then you have an easy way to get end to end connectivity without having use a central server to carry the content. This is really, really massive because no matter how big you are, the central server is just adding latency -- sometimes a huge amount.

The next bit is finding a good UX for doing audio setup. Right now automatic gain control is just broken. The vast majority of problems that people have (after having no audio at all) is that the audio is set up badly. If you want to see how good audio can be in a VOIP situation, take a few minutes setting up mumble properly and talk to someone. It's incredible.

Those 2 things will solve virtually every audio problem because most of the really pathological cases where echo cancellation breaks are initiated by those 2 things. However, for doing video conferencing where you have lots of people, you probably need a hardware solution. This hardware already exists, but it doesn't talk to any voip software that I know about (granted I've been out of the business for a decade). Anybody really serious about this market needs to build hardware (or better yet, pair with someone willing to build commodity hardware to open standards).

Video? It's really no problem. It can lag for seconds and it's totally fine. All of the serious problems are audio.