I think the latency is the killer for our remote conference calls. So much extra delay and effort goes into not talking over each other that isn't an issue in a live meeting.
To test your network, go to https://www.dslreports.com/speedtest and see if you have significant jitter or bufferbloat. If your rating is an A, then your good to go, otherwise its time to clean up your local network :P