Latency can be radically improved with SoundJack and tweaks to router settings, combined with audio equipment that introduces as little latency as possible. (i.e., don't use cheap USB mics.)
Do all of this, and you can collaborate really well with people in the same geographic area as you. Of course you can't get around the speed of light, but you can do a lot better than the default setup or Zoom will give you.