It could be related to how many cpu's you allocate for the vm. I use 1 single cpu, and networking is extremely fast, very low ping times. (mainly usr the headless server version, no x11)