Then, your VM is not the only VM sharing the bare metal. Same thing applies, only on a slightly higher level.
As long as you share the metal with other stuff (be it containers, binaries, VMs), there's always competition for resources, and your average time becomes your real world time.