Hacker News Reader

Comment by thedance | Hacker News Reader

It's a fun fact that on cloud VMs (AWS, etc) vDSO gettime doesn't exist, so if you rely on vDSO to make time measurement free, it's not.

View on HN

Maybe this is true for AWS VMs that use Xen. I believe that Linux VMs on Azure do not have this problem, since they use the Hyper-V reference time page, which can be queried from the vDSO.

I believe newer generation AWS VMs, like C5, use kvm clock source now and not xen. On the older ones switching to tsc speeds up things.

About a year ago I was writing a personal profiling framework in C++. To test it I profiled how long it took to get the time and if it was scalable (multiple threads making the same call don't interfere) ran it on Windows, and a Linux guest on a Windows host, and an AWS instance. Your post finally explains why the AWS graphs were all over the place!

Any idea why this is? I'm even more curious since you've singled out "cloud VMs" from all VMs.

My understanding is one of the reasons that the virtual time stuff in clouds doesn't work in a straightforward way is that your VM can migrate to another host, where reading TSC could give the appearance of discontinuous time, including jumping backwards in time which would be very bad. This is not a problem unless your VMs are migratory, which I guess is something I associate with GCE. And clouds seem to me to have more variety of hardware than I'd expect in my own private infrastructure, and that variety comes with a great many TSC quirks.

Intel's VMCS includes a TSC offset field as well as TSC scaling. These allow for a stable RDTSC across migrations between hosts, modulo actual time lost to migration blackout.

(I work on virtualization in GCE)

So do gettimeofday and the various clock_gettime methods [1] hit the vDSO on GCP, or do they incur a syscall, or something else?

---

[1] Not all of the clock_gettime sources hit the vDSO even on bare metal Linux on typical x86 hardware, but many of the important ones do.

Cool! I looked up more info on this and ended up here, http://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2....

But I only skimmed this and I don't see that it mentions about migration across hosts. Only that the hypervisor is able to expose the MSRs or PMCs.

Wouldn't preventing access to high-resolution clocks also be a mitigation for speculative side channels?

Reply on news.ycombinator.com