Hacker News Reader

Comment by evancox100 | Hacker News Reader

I just posted a comment asking about multiprocessing/preemption - how do the root OS and virtualized OS share the CPU core when the virtual OS assumes it is the only thing responsible for scheduling thread execution?

View on HN

The virtual CPU is (typically) represented as a single OS thread. Depending on how you've configured the virtualization hardware, it may look like a thread that's always runnable, or it may block whenever the guest kernel halts that CPU (while waiting for an interrupt). While the VCPU is running the guest OS is free to context switch what's running there. If it halts, and the CPU has been set up to generate a VM exit on halt, it may take longer to wake it up: and IPI from another VCPU has to hit the host OS, which then has to put the target VCPU thread back into a runnable state, and that thread has to begin running and issue a VMRESUME before the IPI completes. If it sounds like this could have disproportionate performance impact, it can. It works at all because operating systems tend to have fairly relaxed timing expectations (in addition to being largely event driven). Also, virtualization isn't the only source of potential stalls in a modern processor. Consider that a transition from the lowest power state up to a running state can take tens to (iirc) hundreds of microseconds. Generally a pass through the scheduler would be much faster, unless things are contended, _or_ unless the host scheduler needs to wake up a physical CPU -- now the costs compound!

A related problem to consider in the non-virtualized space: the Go runtime scheduling Goroutines onto OS threads. It has to contend with many of them same "is the target thread running" problems, and exhibits some of the same pathologies. Also, once the number of runnable host threads approaches the number of physical threads, the potential for contention, stalls, etc. increases dramatically. Once it exceeds that threshold, it becomes unavoidable. Collectively, all of these issues are commonly filed under the heading of "jitter" -- things that break the illusion of a continuously executing stream of instructions on a computer dedicated just to one thread.

There is a user space thread on the host for each virtual CPU core being offered to the guest. The host OS runs the virtual CPU by scheduling this thread. The host kernel scheduler subdivides the time on the virtual CPU that has been given to it by the host for its threads/processes.

Most hypervisors make this information available to the guest, you can see it as st% in top for example.

Just to expand the abbreviation: steal time percentage

It's important to know for the OS because otherwise your clock will be way off. There's probably other issues that would pop up too that I'm just forgetting right now.

Wouldn't this only be detectable to a guest that's aware it's being virtualized (that is, a paravirtualized one?)

I have an impression that terms "paravirtualization" and "full virtualization" are fuzzy nowadays. KVM, for example, reports steal time. Yet, KVM is considered to be full virtualization.

Also, guest can still guess that it is a mere guest by looking into loaded drivers. Many drivers, like virtio-blk, inevitably lead to conclusion that environment is virtualized, no matter what cpu says.

I might be wrong, but I think "paravirtualized" means that you don't have to modify your kernel to work with virtual CPU / virtual memory. It doesn't mean "guest isn't aware it's being virtualized"

Paravirtualization means that the virtualization does not entirely pretend to be real hardware, but has special interfaces that require OS modifications or special drivers to use (although sometimes optional. E.g. KVM offers full virtualization, but also has the option of providing VirtIO devices, which you might call paravirtualization)

Reply on news.ycombinator.com