For example, let's imagine a toy architecture with two registers: r0 and r1. We can create a little assembly snippet using them: "r0 = load(addr1); r1 = load(addr2); r0 = r0 + r1; store(addr3, r0)". Pretty simple.
Now, what happens if we want to do that twice? Well, we get something like "r0 = load(addr1); r1 = load(addr2); r0 = r0 + r1; store(addr3, r0); r0 = load(addr4); r1 = load(addr5); r0 = r0 + r1; store(addr6, r0)". Because there is no overlap between the accessed memory sections, they are completely independent. In theory they could even execute at the same time - but that is impossible because they use the same registers.
This can be solved by adding more physical registers to the CPU, let's call them R0-R6. During execution the CPU can now analyze and rewrite the original assembly into "R1 = load(addr1); R4 = load(addr4); R2 = load(addr2); R5 = load(addr5); R3 = R1 + R2; R6 = R4 + R5; store(addr3, R3); store(addr6, R6)". This means we can now start the loads for the second addition before the first addition is done, which means we have to wait less time for the data to arrive when we finally want to actually do the second addition. To the user nothing has changed and the results are identical!
The issue here is that when entering/exiting a VM you can definitely clear the logical registers r0&r1, but there is no guarantee that you are actually clearing the physical registers. On a hardware level, "clearing a register" now means "mark logical register as empty". The CPU makes sure that any future use of that logical register results in it behaving as if it has been clear, but there is no need to touch the content of the physical register. It just gets marked as "free for use". The only way that physical register becomes available again is after a write, after all, and that write would by definition overwrite the stale content - so clearing it would be pointless. Unless your CPU misbehaves and you run into this new bug, of course.
With SMT allowing twice the cores on a CPU for most workloads, disabling it would double the cost for most providers!
There are VPS providers that will let you rent dedicated CPU cores, but they often cost 4-5x more than a normal virtual CPU. Overprovisioning is how virtual servers are available for cheap!
Providers usually won't disable SMT completely, they'd run a scheduler which only allows 1 VM to use both SMT threads of a core. Ultra cheap VPS providers may still find that not worth the pennies though as if you sell a majority of single core VPS then the majority of your SMT threads are still unavailable even with the scheduler approach.
Fully dedicated cores aren't necessarily required because in the timesliced case the registers are unloaded and reloaded when different VMs are shuffled on and off the core. That said, they definitely prevent the cross-vm-data-leak case of this bug.
Registers are unloaded and reloaded when different processes / threads are scheduled within a running VM too. That should protect the register contents, but because of this issue, it doesn't, so I don't see why it would if it's a hypervisor switching VMs instead of an OS switching processes. If you're running a vulnerable processor on a vulnerable microcode, it seems like you can potentially read things put into the vulnerable registers by anything else running on the same physical core, regardless of context.
On the other hand, "context switching" for VMs is done via hardware commands like VMSAVE/VMLOAD or VMWRITE/VMREAD which do save/load the entire guest register context, including the hidden context not accessible by software which this CVE is relying on. Not that it isn't impossible for this to be broken as well, but it's a completely different procedure and one the hardware is actually responsible for completely clearing instead of "supposed to be reset by software".
So while the CVE still affects processes inside of VMs the loading/unloading behavior inter VM should actually behave as a working sandbox and protect against cross-VM leaks, barring the note by lieg on SMT still possibly being a problem (I don't know enough about how the hardware maintains the register table between SMT threads of different VMs to say for sure but I'm willing to guess it's still vulnerable on register remappings).
There may well be other reasons I'm completely mistaken here but they'd have to explain why the inter-VM context restore is broken not why it works for inter-process restore. The article already explains why the latter happens, but it doesn't make a claim about the former.
If we assume the register file isn't saved, just the visible registers, what's happening is the visible registers are restored, but the speculative dance causes one of the other values in the register file to become visible. If that's one of the restored registers, no big deal, but if it was someone else's value, there's the exploit.
If you look at the exploit example, the trick is that when the register rename happens, you are re-using a register file entry, but the upper bits aren't cleared, they're just using a flag to indicate the bits are cleared; then when rolling back the mispredicted vzeroupper unsets the flag, the upper bits of the register file entry are revealed.