Aren’t the most affected group of users cloud users (and providers), not desktop users? I thought the biggest risk of specter attacks is the ability to glean information on other server residents who should be segmented off. There are many more concerns in user space which make attacking a desktop with specter pretty cumbersome for low reward wrt opportunity cost.
The performance savings of speculative execution do seem to be worth the risk on a desktop IMO.