Why would anyone enable speculative execution mitigations on a computer used to build software?
Given that, I'd prefer they had shell access as a low privilege user than be able to read my ssh keys from RAM...
Obviously if you compile software as your regular linux user account like most users, you're already a sitting duck, so might as well throw in a few more vulnerabilities.