The practical answer is that, if 99.9% of people out there has system that mitigates these issues, no one will bother using these exploits in the wild and you can turn off these mitigations to get the perf benefit and be reasonably sure that you won't get exploited. Unless you're targeted of course.