What are the possible pitfalls of _not_ runnning spectre et all mitigations, if you're _not_ hosting other people's code, but on your own hardware?
But I believe you're right in that those exploits are not as dangerous in a non-shared hosting scenario.
An attacker could carefully craft network packets to force the control flow of existing software to manipulate the CPU state. They could and then use the timing differences in network packets to read data out.
Would be painfully slow, but theoretically possible.
But if you're just running a local home system the chances of being targeted with such an unreliable attack are virtually zero.
I think that's the catch: it's the set of cases where someone can get code execution without being able to directly retrieve the targeted data, which is already pretty low, and where the best mitigation technique is to harden against these attacks rather than use a dedicated service (e.g. if you're using an HSM for crypto or have a dedicated SSL-offload box, you don't have to worry about keys being compromised when someone finds a bug in the much greater exposed surface area of your application).
So, if you have a system which you cannot expose it is a pretty big damper on performance.
If you're not sharing your hardware with other code you don't trust, it's not a problem.
What code do you actually trust? (to be incorruptible/unhackable, etc)
When I install software I am implicitly trusting it, whether that software is trustworthy or not. If it is exploitable that doesn't mean I didn't trust the software. On my personal computer, the expectation is that only the software I have implicitly trusted will be running. If I install something that isn't trustworthy but trust it anyway it's game over. If it's exploited, that's a different issue to trusting it.
Shared VMs are a totally different ball game. All of a sudden I am sharing resources with a potentially bad actor.whethrt or not I trust them is kind of indifferent, I don't get to choose who I colocate with, and ultimately that's the difference.
Being pedantic about this is not helping anyone actually be more secure: the odds are orders of magnitude greater that someone will be compromised by a different security problem than by the diminishing returns trying this attack against a current browser, and that the attacker will directly obtain the data they want without needing a side-channel attack.
The list of sites I have whitelisted JS for is extremely short.
As for app stores there's typically easier malware attack vectors if you manage to get an app installed & past the scanners in place with that typical back & forth escalation game. And those run in process-level sandboxes with clamped down syscalls. You can't exploit a spectre BPF attack if you can't use BPF at all due to selinux rules, for example.
Moving past consumer what else are you really going to attack? Sure you could rent some virtual hosting somewhere and hope you get lucky attacking a neighboring VM, but what's really your end goal there to justify the time & risk? It's not like you can reliably pick a target with that, you just have to go fishing and react to what you may or may not find.
However, when a vulnerability is found, then spectre etc. make it easier to abuse that vulnerability to do something useful.