I think famous not wealthy people (especially those that are victims of internet mobs) end up in hiding and it’s a pretty bad situation for them. There’s also the middle case too - I’m not sure what level of wealth Sam Harris has, but I don’t know if he’s able to afford this kind of security (and I know he gets a lot of death threats).

At some threshold of fame it seems like you just attract crazy people and the risk of them attacking you goes up substantially.