Thing is, I know plenty of foreigners who have a similar experience to your own, so I know you're not wrong. I don't know what I do differently, but Japanese cops do not bother me at all.
wrt moving back, I will be leaving next year due to the job market here, so there's a data point for you. It's not impossible to find work, but if you're in tech and coming from the US be prepared for a bit of a shock.
iraq, afghanistan, syria are obvious, no? being white makes you a target for assassination, kidnapping.
china and japan have slurs against whites, see them as brutes.
being white and walking into areas in much of latin america is a bad idea. again, kidnapping and other crimes. same is true for pockets in the USA, from SF tenderloin to the bronx.
if you're an affluent white person bigger and bigger areas of the world are being shut off from you.