I've been here for thirteen years and have never had trouble with cops. Not once been stopped apropos of nothing and asked for ID, inspected for whatever, etc. If anything, I've been in a few situations where the cops should have given me a harder time than they did, but for whatever reason did not. I'm a white male.

Thing is, I know plenty of foreigners who have a similar experience to your own, so I know you're not wrong. I don't know what I do differently, but Japanese cops do not bother me at all.