I can't say I've had this experience at all with my camera. I certainly feel more self conscious shooting publicly with my camera, but I've never had anyone look at me funny and certainly haven't had anyone approach me or get mad at me.
I recently got a grip for my phone (TELESIN Master Grip). It gives me a physical shutter button, as well as a much better ergonomic hold on the phone, and a tripod thread to attach my old reliable wrist strap. That, plus Lightroom Mobile configured for high-quality yet editable shots (eg, DNG), allowed me to build what felt like 80% of a proper camera — and yet with far, far less public obviousness.
I won't give up my rangefinder, but now I feel I have two choices: phone+grip for places where I don't feel comfortable with the camera, and then the camera everywhere else.