Yea, It is as if people expect you to pull out the phone and stare at it if you have nothing to do in a public place. Imagine someone catching you, being lost in your own thoughts, in somewhere like that. They would suerly peg you for a weirdo.
Just start talking to yourself out loud - they'll quickly assume you're perfectly normal.
I think the problem for some may be that not physically doing anything in a public space opens them up to strangers approaching with unwanted conversation