For me, one of the main motivations is suspicion of ulterior motives. If it really is just "hey I like your hat okay bye" that's one thing, and is generally harmless. But usually when someone approaches me they want something, either they're selling me something, or asking me to sign something. It's not that the initial comment is necessarily an issue, it's guarding against people pretending to have an innocent interaction as a foot-in-the-door technique.
World is not your amusement park, people are entitle to NOT wanting to talk to you as much as you feel entitled to talk with everyone.
That's literally what the world is. It's the amusement park for all of us. Some of us like sharing our joy with others. It's up to you whether you are open to receiving it.
And notice the subtle difference: in response to your (IMO rather entitled) expectation that everyone should be open to talking to you, I said that the "world is not your amusement park", to which you responded shifting it slightly to "world is the amusement park for all of us" -- I agree with the latter, I just want you to not expect everyone behave like it was yours :)
All the best.