After a few of these negative interactions, I decided to generally avoid visiting ever again. It's not that I have a problem dealing with it: It's very easy to say no, or to run if needed. It's that I simply shouldn't have to deal with this BS in the first place! Please, just leave me alone! I'm leaving you alone.
In places like this, I lock the doors upon entering the car. Depending on location, I also keep the engine running, in case the car must be used in an evasive or defensive manner. (People have tried my door handles while stopped at lights, so the car is set to lock when driving too.) Be aware of your surroundings, and casually pay attention to how people are watching you.
I am very glad to live in a different nearby city, in a more sprawling area, that has much less of this activity. Due to the sprawl, it is not nearly as economically motivating for trolls to congregate.