Your lunch break should be a break, not time you're forced to spend with your colleagues.
Not every perk is about oppressing your freedom or exploiting your time and labor.
I always went out by myself when I wanted to eat alone, or sat at my desk with headphones in watching Youtube and eating.
You can communicate with people to help them understand you'd like your break to be a solitary one.
We're a social species. Building relationships is something we do. Is it work? Yes. But it's also something we need to do.