Sometimes I'm on a very specific diet, and I do like getting out of the office from time to time, so I'd hate to feel like an outcast for not eating with everyone everyday.
I think it might be more in keeping with the spirit to suggest group lunch outings just to get out of the office. My old team at IBM used to take our lunches to the park across the street by the Charles River when the weather was nice or take lunch bicycle rides.
Our dept was 7 people at it's largest and it had always had a tradition of ordering from one restaurant together. It was a very tedious process to get everyone to agree and hellish for people with dietary restrictions. As for being seen as an outcast if you brought your own food, or didn't eat with the group, yes , the manager tended to read you the riot act if you tried to go out for lunch or brought your own.
Folks also bring their own food sometimes. I'm sure something can be worked out for you too if you have special dietary needs and what we have isn't appropriate.