I'm one of those people you mention who went back because it was hard in social settings. For context, I stopped being vegetarian because I wanted to be more free to eat with my friends and family. It was incredibly hard growing up, as I could rarely have dinner at a friends house or eat meals in Boy Scouts. People didn't want to eat food that I cooked because there wasn't meat in it, so no one ate at my house. Even going out to restaurants was hard. Telling people I didn't want to go to their favorite burger joint on a road trip is challenging when you do it all the time. I had to either bow out of social situations and do my own thing, bring my own food, or not eat.
Going through that for my entire childhood definitely changed me. You have to fight against the ingrained American mentality that meat always belongs at the table. You have to make people leave their comfort zones or do extra work in order to accommodate your needs. You have to be comfortable being "that person" who isn't normal. It wears on you over time, and at some point it just wasn't worth it any more.
Now, I eat meat occasionally when I go out and I cook it for special occasions at home. Otherwise my meals are vegetarian most of the time. I enjoy meat in a limited capacity, and I personally want to keep it that way. I wish those who advocated for vegetarianism and veganism had some more capacity to see consumption reduction as aligned their cause. Black and white views exist on both sides here, and I think the solution is somewhere in the middle. Eat meat when its special. In my opinion it tastes better and you appreciate it more!