I was actually fairly well able to deal with the personalities and political stuff. I've always been fairly easy to get along with and able to establish good working relationships with a wide variety of people. (Even the jerks, although I've become increasingly less tolerant of such behaviour.)
It was the physical environment that did me in. I was responsible for fairly essential financial analysis of about a billion dollars in revenue, at one point, including managing the transition of/to some fairly radical and poorly planned systems conversions. Yet they couldn't find anything better for me than a cube where my kitty-/katy-corner neighbor spent a couple of hours a day on speakerphone, and the senior manager next to me spent a different hour of the afternoon chatting on the phone with his son -- a fabric wall is not effective when two people are sitting circa five feet apart.
I didn't even care what they were doing -- or, in the case of the family chats, also not doing. I just didn't want to fight the noise for my concentration.
In another position, they finally agreed to move me to an office. Which, despite better rooms being available (and turned down by my manager), they made a windowless box (fine -- well, ok, I can cope) located next to a conference room, with no sound insulation in between (not fine).
The irony in that was that about half of the team I was on at that point was located either at other offices or virtual. That aspect worked our really just fine. One of the fellows I grew to have a lot of respect for, over the course of a couple of years, I never met in person. Yet we could pick up the phone and call -- or just email or IM or whatever -- and immediately address and plan a course of action, or take action when immediately needed, with no problem. He was a seasoned developer who knew how to get stuff done.
So... Personalities and experiences differ. But, I wouldn't worry too much about the virtual part. From my perspective, it would be likely to make me more effective in doing the actual work. And I would welcome the separation from all the environmental crap. Perhaps also the ability of people to just stop by my cube or office and draw me into their political BS or social BS. There's good social. And there's wasting an hour debating the best Jim Carrey movie. I can do without the latter.
Actually, I'm dealing with a little environmental noise now, at the moment. So, I hope I remained coherent in this response.
TL;DR: Do the work, and unplug from the crap. Keep the network active, for if/when the crap nonetheless starts to exceed the work. Use virtual as a crap buffer/filter.
--
P.S. The guy I mentioned coming to respect? Well, I got to know more than a bit about his golf fetish. There is -- or can be -- quality socializing in a virtual environment. What's nice, though, is not to be forced to tune out your neighbors' two hour discussion of their weekends.