My sentiments are similar to OP's, and for me a large part of my job satisfaction was getting to hang out with really incredible people all day, getting lunch with those people, getting afterwork drinks with them, etc. Hanging out on video chat just isn't the same.
Of course, an argument can (and maybe should) be made that it's smarter to to keep your work focused on the work, and find your social outlets elsewhere. However, the unfortunate reality is that for those of us far removed from school and our home towns, work is where we tend to make our friends.
On top of that, I just enjoyed the change of scenery in leaving my apartment in the burbs to go into a nice office space in the city. Feeling restricted to the same space for 24 hours a day is driving an unhealthy level of cabin fever and boredom. For the first time in my life, I envy those with the big house and a yard.
Of course that later point would be somewhat mitigated in a world where I could work out of shared space or the local coffee shop, so I'm not sure if that's an indictment of remote work in general for my purposes.