There are lots of reasons this will not work:
In several organizations, there are a range of middle management layers who only do communications and lights on work. These people will be laid off in as little as a year when their positions start becoming obsolete. Really once the brokers get separated from party the culling will be ruthless. To me the biggest opposition will come from middle management layers. The obvious reason cited will be 'collaboration','productivity' etc, but the real reason will be saving their own jobs.
There is also an added problem of work being easily shipped out to foreign lands. Outsourcing is likely to increase and that will invite more backlash. New hire onboarding is one more area of problem, mostly because of the absence of face to face initial assistance in getting onboard. This is true for big companies given the range of in house systems and tech some one needs to be aware of before starting to contribute.
Lastly, I feel even at individual levels this will effect us all bad. Most programmers are bad at exercise and sitting at home will only amplify unhealthy lifestyles and eating habits. I expect such people will get worse off over time, obesity and diseases will increase. Absence of peer pressure/influence on skill gaining is one area I suspect will suffer in the absence of an office.