I have no moral opposition to contract work - I've been a contractor, and I've hired contractors.
There's a trade off.
Contractors cost more per unit time than do employees (in the short term), and are usually looking for the next gig. At a startup, we don't have tons of free cash, but we do have tons of potential. We'd rather reward someone with a promise of sharing in our success (which we have) than with twice as much cash (which we don't have).
As far as working on site versus off site: again, I have no moral objection to this, and I've done it myself. However, I'm not just hiring engineers to produce lines of code: I'm hiring them for their wide-ranging intelligence, their ability to contribute to the ideas bouncing around the office, and their ability to tell me when my ideas are full of shit. All of this takes place much more easily in an office. Finally, while most people who work remotely are professional about it, I have noticed that it becomes easier for someone to disengage, grow tired or frustrated, and mentally "check out"...and it's harder for me to detect this, and take appropriate action (a word of encouragement, an announcement that we're all bagging work for an hour to go get a burrito on the company's dime, or whatever else).
As much as I like the IDEA of remote work, and easily compartmentalized tasks, I have come to believe that humans have evolved to be highly social, and to work best when they are immersed in a social environment.
I note that Joel Spolsky has company lunches every day, and considers this important.