Of course that doesn't do much good for the company's employees that don't wind up volunteering / doing the work, perhaps even increasing the separation the employees may feel toward the less-privileged. Service projects, mission trips, etc. can do a lot toward increasing humility, gratitude, and a spirit of giving in a more-privileged person. One could argue this is partly why companies want to do this sort of thing, so their employees grow (among other good / less-than-good reasons).
Perhaps a company could split the difference. Hire others to do the manual labor that day, but take half the company's team to the jobsite do the work with them. Then on another day you do the same with the other half. That way, all parties involved receive some benefit, and everybody is getting to meet a human being behind the statistics. Just a thought.
P.S. It's a certain level of irony when a party steps in to "help," when in reality they're taking away opportunity from parties more prepared for that sort of work, and are probably doing a worse job, aka building shoddily-constructed houses for the poor. This is especially a problem in developing countries, when an organization or church flies a bunch of inexperienced people into the country, brings in supplies, etc., and virtually takes over the available construction projects in an area because they're doing it for free. All of a sudden the local tradespeople have their businesses dry up, and they may not be able to recover -- and they're the ones that the area needs to continue the work afterwards! A far better method, in my opinion, is to get the locals involved and ask how you can help. Volunteerism is fantastic, but we should do it wisely. </soapbox>