Quadcopters build a 6 meter tower

Not to sound entitled already, but why are they so slow? They're computers and robots already, I'm wondering why they move to where they need to go and pause for a second like that.

you can get them now - see diydrones

For the foundations, you're probably right. For the rest, some fundamentally new techniques may be employed. It is for instance possible to to extrude concrete like you would plastic in a little 3D printer. Except your concrete extruder would typically be placed at the tip of a regular articulated robotic arm instead of a giant replica of a RepRap. (Edit: the technical term for this is "contour crafting".)

Note however that what currently requires most human intervention isn't the heavy duty. It is all the small work you have to do after: paint, floor, electricity, water, basic appliances. To automate all that, you need to replace not only the human mind, you have to replace the human hand.

Also replacing the human mind is obvious, though not necessarily easy: as you suggest, keep the machine, automate it, throw the human operator out.

The human hand, on the other hand, will take some insight to be replaced: how will you lay out all the tubes and the wires ? You cannot really expect human-sized machines do the same work the humans currently do, in the same way. The best solution is probably to to come up with clever ways to render the human hand unnecessary. You could for instance blend heavy duty work with the rest, (laying out cable as you put the bricks together, or something), or somehow changing how things are done.

But it looks like it could work.

Heavy lifting on construction sites involves quite a lot of unknowns and is safety critical. Some excavation could possibly be automated on probably on greenfield sites. Depends very much where you are building. In the middle of pretty much any European city for example, building the foundations almost certainly couldn't be automated much more than it is all ready. This is because you never know what you will find when you are digging. Example, how would you do auto plague pit detection? There is a fair amount of automation on sites already, piling rigs are semi-automatic and use relative GPS for setting out and various monitoring systems to control the speed of drilling and inflow of bentonite etc. Where automation will work best is when you have an easily defined problem. I don't know why we still bolt steel framed structures together, this is a problem that could be relatively easily automated with robots to lift, hold and weld up the pieces.