So the prefab parts are just going to put themselves together in the factories? Then walk themselves to the job site? And erect and inspect themselves?
This is "blue sky" thinking - but what if, just imagine - the bricks were each a robot?
That is - your house is a robot you live in. Maybe it could even reconfigure itself as needed. Complete science fiction at the moment, and may never be practical - but who knows?
I get what you're saying though - there'll always be some kind of jobs for humans - but they might be the low-hanging, anyone can do them type of jobs...