The main problem is transportation. You can make every box the size of a standard shipping container, and thus greatly reduce your home design options and size of rooms while still spending plenty on shipping. Or you can make bigger boxes, which gives you more design flexibility but still a lot less than stick building does. And with the bigger boxes you need special highway permits, escort vehicles, etc.
And then, when you have everything on site, there's still a lot to be done. Tree clearing, grading, excavation, concrete work, paving, utility lines, on-site assembly of the boxes (using an expensive crane)...
When you've added it all up, modular isn't the obvious improvement that it seems to be at first glance.