It's not just the cost of building the house. Utilities and infrastructure (and the taxes to pay for it all) add up. Now, if you were to build smaller, densely, and close enough to amenities that residents could walk or bike to their daily activities, or to public transit that can take you farther away, you save a significant amount on initial AND recurring costs. And what's better, people WANT that! Or, a least, they're willing to eat the downsides in order to benefit from the upsides, including lower costs. But there lies the rub: when these types of places get built, they get offered at market pricing, not in a way that reflects their lower costs. So people say, "Why spend the same for less?" and move to a traditional suburb (or, more likely, put off home-buying altogether).
The problem with housing in America always comes down to the way it was financialized and securitized: too much relies on "line go up, forever". There's no room for new blood/capacity (read:competition), there's no room for "investments" to lose value.