> What is the alternative for someone who wants to build a house for a large family while spending 6 (or low 7) figures

Don't buy in a "prestigious" zip code, build with a simple design. My parents live in Great Falls, an area that's an hour commute into D.C. and full of McMansions that run $1.5 million for cheaply-built crap. If you buy some land an hour in the other direction from D.C. (toward rural Maryland), you can build a tasteful 5,000 square foot house for $1 million including land. People by and large don't do it, because they might meet a Trump voter and thereby have a stroke.