Not necessarily, you just might not build 100% of your eventual home immediately.
It used to be (i.e. prior to the postwar expansion of credit to the American consumer) pretty common to build a "starter home" that was intended to be expanded. You start with the basics (basement, kitchen, bathroom, bedroom) and expand modularly from there as your prosperity and needs allow.
In the midwest, they would start by building the foundation, basement, and a very modest first floor. As the family prospered, had children, and developed roots in the neighborhood, they'd expand the above-ground portion of the home in the summer months. You can find homes frozen at each stage of development from the pre-depression 1900s still around today, although they're of greater archaeological value today than housing stock value. A lot of present-day "colonial" housing stock in the Northeastern U.S. have evidence of this gradual accrual over the decades.
This idea of "build a whole mansion at once" was historically reserved for the truly wealthy, and has only been "practical" for a greater segment of society for the last 50 years (again, with the expansion of consumer credit, not because we can actually afford these things).
You could still choose to build things this older style. Our culture of "obtain everything all at once, immediately up to the limit of your purchasing power" is the main impediment. That, and building codes might frown on some earlier-phase iterations, though YMMV.