(As an Australian - we completely fall on the American side of that remark)
Probably partly from ownership of buildings transitioning form wealthy capitalists to middle class. For the capitalist the house is an investment since they get rent from it, but for the middle class family who own their house the house constitutes a huge portion of their wealth. Banks are the main winners, as they profit from loaning money to middle class to purchase the house.
The second factor is the rise of cities. As more people come closer together, the value of land naturally rises. Hence, the "natural" portion of the houses price does not come from the structure itself, but from the value of the land, and the market forces demand for the building.
If the building becomes... defunct, a huge portion of the middle class familys wealth disappears (sans the land - but, often the land is rented from someone).
Hence it's nice to imagine buildings last forever, while paying back your 50 year loan on the house.
Wooden houses can (from experience) survive magnitude 6-7 earthquakes.
Japanese pagodas have very interesting designs that keep them from toppling. My favorite is the Horyu-ji pagoda. Each floor is not connected to the one above it - they're just stacked, loosely. The floors shift independently during an earthquake. A gigantic central beam in the middle acts as a tuned mass damper, preventing them from sliding off entirely. One person can make the central beam sway. https://gizmodo.com/5846501/how-japans-oldest-wooden-buildin...
Tō-ji pagoda's design: https://www.youtube.com/watch?v=uG37gQSvrf4
Design really is everything.