In many US cities, low lying areas near water (rivers, large lakes, oceans) were often used as cargo ports. Factories and other industrial facilities were then built nearby which made it easier to obtain raw materials and ship out finished products. And naturally low wage, blue collar workers tended to live near those jobs. Such areas were also at greater risk of flooding, and suffered more from air, water, and noise pollution. So, affluent people tended to build homes on higher ground away from the dirty waterfronts.

Now that trend has somewhat flipped. The US has deindustrialized and offshored most heavy industries. Stricter environmental laws have reduced pollution. And civil engineering measures have somewhat reduced flood risk. So, now waterfront residential areas have become expensive and fashionable.

This is just a general trend. We will still see exceptions in SF and other cities.

In Brazil, the favelas are up on the hills. Unclear why it's this way in some places, but the opposite in others