The fears are/were that the natural beauty of certain types of development would be spoiled by low-income housing, factories, and commerce. Of course, as you say, it's very any I - market. People already tend to naturally cluster based on income, level of education, desire for public goods, etc, so it shouldn't be necessary to have such restrictions.
It inevitably comes down to money, too, where minimum lot size restrictions shut out potential tenants/owners who can afford the per square foot cost in low quantities but not a large property. These restrictions are supposed to ensure a high enough tax yield per dwelling, but it's in the name of beautification.
You could say it all goes back to the car. When it suddenly became much cheaper for low income people to move out of the city center and commute from traditionally rich suburban areas, the rich got scared. Some moved further away, but governments also used zoning laws to make sure to poor stayed in the city, where they belonged.