The difficulty here, as in many cases, comes down to uneven enforcement and unclear lines. Consider the typical example of a consistently noisy neighbor and filing a noise complaint - in many/most cases, often nothing gets done.

Not necessarily so. Lots of expensive developments have HOAs so it's not about making life difficult for people of little means as they cannot afford property in these places -or even rent in these places. Yet, the neighborhood want to have some governance over the development. I prefer less oversight, but some oversight is appreciated to keep things from becoming unmanageable. On the other hand trailer parks, where many poor live, seem to not care much about appearance.

Or weirdos who like art and vegetable gardens.