I don't fully understand why it's so hard for a legislative body to eliminate such covenants.
People generally like setbacks. It prevents their neighbors from being too close, or the street getting crowded in. Property owners might want to develop along the edges, but their neighbors may not want them to.
So clearing these covenants out wholesale is going to be a fight. And going through the legislative process for a single property is really inefficient.
So what you're saying is that others in the vicinity would be able sue to make sure the covenant was honored? I have to assume this has been tested in court many times before and upheld.
If none of the neighbors care, theoretically you'd be fine to ignore since no one else has standing to sue, but that doesn't preclude the possibility that a neighbor would sell it someday to someone who wants to shake you down. If everyone's fine with it, you really should get them to give up their interests in the covenants.
See https://en.wikipedia.org/wiki/Shelley_v._Kraemer where the United States Supreme Court ruled the 14th amendment made "racially restrictive covenants" unenforceable.
Also, depending on the state, you might be able to get a covenant removed by a court if it has been violated for a sufficient period of time.
So maybe a municipal action could do something similar?
Of course the easiest way to not deal with covenants like this is don't buy property with them. Condos have tons and I will never live in one.