>Getting rid of it would bring housing costs down.
You answered your own question. Housing costs coming down is a good thing for people who need housing, but it's not really a good thing for people who own valuable homes, and people who own valuable homes tend to have much more political power.