The problem is that a house is seen as in investment vehicle. Buy a house, its value will always grow, they said. So people who bought houses expect their values to grow, and loathe their values to go down even a little bit.
Now the value (aka cost) of a house has grown so much that it's unaffordable for large swaths of regular working population, even well-paid. Cheaper housing is badly needed.
But cheaper housing also means that "value" of existing houses will go down! Either from being next to a cheaper house, or from a general downturn on the housing market (aka housing becoming more affordable). Every house owner will hate that! And these house owners are voters.
Now life savings of an old lady from the working class depend on a young family from the working class being unable to afford a similar (or any) house. We're in a pretty bad situation here :(