The argument is being able to look at some court house built in 1925 is not more important than people in 2018 (and forever into the future) having more affordable places to live.
I tend to agree that the "no historical preservation" viewpoint is way overblown. E.g., there are lots of monuments and buildings in Boston and Philly that should always be preserved because they are integral to telling the nation's founding story. But there are lot of "historical landmarks/buildings" in the USA that are really only significant to the handful of people who pushed to make them "historical" (and are probably now dead anyways).