Are you sure this is how he'll be remembered? Half the US thought him preferable to AOC and Hillary Clinton. It's hard to conclude in any other way than that the perception of his legacy will be equally divided.
But demolishing a third of the White House? That'll be clearly visible in every single aerial shot of the building during every single political event for years. It is, quite literally, a scar on the political face of the country.
It's like turning the Pentagon into a Square, or blowing Washington's face off Mount Rushmore, or selling Alaska back to Russia: you're not going to forget when you are constantly being reminded of it.
What do those people have to do with anything besides being popular right-wing targets?
His approval rating is currently around 42%.
https://en.wikipedia.org/wiki/United_States_presidential_app...
This, of course, grinds the incumbent's gears more than anything else.