What does honouring someone who is extremely pro-slavery even for their time have to do with preserving history? That history still exists whether they have a building named after them or not. And I don't think you will honestly find many people who believe that the ends justify the means no matter what, although you will of course find some people who believe anything. So I think the "ideal world" bit is also somewhat a strawman and a mischaracterisation of people who are anti-racist or whatever.
I do however agree that a lot of this stuff doesn't really help those who need it most. The rich kids paying a 2% wealth tax funnelled into social welfare and education for the working class would do more, as would having Yale include more working and lower-middle class student. But it's easy for me to say as a white guy that having many buildings named after pro-slavery politicians is not really doing any harm. Maybe we should ask those who would be more effected.