I share this view, but I also recognize that as a white dude, what is to me a learning opportunity about the things that folks (who mostly look like me) have done in the past isn't necessarily seen that way by everyone.
Unfortunately, even if you get a whole mess of folks of different backgrounds to have an honest and meaningful conversation about what to do about these many things, you aren't going to please everybody. The easiest solution is to just remove the materials rather than having to continuously justify leaving them available and reinterpret the historical context for them from a contemporary standpoint.