That's looking at his life through a current day PC culture lens... he was the epitome of class and cool in his day, and definitely did his share of fighting for civil rights. He probably could have aged out more gracefully, but he was a product of his times and rich... meh. Even in his older age, he lived perhaps a bit sleazy, but in a "like you wouldn't do the same thing with that kind of money and status" sort of way, if we're all being entirely honest with ourselves.