It really wasn't. We would like it to have been, but at the risk of stating the obvious, it's not an inalienable, fundamental right if someone can take it from you without fear of repercussion. Slavery was legal -- the scare quotes add nothing. The difference between legal and "legal" here is just personal ethics.
Now, we can of course decide to proceed today as though the laws were never legitimate, and offer reparations, for example. Or in the case of the Nazis, as you mention, to try them for crimes committed in the past. It's worth noting though that the Nuremberg trials were very controversial for exactly that reason -- many people sympathetic with the desire to punish the Nazis were extremely uncomfortable with the idea of trying people for violating international laws that didn't exist at the time of the offenses (or only existed in treaties the Germans never signed). We did it anyway because, well, they deserved it, but it's far from the truth to say that it was a routine application of some tradition in which there are some laws that are so fundamental they need not be stated anywhere. And as a matter of constitutional law, we strictly prohibit doing that in our own legal systems.