"I don't think it's even relevant that it was not legal at the time".
Imagine setting the precedent that the government can unilaterally decide to steal your property based on something your great grandfather did which wasn't even a crime. Do you not see how such a power might be abused in future when you can retroactively punish people for laws where the ink has yet to dry and that they never even committed?