You can get your site taken down, if that's your worry. But like generally on the internet, your best recourse is to control your online presence at the source. Don't post regrettable things publicly, and you should be fine. If other people do it to you, your issue really would be with them; fortunately, if something terrible is done, you've the Archive to prove the damage and back you up.
It really is a different beast from your normal panopticon internet tracking. This is more akin to showing up in the newspaper, rather than a surveillance camera.
[0] https://archive.org/about/faqs.php#29