If that's the case you could buy up defunct domains, exclude everything via robots.txt and selectively purge sites from archive.org
That seems like a pretty glaring flaw in something designed to create an enduring record.
That seems like a pretty glaring flaw in something designed to create an enduring record.
from the faq: http://netarkivet.dk/in-english/faq/#anchor8
8. Do you respect robots.txt? No, we do not. When we collect the Danish part of the internet we ignore the so-called robots.txt directives. Studies from 2003-2004 showed that many of the truly important web sites (e.g. news media, political parties) had very stringent robots.txt directives. If we follow these directives, very little or nothing at all will be archived from those websites. Therefore, ignoring robots.txt is explicitly mentioned in the commentary to the law as being necessary in order to collect all relevant material
I wonder if there are any other national archives of the internet that do the same.
https://www.bl.uk/collection-guides/uk-web-archive describe the much more limited aproach taken by the british library much later in time, but might extend to a similar scope.
I think if I were running a national or internationally mandated archiving initiative I would basically want to take in content from Internet Archive, and not remove things, and probably it would be less expensive that way than having my own crawler.
The key is it works both ways. By respecting the live robots.txt, and only the live one, data hiding must be an active process requested on an ongoing basis by a live entity. As soon as the entity goes defunct, any previously scraped data is automatically republished. Thus archive.org is protected from lawsuit by any extant organisaton, yet in the long run still archives everything it reasonably can.
[0]https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
> We are now looking to do this more broadly.
That's the part I'm asking about.
[1]: Sure, some people just want to hide embarrassing or incriminating content, but there’s also cases where someone is being stalked or harassed based on things they shared online, and hiding those things from Archive users may mitigate that.
I don't think it's mentioned in an official document, but it's usually referred to as "darking".
It probably safe to assume that the same concept applies to the Wayback Machine as to the rest of IA.
Edit: Here's a page that indirectly conveys some information about it: https://archive.org/details/IA_books_QA_codes
The solution (which the Internet Archive really needs to implement) is to look at the domain registration data or something, and then only remove content if the same owner updated the robots.txt file. If not, then just disallow archiving any new content, since the new domain owner usually has no right to decide what happens to the old site content.