Note that archive.org stopped respecting robots.txt since 2017. [1]
In my experience, the site owner must email archive.org support to be excluded from its crawler and archiving.
[1]: https://boingboing.net/2017/04/22/internet-archive-to-ignore...
In my experience, the site owner must email archive.org support to be excluded from its crawler and archiving.
[1]: https://boingboing.net/2017/04/22/internet-archive-to-ignore...
No comments yet.