Hacker News Reader

Comment by gigel82 | Hacker News Reader

I'll look into implementing this, nice tip :)

archive.org is the worst offender for me; not only do they ignore robots.txt, there is absolutely no way to get something removed once they archived it (despite the data including accidentally leaked PII for example - which can cause actual harm to someone).

View on HN

I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.

Sometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.

I hope you never accidentally publish private information anywhere on the web.

Interesting, I've heard of the exact opposite issue with archive.org. That they take down archived pages at the request of the domain owner, even if registration lapsed and was picked up by somebody else since the snapshot was taken.

Are you mixing up archive.org (the Internet Archive, a nonprofit company headquartered in San Francisco) with archive.is/archive.ph/archive.today, which ignores robots.txt, and is a for-profit company with unknown owners and an unknown location?

No, I'm talking about archive.org, specifically their "wayback machine".

I had to have content removed from the wayback machine and it was pretty simple and painless all said. Perhaps it's gotten better over time?

Just send them a DMCA request, that's their takedown mechanism. Is it a good one? No, but that's how they do it. You see it posted about all over in their forums.

ex: https://archive.org/post/1022869/site-removal-request

They ignored all emails sent to info@archive.org from the actual domain in question which I owned, with a link to a URL on the domain asking them to remove it. Don't know what the DMCA process is, I presume it involves lawyers and such.

which is the point of archive.org - if there were human discretion it would turn into modern day Twitter

Reply on news.ycombinator.com