Hacker News Reader

Comment by avodonosov | Hacker News Reader

I do a similar thing with web crawlers that do not respect the robots.txt

https://github.com/cl-test-grid/cl-test-grid/blob/873b2fa978...

I don't know if this snippet is really effective, can be improved a little, especially that I noticed a couple of new crawlers that ignore `User-agent: * Disallow: /path` in robots.txt, and do not fix that even after reported.

View on HN

I'll look into implementing this, nice tip :)

archive.org is the worst offender for me; not only do they ignore robots.txt, there is absolutely no way to get something removed once they archived it (despite the data including accidentally leaked PII for example - which can cause actual harm to someone).

Just send them a DMCA request, that's their takedown mechanism. Is it a good one? No, but that's how they do it. You see it posted about all over in their forums.

ex: https://archive.org/post/1022869/site-removal-request

They ignored all emails sent to info@archive.org from the actual domain in question which I owned, with a link to a URL on the domain asking them to remove it. Don't know what the DMCA process is, I presume it involves lawyers and such.

I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.

Sometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.

I hope you never accidentally publish private information anywhere on the web.

Interesting, I've heard of the exact opposite issue with archive.org. That they take down archived pages at the request of the domain owner, even if registration lapsed and was picked up by somebody else since the snapshot was taken.

Are you mixing up archive.org (the Internet Archive, a nonprofit company headquartered in San Francisco) with archive.is/archive.ph/archive.today, which ignores robots.txt, and is a for-profit company with unknown owners and an unknown location?

No, I'm talking about archive.org, specifically their "wayback machine".

I had to have content removed from the wayback machine and it was pretty simple and painless all said. Perhaps it's gotten better over time?

which is the point of archive.org - if there were human discretion it would turn into modern day Twitter

I wonder if you could abuse gzip compression on responses to send a zip bomb back to them.

You can, but most bots do timeout. If you get a lot of bad bots that are vulnerable, then you'll probably waste a lot of resources on those connections.

Yes you can. There ar also ways to protect the crawler server from crashing.

I did just that for a while with a spare server I had. I set it up to literally only respond to bad bots. I know the crawlers don't care but it amused me at least. I tried to also keep redirecting slowly before it could time out. There was one bot that seemed to create a new instance each redirect so I could keep it in a loop for essentially ever. Just about every other bot only followed a few redirects before giving up. Fun times.

That's pretty neat! I should set up something similar for my domains that keep being spammed.

Reply on news.ycombinator.com