Hacker News Reader

easy guess that length breaks some legacy stuff

but every robots.txt should have a auto-ban trap line

ie. crawl it and die

basically a script that puts the requesting IP into firewall

of course it's possible to abuse that so it has to be monitored

I thought about doing something like that, but then I realised: what if someone linked to the trap URL it from another site and a crawler followed that link to the trap?

You might end up penalising Googlebot or Bingbot.

If anyone knew what that trap URL did, and felt malicious, this could happen.

A crawler could easily avoid that by fetching the target domain's robots.txt before fetching the link target. However a website could also embed the honeypot link in an <img> tag and get the user banned when their browser attempts to load the image.

How do you discern a crawler agent and a human? Is it easily as the fact that they might cover something like 80%+ of the site in one visit fairly quickly?

Crawlers/archivers will be hitting your site much faster than a human user.

Reply on news.ycombinator.com