Comment by yetanotherjosh | Hacker News Reader

Comment by yetanotherjosh | Hacker News Reader

I've worked at places that have tried this trick. It doesn't work - it's always been removed because of real users complaining they've lost access.

Several scenarios can trigger it, and probably more. The internet is a weird place. Consider:

1. Some clients, browser plugins, and proxy servers implement link prefetching. These agents will not care that the link is attached to a 1px gif that the user won't see. This is not really breaking the rules, either, it is quite permissible and in the scope of HTTP implementations - unless you've put your black hole behind a form POST (which bots won't for anyway.)

2. Internet explorer among other tools allows users to download content for offline viewing. The client does not respect robots.txt when such fetching has been initiated by a user.

3. Not all users browse the web visually, and your 1px gif is discriminating against the visually impaired. When browsed with a screen reader, a linked image is a linked image is a linked image.

Additionally, outright blacklisting by IP address as noted by others on this thread is highly problematic, especially when the behavior that triggers it could accidentally come from real users behind a NAT firewall (at a typical office, library, etc). A single user performing any of the above behaviors would block the entire group from the service.

There are better ways to fight misbehaving robots that do not so easily trigger false positives...

I would suggest burying the "bad link" several pages deep under the blackhole. (i.e. get a page, with a link, which leads to another page that has another link, etc, etc.)

The spider will crawl all the way down, but real users and prefetch won't. (I'm not sure about offline, but I can't imagine they go down very far.)

And in case the user does end up triggering it, perhaps also replace the "You have been banned" page with a captcha challenge to get yourself removed from the blacklist.

I've worked at two places that did this.

One place was run with an iron fist by the meanest physicist on the planet, a guy who eats nails for breakfast and always asks the toughest question at the seminar. His attitude was "delete 'em all and let got sort 'em out"

The other place was my own personal network of websites, where I was trying to keep spam-harvesting robots from getting my email address. I had a process that had several steps to protect the page that had my address on it, one of which was a 'tripwire' which would instantly block any stupid robot that followed one of the hidden link.

I can say that I've gone for years without getting any email complains about difficulties reading my email address ;-)

At least not by email?

You're missing the point - there is no 1px gif, or hidden link. The only place that you can find the link is in the robots.txt file, and it's set to Disallow:

http://perishablepress.com/robots.txt

So if you're a "bad bot", you'll go to /blackhole/ looking for juicy stuff, and get banned. If you're Google you won't follow it (or he has a whitelist) and if you're an end user you won't even see it.

Edit: Ah, my bad. He does put a link to it in his footer.

Reply on news.ycombinator.com