Hacker News Reader

You can use the AOPIC algorithm that has a credit and taxation system to penalize these types of websites.

https://stackoverflow.com/questions/5834808/designing-a-web-...

I used to have what was basically a spider trap on my website while learning Django. The truth is that every bot that found it hammered my site with tens of thousands of requests per day (even Googlebot). I don't think it stopped until I blocked the page in robots.txt

A well-behaved crawler will be self-limiting in this regard in that they seek to avoid requesting URLs from the same domain too often, and then the better you are at spreading requests over as many domains at possible, the less you'll be affected by such traps.

You can do a lot of other things too, but the above already takes most of the sting out of such traps unless you use a huge number of domains.

Just have a maximum traversal depth