Ask HN: Is it possible to overwhelm a web-crawler with infinite links?
mysite.com/:id
shows some text, links to
mysite.com/:id+1
Would a crawler keep hitting these infinity of pages? What scale would it take to bring down a crawler?
shows some text, links to
mysite.com/:id+1
Would a crawler keep hitting these infinity of pages? What scale would it take to bring down a crawler?
Add to this the fact that a crawler is able to stop and back off if its resource constraints are being overwhelmed, but as a webserver that may be serving content that is "mission-critical", for lack of a better phrase, your server probably needs to maintain a low latency. In addition, there are multiple web crawlers out there, that could hit your server at the same time.
Given that, I think the scheme you've described is a mug's game, where for every dollar's worth of damage you do to a crawler, you're doing more than a dollar of damage to yourself.