Hacker News Reader

I see this going the way of email, with larger, well-known, and more well-behaved crawlers being allowed to index websites for free, and smaller, unknown crawlers suffering brownouts, getting banned, or having to pay for access. It will be harder to self-host a website or run your own crawler.

View on HN

Yes this does remind me of the old spam wars of early 2000s. Back then collaborative block lists were useful to reject senders at IP level before using a Bayesian system on the message itself.

Even though these bots are using different IPs with each request, that IP may be reused for a different website, and donating those IPs to a central system could help identify entire subnets to block.

Another trick was “tar-pitting” suspect senders (browser agent for example) to slow their message down and delay their process.

Reply on news.ycombinator.com