Hacker News Reader

Are you sure you're not just encouraging more people to run them?

Even so, traffic funnels through the same cache, so website owners would see the same amount of hits whether there was 1 crawler or 1000 crawlers on the platform

So you're forcing me and OP to sign-up for the protection racket.... Thanks

Reply on news.ycombinator.com