Hacker News Reader

a) There are too damn many of them.

b) They have a complete lack of respect for robots.txt

I'm starting to think that aggressive scrapers are part of an ongoing business tactic against the decentralized web. Gmail makes self hosted mail servers jump through arduous and poorly documented hoops, and now self hosted services are being DDOSed by hordes of scrapers…

Do people truly dislike an organic DDoS?

So much real human traffic that it brings their site down?

I mean yes it's a problem, but it's a good problem.

If my website got hugged to death, I would be very happy. If my website got scraped to hell and back by people putting it into the plagiarism machine so that it can regurgitate my content without giving me any attribution, I would be very displeased

Yet HN does it when linking to poorly optimized sites. I doubt people running forges would complain about AI scrapers if their sites were optimized for serving the static content that is being requested.