Hacker News Reader

Comment by jeroenhd | Hacker News Reader

Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.

It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.

View on HN

> Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use

That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.

And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.

Sure, there are loads of easy solutions to this problem. Putting everything behind Cloudflare is the most common one, Anubis and its competitors are another. I could also block all data centers, countries with loads of CGNAT + botnets like Brazil, China, and probably India, and refuse any user agent not in the top 5 modern user agents with a little TLS fingerprinting to make sure.

I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.

The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.

Reply on news.ycombinator.com