Hacker News Reader

Comment by santah | Hacker News Reader

Update 2: After banning close to 2k individual IPs, it looks like I got it under control, for now.

I wonder how banning so many IPs affects CloudFlare performance and if I should optimize it to block whole IP ranges instead ...

View on HN

Update 3: After about a day - they expanded their pool of IPs to more than 2k new ones.

I blocked what I could and then I blocked the whole country (Russia) behind a CloudFlare javasript challenge (so that any legit traffic that wanted to pass it - could).

Everything stayed blocked for about a day and now it seems they gave up:

- all domains that pulled dynamic content from my site now show some other content and do not try to scrape Next Episode

- all static domains that delivered the cached version of the Next Episode homepage now deliver some other website(s)

- CloudFlare shows no further traffic on the firewall rules I've set (so the bots from those IPs are, for now, gone)

As this post is relatively old now (more than 3 days) I doubt many people will go back and check for new developments, but I wanted to give this the proper closure and let you know of the apparent happy end.

Hey, I read it! Thanks for the update! Now I see why some people block countries or IP ranges in their sites.

Are the IPs from the same ARN block? Cloudflare should show you that in the Firewall events page. If they're using IPs from a single VPN/Server company then you might be able to just block the ASN.

You might also be able to find common user agent headers through the CF firewall page and block them based on their UA. That'd not work if their scraper tool was randomizing the UA string but quite a few of them don't

Including all IPs I blocked today, they're spread between 5 different ASNs. I may resort to blocking them eventually, but for now - individually blocking the IPs (even in the thousands as it is) - seems to be working well enough.

As for user agent - they're using a very common, real browser user agent that's impossible to distinct from legit users.

Do you have many users in Russia? You could block the whole country ;-)

Russia has no GDPR or something. So you could put (special key in) a cookie? They probably do not process it so subsequent requests without a cookie are to be discarded?

> Russia has no GDPR or something. So you could put (special key in) a cookie?

It is entirely permissible under the GDPR to use cookies for security purposes.

If you can look closer at the HTTP requests, it may still be distinct. For example, the header order may not match any legitimate browser, or some other header doesn't make sense in the context of that UA.

Reply on news.ycombinator.com