Hacker News Reader

Nice - I like that most of the ai scraper bot blocking was done using Nginx configuration. Still, once fail2ban was added to the mix (meaning: additional service and configuration), I wonder if considering something like Anubis (https://anubis.techaro.lol/) would have been more automatic. I’ve seen Anubis verification pages pop up more frequently around the web!

View on HN

FWIW, the reason I like their approach is that fail2ban is still lean, works off of the same logs, and doesn't start with the requirement to affect everyone's experience due to bad actors.

Making the visitor put up with your bot because you don't like someone else's is just bizarre to me. I disable js by default, so Ive been seeing this more and more. It's all rubbish these days.

Anubis author here. I have been working on a no-js approach. I just have to be careful so that the scrapers don't just implement it and bypass it forever.

go-away has one, it only works in browsers that follow meta refresh though.

Reply on news.ycombinator.com