Hacker News Reader

Comment by bwb | Hacker News Reader

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter.

I wish there were a better way to solve this.

View on HN

put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.

Got a good link to something on github that does this?

I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.

I did it manually and got fail2ban to read the access log anyway.

The it's the permanent iptables rule, but could be CF API call as well.

here's one example using rack attack on a rails app:

https://github.com/pinballmap/pbm/blob/302ac638850711878ac61...

but it only bans for 3 hours. if they don't respect the hidden link and robots.txt they get banned.

If you're not updating the publicly accessible part of the database open, try to see if you can put some cache strategy up and let cloudflare take the hit.

Yep, all but one page type is heavily cached at multiple levels. We are working to get the rest and improve it further... just annoying as they don't even respect limits..

At this point I'd take a thermostat that can read when my dashboard starts getting heated (always the same culprits causing these same server spikes) and flicks attack mode on for cloudflare.... it's so ridiculous trying to run anything that's not a wordpress these days

ah, you're the one who stopped me from being jerred@shepherd.com!

hah eh?

Reply on news.ycombinator.com