Hacker News Reader

might as well do the full list:

https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...

cloudflare have a button for this:

https://blog.cloudflare.com/declaring-your-aindependence-blo...

Don't stop at robots.txt blocking. Look through your access logs, and you'll likely find a few IPs generating a huge amount of traffic. Look them up via "whois," then block the entire IP range if it seems like a bot host. There's no reason for cloud providers to browse my personal site, so if they host crawlers, they get blocked.

Thank you very much for mentioning these! These parasites deserve to burn in hell for their greed and violation of consent

Reply on news.ycombinator.com