Hacker News Reader

Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.

More precisely described as "block non-sanctioned user agents".

Using that feature will ensure I never visit your site again.

It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.

> is a right

I believe they're saying that cloudflare will block them just for using a blacklisted client, even if they're legit users and not bots

Not even that; it's a whitelist, not a blacklist, and the only clients whitelisted are essentially those of Big Browser.

On this point, if you turn on bot fight mode it also says it blocks verified bots.

But, bot fight mode says "there is a newer version of this setting" however it does not link to it.

Anyone have any insight on the blocked verified bots or the supposed new version?

Reply on news.ycombinator.com