Hacker News Reader

Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.

View on HN

In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.

That is true, but if you have Anubis you presumably do intend to `Disallow` all bots.

Not necessarily. You might want to rate limit bots instead of disallowing them.

One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.

That makes sense, Xe, but you don't enable it at:

- https://techaro.lol/robots.txt

- https://anubis.techaro.lol/robots.txt

and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.

Reply on news.ycombinator.com