Hacker News Reader

I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it.

Never assume malice what can be attributed to incompetence.

Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.

Its likely just a shitty attempt to rate limit bots