Hacker News Reader

In line 1 of of the linked page: "waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives".

Does that indicate the robots.txt is how "no crawl" is indicated? robots.txt doesn't have "no crawl", it has allow and disallow.