Hacker News Reader

Comment by seanmcdirmid | Hacker News Reader

I thought robots.txt was provided an opt out for all web crawlers, not just vendor specific ones? What would be the use case for not using "*" if you didn't want something to be crawled?

View on HN

Not exactly. It opt outs of respectful web crawlers by any name, or if you specify it, by some name in particular. Misbehaved ones requires a bit of more effort.

robots.txt has always had mechanisms for allowing or denying specific crawlers. Most of the AI labs that crawl the web support something like this now, here are a few relevant examples from the NY Times robots.txt file for example: https://www.nytimes.com/robots.txt

    User-agent: anthropic-ai
    Disallow: /

    User-agent: Applebot-Extended
    Disallow: /

    User-agent: FacebookBot
    Disallow: /

    User-agent: Google-Extended
    Disallow: /

    User-agent: GPTBot
    Disallow: /

If apple hired one billion employees to look at websites and manually write the data they find, they wound naturally not respect robots.txt.

It’s very reasonable to want Google and Bing to index your page for search, but not have your data collected for training models, i think. I’m not familiar with robots.txt to know if it has a whitelisting mechanism

I see a lot of Robots.txt files that use non-wildcards for Allow but what would be the use case for using a non-widlcard for Disallow?

Reply on news.ycombinator.com