> respecting robots.txt
What agent name should we put in robots.txt to deny your crawler without using a wildcard? I can't see that documented anywhere.
What agent name should we put in robots.txt to deny your crawler without using a wildcard? I can't see that documented anywhere.
User-agent: Crawlspace
Disallow: /
The main audience for the product seems to be AI companies, and some people just aren't interested in feeding that beast. Lots of sites block Common Crawl even though their bot is usually polite.
Yes, it should — I use the library below, and it should split at the slash character, treating it as a prefix match, per spec.
https://github.com/samclarke/robots-parser/blob/master/Robot...