Show HN: Fetchcheck – send a signed crawler to any site and see if it gets in

If you were ever curious whether a site allows signed crawlers, you can use this tool. This tool respects the website's robots.txt directive.

You will be surprised how many sites allow all robots, but then block crawlers