>sites that won't let you read their robots.txt file if they don't like your user-agent string
That's hilarious. So do you use borrow a browser's user-agent or do you ignore the robots.txt?
That's hilarious. So do you use borrow a browser's user-agent or do you ignore the robots.txt?
That would get them to fix the issue pretty quickly :-)
More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.