That doesn't work when it's software on your client's computer. Remember when all the web devs revolted and ditched IE6 back in 2001, because MS was just taking the Mickey? Yeah, me neither.
(Current web crawler problem: sites that won't let you read their robots.txt file if they don't like your user-agent string.)
That's hilarious. So do you use borrow a browser's user-agent or do you ignore the robots.txt?
That would get them to fix the issue pretty quickly :-)
More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.