I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws/all/), and have just been made aware that it appears to have spread across the internet:
https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fi...
Favourite sites I've found with it so far: php.net, princeton.edu, nest.com, songkick.com.
THIS is evil. You could use this argument for banning any new search engine.
This is the reason they're blocked, not because they're new or non-English.
Not tested. :)
This site coolshell.cn is boycotting baidu, so it tells Baiduspider that it doesn't want to be indexed by baidu.
Baidu respects this and doesn't index anything from this website.
http://help.yandex.com/webmaster/controlling-robot/robots-tx...
Yandex's online tool to check specific URL on allowness in robots.txt: