If I remember correctly they changed their policy quite some time ago and started to ignore the robots.txt, but not 100% sure about it