Just because you put something in robots.txt doesn't mean they will adhere to it.