IA should start respecting robots.txt disallow, because right now they ignore it.