it's not my fault, robots.txt made me do it!