I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?