Do your crawlers respect robots.txt and have a fixed IP pool that can be blocked?