If you crawl sites without a distinctive user-agent; do not obey robots.txt; crawl at an unnecessarily high rate, I _will_ ban you, regardless of your intent.
> Identification of well known legitimate bots;
> Hand written rules for simple bots that, however simple, get used day in, day out;
> Our Bot Activity Detector model that spots the behavior of bots based on past traffic and blocks them; and
> Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.