As a sysadmin occasionally responsible for resolving load-spikes caused by bots/crawlers: well said!