For me the interesting part is that the crawler is going bezerk. Never ever should a single crawler be the cause of 80% of traffic.

Also why should they not respect the 403? Crawlers just go to anything they can find. It is not a targeted attack.