Do you have any factual corrections? Your post reminds me of those "I'm getting a kick out of these replies" copypasta--declaring someone wrong and claiming authoritative knowledge, but without actually correcting any of the errors of fact.
When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits.
1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to 10qps and watched the traffic follow it down. Even at 3qps you would get traffic at bit over 2 qps trying to limbo in under the limit.
2) At that time, lots of people who highjacked browsers with toolbars sold scraping as a service to third parties. Their toolbar would check in to see if it should do a query and it would launch a query and return the results without the user even knowing. One company, 80 legs, was pretty up front about their "service", SEO types would use it to scrape Google results to see how their SEO campaigns were doing.
3) The majority of the traffic had criminal intent, looking for metadata on web pages to indicate they were running an unpatched version of some store software or had sql injection bugs. These would often come from PCs that had been compromised for other purposes or "zombie" PCs. We could rapidly map out these networks when we got 100 queries from 100 different IPs looking for "joomla version x.y",p=1 through "joomla version x.y",p=100. We briefly played around with sending them official looking SERPs but all the links went to fbi.gov though an obfuscator.
One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting.
It really was a never ending game. We mass banned an entire Ukranian ISP because out of billions of queries not a single one was legitimate.
Mailinator would do a similar thing with their custom email server hardware. Since they didn't really use sockets in the traditional sense, they were happy to give slooow replies and never disconnect "bad" connections.
If you don't mind, can you elaborate on the 2nd point? Specifically, what do you mean by hijacking browsers with toolbars?
Lots of people did it, even Blekko (although we stopped after we figured out it was just a scam), and the way it worked is toolbar company X would approach your internet site and say "we can send you a lot of traffic, just use this toolbar, we'll even pay you every time one gets installed."
Anyway, the toolbar would hook itself into the address bar and search selection hook in the browser and redirect every search to where ever it was told to. The nefarious part is that these tool bars were often shipped to the company as binaries not source, so you didn't really know everything they were doing. Once we figured out they were scumbags we also found out that they did some really scumbaggish things.
We stopped using them but lots of people did and finally the browser makers re-wrote the browsers to make what they did either impossible or easy for the user to revert and those guys rolled up their shops and went on to become some other type of scam.
In this specific case, the people wanting to detect bots want to avoid having their signals burned, the scrapers don't want the defenders to know which signals they're able to cloak since it will spur new signals development. So what gets disclosed publicly is just the really simple stuff.
It's kind of sad. There's a huge pipeline for getting people up to speed on security engineering, since there's a lot of incentive for everyone in the ecosystem to share information as publicly as possible except for relatively short responsible disclosure windows.
In contrast, the only way to learn abuse engineering is to happen to work in an organization with an abuse problem (and live with the frustration of an abnormally long ramp up period), or to go black hat. And likewise it's quite hard for the good guys to actually learn from each other, since they're spread across so many companies and it's thus hard for them to exchange information on what works and what doesn't.