Hacker News Reader

20 requests per second per 7 servers so it comes out to 3 requests per second. Not even close to a DoS attack.

It's 20 rps -- not sure why you divided by 7. A DoS attack is measured by total, not per server.

Well if you are going to total might as well add in the traffic from all the other users. And in that grand scheme 20 requests per second won't matter.

I only divided by 7 because I thought he wasn't running all the 7 servers simultaneously, come to think of it, I was wrong.

I can't believe a company as large as okcupid has nothing in place to detect and stop this.

It's not fast enough to be a DOS attack by any stretch. But it's easy to detect that a given IP address has downloaded e.g. several hundred profiles in a few minutes, and OKC should have been throttling such obvious crawlers to around one per minute. If the behavior repeated, they should have blocked the address altogether for some period of time.

At the point where you are negatively affecting their servers availability and doing so with a malicious intent.