Hacker News Reader

Amazon's scraper has been sending multiple requests per second to my servers for 6+ weeks, and every request has been returned 429.

Amazon's scraper doesn't back off. Meta, google, most of the others with identifiable user agents back off, Amazon doesn't.

View on HN

If it's easy, sleep 30 before returning 429. Or tcpdrop the connections and don't even send a response or a tcp reset.

That's a good way to self-DOS

That's why I said, if it's easy. On some server stacks it's no big deal to have a connection open for an extra 30 seconds; others, you need to be done with requests asap, even abuse.

tcpdrop shouldn't self DOS though, it's using less resources. Even if other end does a retry, it will do it after a timeout; in the meantime, the other end has a socket state and you don't, that's a win.

Reply on news.ycombinator.com