Hacker News Reader

Comment by abronte | Hacker News Reader

We have a pretty big e-commerce scraping project, and we don't really run into many problems regarding blacklisted ip's. There are a few sites who we get consistent bans from, but with elastic ip's it's pretty much a non-issue. I have yet to see a site that ban's all amazon ip's.

View on HN

I run http://whirlpool.net.au and I religiously check the Amazon EC2 forum announcements[1] for new IP ranges to ban.

[1] https://forums.aws.amazon.com/ann.jspa?annID=1030

> I run http://whirlpool.net.au

An excellent and very useful forum! It seems like whatever topic I'm searching for, google(.com.au) returns a useful result on your site.

Would you mind telling why?

Why?

Name me one good reason. Name me one.

Shitloads of rogue bots doing "social media monitoring".

Shitloads of rogue bots stealing content for black-hat SEO.

Shitloads of rogue bots harvesting email addresses.

Shitloads of rogue bots submitting spammy replies.

That's unfortunate. I don't know what the answer is, but real people do route their traffic out of AWS endpoints.

Far fewer than you might think.

So maintain a blacklist of elastic IPs. If it's too big for you, make it a community effort.

Those are bad reasons to close your site to all of AWS.

As nupark2 mentioned, there are legitimate users routing traffic through EC2, even some bots that you'd want to visit your site. Archive.org comes to mind (many of there scrapers are or were behind AWS). Closing your site or app to a large swath of the web is the wrong solution. It's like killing a spider with a bazooka.

Unlike the assumptions you're limited to making, I know how much of my AWS traffic is human, and it's really very very very small. The sad reality is I'm sick and tired of rogue bots, and the tiny sliver of collateral damage can fill out the CAPTCHA validation every so often.

(I also blacklist GWS, rackspace, linode, softlayer, reliablehosting, ovh.net, node4, netdirect, layer42, all TOR exits... it's actually a pretty huge list.)

I whitelist archive.org, and they've never hit through AWS.

It is annoying when it happens and you're on something like Heroku.

Reply on news.ycombinator.com