Hacker News Reader

Hacker News

Comment by ck2 | Hacker News Reader

Full thread

ck2

More ip ranges to block - are they published yet?

Don't see them here or the subforum, yet

https://forums.aws.amazon.com/ann.jspa?annID=1701

View on HN

Two important things:

1 - The forum post has been updated and now includes a "/16" IP address range for the new Region.

2 - Please consider taking the time to report suspected AWS abuse using the form at http://portal.aws.amazon.com/gp/aws/html-forms-controller/co... .

Why do you block AWS?

Maybe they want to control indiscriminate acquisition of infrastructure across many different departments. You'd be surprised how many CFOs/CIOs don't know there is an invisible budget item somewhere in every small department which if added up would be a big item for the whole company.

They have unlimited free inbound.

Some people there will destroy your servers with insane spidering rates.

iptables conntrack helps but there are just too many and eventually the firewall takes more resources than the rest of the services you are providing

Are you hosting stuff that's particularly prone to crawling (and by crawlers that don't respect robots.txt)? Of the spider traffic we see, the vast majority of it comes from Google and the other major search engines.

One example: there are several people who apparently scrape the front page of HN (and proggit, etc.) and then proceed to download all of those links repeatedly every minute (or second!) for several hours. Same link, over and over and over. I can only imagine what get rich quick scheme would require such behavior.

Woah, that suddenly explains why sometimes websites go down so quickly after they get linked on reddit. Surely most hosting won't be able to host 100's of requests, but some times I've seen it happen that websites linked from smaller subs went down quickly.

I do crawling from EC2, and yes, I would not like a 1Gbps traffic spike myself.

Do you deal with generic webpage crawlers that way, or targeted API abuse? Because the first ones can be smoothly shaved away with the help of Cloudflare, for instance.

Did you look at the iptables hashlimit match? It can be used to rate limit per source (inbound) or destination (outbound), without needing conntrack.

Reply on news.ycombinator.com