Ask YC: Dealing with DDoS | Hacker News Reader

Ask YC: Dealing with DDoS

18 pointsPStamatiou15 comments

How do you do it?

At least once a week my load averages will stay at 5 or higher for about 30 minutes at a time. It's mysql that's eating up all the cycles even though page caching is setup for frequently accessed pages. netstat shows many blocks of IPs, each trying to connect to port 80 many times. I think it's coming from a zombie computer network.. one set of IPs included a Purdue connection.

My CS friends have told me about things ranging from iptables and manually adding offenders to my firewall - any automatic solutions that append offending IPs after X connections?

thanks

View on HN

I'll preface this with "I used to work for an online poker company and every mofo in the planet used to attack our network"

Firstly consider what type of target are you. Most DDoS attacks are launched for a lot longer the 30 minutes and they are certainly not regular. They are looking to see if they can bring you to your knees so they can blackmail you. Is your site that valuable? If so forget everything and get yourself a cisco firewall and pay someone who knows what they are talking about. Cisco are expensive but if you have a valuable site that 1% of real traffic needs to get through.

Assuming your not in a position to be blackmailed: As other ycombo's have mentioned logging and blocking are your friends but be careful. You say it's mysql taking the time despite caching. Looks to me like you've found a bug in your code (or at least your caching). Log what these IP blocks are requesting. If it looks algorithmic then the chances are you've got a crawler ignoring your robots.txt. Contact Purdue. Call them up (they will ignore your email) and ask them what's going on.

You could chose to block these ip ranges but if you make your site weather this storm it will be stronger in the future.

Good luck

Which poker site?

I was lead programmer at PKR http://www.pkr.com

I've never even heard of that one. Very odd. Did they have a lot of action?

It's one of the fastest growing poker networks in Europe. Only been going three years, I left after two.

Are they accessing pages, or just opening connections? Use netstat to see what state the connections are in.

Make sure syncookies are enabled if they are just opening lots of connections (http://cr.yp.to/syncookies.html).

You can limit parallel connections per host with iptables. See 'connlimit'. Drop any invalid SYN packets. There's also 'recent' which you can use to keep a dynamic list of ip addresses sending n SYNs over the past m seconds (then drop new connections). Bonus is that's the list of IPs is accessible/modifiable from /proc. Be careful not to kick out legitimate clients by setting too low a limit, though. Iptables can log, too, so maybe you can sample connections for a few seconds, 'sort | uniq -c' the ips, and decide on a cutoff.

There are also network appliances that will do similar things without loading your web server.

SHOW PROCESSLIST on your mysql, figure out what queries are happening. It could just be that you need a new index, more appropriate configuration, or better queries. In any case, at least it will give you a clue as to what is causing the load.

Use mod_status (or similar for your web server) to figure out what your Apache workers are doing. Modify keep-alive times.

If all else fails, see if your ISP can enable TCP Intercept on your nearest router. (http://www.cisco.com/univercd/cc/td/doc/product/software/ios...).

Could be web crawlers for search engines (or research, at Purdue?)

They're supposed to play nicely, especially if you have a robots.txt file set up. Try that first if you don't.

Squid as a reverse proxy can rate-limit inbound connections of various kinds of groupings, which may help, especially if you can define a grouping which is exclusively the kinds of remote IPs/user-agents/URIs that tend to become abusive. I believe Squid 3 even has a way to delegate the grouping decision to an external script, so you could write your own kind of triggering logic.

This does the automatic iptables manipulation you're looking for based on inspecting netstat:

http://deflate.medialayer.com/

That said, it's not really DDOS protection. It is good for blocking bad crawlers or misconfigured / casually malicious clients. When attacks get to be true DDOS - hundreds or thousands of zombies intent on taking you down - you'll almost certainly need purpose-built hardware and an expert network admin to deal with it.

Use cache priming and serve slightly stale content out of the cache when the most recent data isn't cached and then update the cache in the background.

And check your indexes!

Plus, write a decorator around your database query function to log all queries and make a list of longest queries and longest average query times.

If you want to go the iptables route (have fun...) check out the --limit option.

Here's an example: http://www.webservertalk.com/message1796841.html

I'm a fan of turning on apf's anti-dos whenever I see load spike.

This question has been on my mind for while.Hope to hear some great responses.

Are you sure you are not being "dugg" or anything else similar?

I was looking at apache-top referrers the whole time - nothing crazy there.

Reply on news.ycombinator.com