Comment by andrem | Hacker News Reader

Comment by andrem | Hacker News Reader

Parent Full thread

DDoS attacks today are such a commodity. It takes next to nothing to launch them.

You can get upwards of 200Gbps for 1/6 of a bitcoin. It is very easy to setup and you can DDoS your favourite site in a matter of minutes.

These are not very smart attacks and can be mitigated even using the free tier of cloudflare.

I don't have the background on the mail provider attacks but 6.5k ransom seems to come from attackers who use easily available booters. Hushmail switched to Cloudflare throughout their attacks and that seemed to have helped, not sure what fastmail will do.

But any public web service should not be in a position where they are vulnerable to off the shelf DDoS attacks.

DDoS attack as suffered by for example Githuh with heavy coordination and nation states behind them require more specialised defenses. There are commercial alternatives out there that go from anywhere between 9k-40k per month depending on bandwidth and technology - see Imperva, Prolexic, Neustar, Nexusguard, Blacklotus, Incapsula, etc..

Apart from the initial setup which is more involved than Cloudflare's there is not much to do apart from throwing money at it. Quite the money making business really :)

If we were pure web we would have ducked behind Cloudflare immediately. Since we do SMTP/IMAP/POP3 as well, we've had to go with a more complex (and costly) solution.

This is our theory for why they're currently attacking email providers. We're not "just a web site", and attackers realise that the situation is more complex for email sites, we can't just hide behind Cloudflare.

We're not sure who's actually attacking us, the ransom note comes from a freemail provider and a connection from a tor exit node. We can only guess at their total capabilities.

Sorry to hear about that... Do get in touch with one of the providers listed above, they may be able to help you out in the short term for free in exchange for publicity.

Yep, thanks - we've already spoken to a few of them and got some things in place. Obviously don't want to spell out all our measures in public.

0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gear with either 2 (or 3, if deploying a private, admin network too) NIC teams for isolating traffic). This will let you do raw L3 traffic measurements on each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network banning (if you don't own/admin the network gear or don't want to touch it in production). These are super cheap and only need ~128 MiB RAM each and very little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD partition if possible). (Your public IP(s) should point to these boxen.)

1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy.

nginx: Compile it from source if that's all you need, and reduce your attack surface. http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html

haproxy: http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infr...

2. Setup something like fail2ban: https://rtcamp.com/tutorials/nginx/fail2ban/

3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.)

Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.

That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition.

The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost).

The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.

This doesn't stop traffic from getting to you. If the attacker has more bandwidth than you, you lose. Complete outage. All your customers think you're down because they can't get packets through.

You need the DDoS traffic filtered upstream, not stopped at your firewall. This is not a trivial problem to solve.

If you're running BGP you can stop small-ish DoS attacks by setting up a blackhole BGP community that is propagated up to your providers. Any IPs you put in will no longer have their traffic forwarded. This doesn't work for DDOS which has countless IPs attacking you. You will have to blackhole yourself and take the target IP off the internet to stop wasting all your bandwidth.

So the solution is to have it filtered upstream by someone who can clean and absorb the attack. It adds complexity to your network architecture (filtering provider has to announce your routes for you) and it's not cheap.

this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.

Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe.

(we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing)

We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping.

I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host.

It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary.

Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.

Bummer to hear that :( Keep up the good work! I'm a very satisfied customer, and I wish you all the best.

What's a good setup for a medium sized site to use that would protect it from off the shelf attacks?

The only real defense is to serve your site from somewhere with more incoming bandwidth than incoming abuse. Afaik, the only type of filtering you're likely to get from an upstream provider is null routing of attacked ips, which helps protect their network, but doesn't help you serve users (you can switch ips, but abusers will likely switch too)

We used to use TCP proxies when I ran a game server where skiddies would try to attack it. I know buyvm.net is where we got one small $15 a year VPS with $3 a month "DDOS protection", not sure if we ever had issues after that with DDoS or not, we kept getting more and more proxies. We would give users a different one upon refreshing the page (it was a browser based game, but they would connect to a game server via TCP). I know the Minecraft community offers proxies as well. If you're on HTTP though, cloudflare is one of the cheaper options.

If you're only web, you should be perfectly fine behind CloudFlare.

Reply on news.ycombinator.com