Hacker News Reader

Comment by brongondwana | Hacker News Reader

If we were pure web we would have ducked behind Cloudflare immediately. Since we do SMTP/IMAP/POP3 as well, we've had to go with a more complex (and costly) solution.

This is our theory for why they're currently attacking email providers. We're not "just a web site", and attackers realise that the situation is more complex for email sites, we can't just hide behind Cloudflare.

We're not sure who's actually attacking us, the ransom note comes from a freemail provider and a connection from a tor exit node. We can only guess at their total capabilities.

View on HN

Sorry to hear about that... Do get in touch with one of the providers listed above, they may be able to help you out in the short term for free in exchange for publicity.

Yep, thanks - we've already spoken to a few of them and got some things in place. Obviously don't want to spell out all our measures in public.

0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gear with either 2 (or 3, if deploying a private, admin network too) NIC teams for isolating traffic). This will let you do raw L3 traffic measurements on each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network banning (if you don't own/admin the network gear or don't want to touch it in production). These are super cheap and only need ~128 MiB RAM each and very little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD partition if possible). (Your public IP(s) should point to these boxen.)

1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy.

nginx: Compile it from source if that's all you need, and reduce your attack surface. http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html

haproxy: http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infr...

2. Setup something like fail2ban: https://rtcamp.com/tutorials/nginx/fail2ban/

3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.)

Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.

this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.

Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe.

(we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing)

We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping.

I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host.

It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary.

Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.

That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition.

The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost).

The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.

This doesn't stop traffic from getting to you. If the attacker has more bandwidth than you, you lose. Complete outage. All your customers think you're down because they can't get packets through.

You need the DDoS traffic filtered upstream, not stopped at your firewall. This is not a trivial problem to solve.

If you're running BGP you can stop small-ish DoS attacks by setting up a blackhole BGP community that is propagated up to your providers. Any IPs you put in will no longer have their traffic forwarded. This doesn't work for DDOS which has countless IPs attacking you. You will have to blackhole yourself and take the target IP off the internet to stop wasting all your bandwidth.

So the solution is to have it filtered upstream by someone who can clean and absorb the attack. It adds complexity to your network architecture (filtering provider has to announce your routes for you) and it's not cheap.

Bummer to hear that :( Keep up the good work! I'm a very satisfied customer, and I wish you all the best.

Reply on news.ycombinator.com