FastMail under DDoS Attack
blog.fastmail.com
blog.fastmail.com
Our upstream implemented layer 7 mitigation which did an unbelievably effective job at stopping the attack in it's tracks. I don't know the tech that they used, but it performs deep packet inspection up to the application layer and they charge a modest additional fee for passing our traffic through that system.
The effect was that our traffic dropped to very slightly below normal levels during the attack, which would indicate that there were probably a few false positives, but we didn't have a single customer complaint.
Fastmail, pro tip: don't pay any ransom.
It could seem to some people that such a stance is easy, but no matter the strength of principle, when you see your business go offline and customers start banging at the doors for you to sort it out then the situation becomes more complex.
So from this customer - keep up this stance, keep being transparent and I for one will stick by you guys without hesitation!
DDoS mitigation isn't that expensive antway considering the size of FastMail.
Turns out they are the type of people who pay to make the problem disappear.
You can get upwards of 200Gbps for 1/6 of a bitcoin. It is very easy to setup and you can DDoS your favourite site in a matter of minutes.
These are not very smart attacks and can be mitigated even using the free tier of cloudflare.
I don't have the background on the mail provider attacks but 6.5k ransom seems to come from attackers who use easily available booters. Hushmail switched to Cloudflare throughout their attacks and that seemed to have helped, not sure what fastmail will do.
But any public web service should not be in a position where they are vulnerable to off the shelf DDoS attacks.
DDoS attack as suffered by for example Githuh with heavy coordination and nation states behind them require more specialised defenses. There are commercial alternatives out there that go from anywhere between 9k-40k per month depending on bandwidth and technology - see Imperva, Prolexic, Neustar, Nexusguard, Blacklotus, Incapsula, etc..
Apart from the initial setup which is more involved than Cloudflare's there is not much to do apart from throwing money at it. Quite the money making business really :)
This is our theory for why they're currently attacking email providers. We're not "just a web site", and attackers realise that the situation is more complex for email sites, we can't just hide behind Cloudflare.
We're not sure who's actually attacking us, the ransom note comes from a freemail provider and a connection from a tor exit node. We can only guess at their total capabilities.
1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy.
nginx: Compile it from source if that's all you need, and reduce your attack surface. http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html
haproxy: http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infr...
2. Setup something like fail2ban: https://rtcamp.com/tutorials/nginx/fail2ban/
3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.)
Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.
(we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing)
We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping.
I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host.
It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary.
Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.
The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost).
The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.
You need the DDoS traffic filtered upstream, not stopped at your firewall. This is not a trivial problem to solve.
If you're running BGP you can stop small-ish DoS attacks by setting up a blackhole BGP community that is propagated up to your providers. Any IPs you put in will no longer have their traffic forwarded. This doesn't work for DDOS which has countless IPs attacking you. You will have to blackhole yourself and take the target IP off the internet to stop wasting all your bandwidth.
So the solution is to have it filtered upstream by someone who can clean and absorb the attack. It adds complexity to your network architecture (filtering provider has to announce your routes for you) and it's not cheap.
I've been trying out different email services lately in an effort to untether myself from Google, and this one seems like a really good choice.
I'm not into the iOS themed non native Android app though, but being an Android developer I'm more averse to that nonsense. It seems to work great and is really fast though. Does anyone know if there is a native app planned?
I'd be willing to pay a subscription for email if it's worth it. I'd love to hear others' experience with them.
Could probably make a workaround by creating a folder and setting a rule on it, or something.
So I tried out Fastmail with my alias and had no trouble getting it working the way I wanted it to work. So I've been with Fastmail ever since.
This doesn't happen if you configure an external SMTP server in Google Mail for the alias.
http://gmailblog.blogspot.de/2009/07/send-mail-from-another-...
https://support.google.com/mail/answer/22370?hl=en
Moreover, you don't want to use a different 'from' address without sending through the appropriate SMTP server. If the domain has SPF or DKIM set up, a receiving server might reject your mail if it wasn't sent from an expected SMTP server.
Ideally of course I'd be self-hosting, but that's a tarpit I'll leap into at some time in the future.
Wish I never bothered, that is a hassle you just don't need. Getting it up and running so it just works is a hassle enough.
Thinking of going on holiday and want auto responses? You better plan that about a month in advance to figure it out :-(
As you may tell, I am thinking of moving to fastmail for domain email hosting
It's a very good no-nonsense service based on standard protocols, although a bit pricey (compared to a grandfathered, free Google Apps account). They've lately been expanding to offer a more complete offering (CalDAV, CardDAV, etc).
This means it plays nicely with any platform where these standards can be applied (Googley or not) and it also means it's easy to take data ownership seriously, if that's your thing.
I only have positive experiences with fastmail, and I like that they don't have messed up and bloated their WebUI like Google have.
Currently I'm only using it for my personal email because of price. It's not that expensive, but because it's 100% free for me, I still have the rest of my family on the Google apps account.
The reassuring thing about paying for a service like this is that you know who the customer is. Fastmail will not change their UI to sneak in a "social network" in your inbox just to mine more data and milk more ad-dollars out of you.
Edit: Clarifications about "pricey"
As mentioned, OP had a grandfathered Google Apps for Business (so, free forever?). Another alternative is to sign up to a $5-$10/month web hosting package which normally comes with free email hosting.
Disclaimer: I'm a FastMail customer, been really happy with their service :) I also have a grandfathered Google Apps for Business account.
I hadn't noticed any slowdown, so kudos to the Fastmail guys!
BTW, Fastmail is amazing, been using it for a while now and super happy. It's solid and has some really good features.
Which means if I were to migrate all my (perma-free) Google Apps account (for the whole family) and were to retain my current quotas, I would have to shell out $40*10 or so.
Fastmail is nice and all, but I know lots of ways I'd rather spend $400 which doesn't involve being tech support for 10x family members needing help having their accounts migrated on all their devices.
Even if they did, it's $3.33 per family member per month.
And if they don't need that much, it's only $0.83 per family member per month.
However it's a total non-starter if they can't archive email forever. Bonus, like faxing still is for some in business, today's worker (and end user) is going to force everything through that hole, because it's the magic service that just makes it work... and when it doesn't they can blame it.
Also, additional storage is much cheaper. You get 100GB extra storage for $1.99 per month or 1TB for $4 per month (or 'unlimited' if the domain has more than five users).
But I agree that $40 per year is not pricy for a fast e-mail service with a lot of redundancy. Plus, Fastmail contributes a lot to open source projects such as Cyrus.
(Note: I have both a Google Apps and a Fastmail account.)
I've been a Dropbox user even before switching off Google Apps, because Dropbox has a much better client and compared to alternatives, storage for them is not just a complementary to something else, so for example they support Linux as well. Which is very important for a multi-platform guy such as myself. Dropbox is also integrated with Microsoft's Office Online, with Gmail (by means of a Chrome extension) and with Fastmail's web interface. So from Fastmail's web interface you can attach files straight from Dropbox. Plenty of apps have integration with Dropbox actually, like for example 1Password. Dropbox is also the suggested alternative to iCloud by Apple.
And that's not the only option. If you're a power user interested in security, there's also SpiderOak. It's a bit more pricey, but that's because they are doing encryption and so cannot take advantage of duplicate files and other gimmicks like that. And it's worth it for people worrying about the privacy of their data.
> 1TB for $4 per month
The pricing you're talking about is about the Vault option and is $5 per user per month and not $4. And the big problem is that's misleading. That's $5 per month per user and is applied for all users in your Google Apps account, whether they need it or not. My wife for example certainly does not need 1 TB and for small businesses that can be very problematic, as you can easily pay an extra $100 per month.
In other words, I see no reason to encourage a monoculture on the basis of tighter integration or complementary pricing that's misleading. We've been experiencing this strategy time and time again in the past from companies like Microsoft. You'd think we should have learned by now. In fact such marketing strategies are exemplified in books such as "Predictably Irrational" by Dan Ariely. That was an interesting read if you're interested.
And it never ends up well, either for consumers or for the industry at large. And you've got good options available that I think are better than Google Drive.
> an office suite with collaborative editing
But nobody stops you from continuing to use Google's Docs, in combination with Fastmail and Dropbox or whatever. I've done that, it's not bad and should not be a reason to keep using Gmail. Big companies like Google, Microsoft or Apple want you to get from them everything but the kitchen sink, because that's how they achieve lock-in, that's how they can use their brand muscle to make you buy shit you don't need or stick to inferior options. You shouldn't forget that Gmail is about email and if Gmail no longer does email well for you, then complementaries like Google Drive or Google Docs won't make it magically work better at email.
But btw, did you know that Microsoft's Office Online can edit and save files as ODF, the standard document format and Google Docs does not support ODF? In fact Google Docs doesn't support editing any of the common formats, as they require conversion in their own format in order to edit those documents, leading to a form of lock-in that Microsoft has only dreamt of.
Apart from the idea of not putting all my eggs in one basket, this is crucial for me too.
Dropbox is the only cloud-storage service with a good and working Linux client. All my machines at home runs Linux, so not supporting that means I wont even consider using the service.
Oh, definitely. Outside mobile, the Dropbox client is miles ahead. I was just saying that a Google Apps account and Fastmail account is not directly comparable, since Google Apps offers so much more.
If you're a power user interested in security, there's also SpiderOak.
I don't see the added benefit. As long as the standard client is closed source, it's only a bit better from a security perspective.
That's $5 per month per user and is applied for all users in your Google Apps account, whether they need it or not.
Definitely. But we were comparing to Fastmail, where storage costs 1GB for $5 USD/year for enhanced accounts. Just for comparison, for 100GB that is $41 per month above the base account cost, compared to $1.99 per GMail. Then a $4 per month account plus $1.99 for 100GB or $5 for 1TB doesn't look so bad.
But nobody stops you from continuing to use Google's Docs, in combination with Fastmail and Dropbox or whatever.
If you use Google Docs outside Google Apps, your documents can be mined for advertising. No thanks!
You shouldn't forget that Gmail is about email and if Gmail no longer does email well for you, then complementaries like Google Drive or Google Docs won't make it magically work better at email.
I have Fastmail and Google Apps and I still like Google Mail more in general. For instance, I prefer labeling over folders and the mobile GMail/Inbox apps are a far better experience than the Fastmail app.
But I don't agree with the premise. One of the nice things of Google Apps is integration, e.g. mail <-> calendar, Google Now, and Inbox. Dropbox realized how important this is and started pushing integration beyond providing an API for apps (Office Web integration, Office plugins, Google Mail extension, etc.)
I use Fastmail, and they support standard internet protocols like IMAP and CardDAV. Which means I can use the standard Android email-apps, CardDAV sync, and have everything still work just fine.
No need to use an email-provider specific app, although Google and Gmail has tried to mentally brainwash everyone and their grandmothers that this is how email actually works.
Thanks for being one of our customers :)
I'm seriously looking forward to JMAP, I hope some other big services adopt it when it stabilizes. Give your JMAP dev(s) a hug from me, please!
Oh, and I hope the DDOS doesn't cost too much - any chance we could have a blog post about the costs of the DDOS attack after it's all over? I know most companies don't like talking about operational costs, and I'll understand if you guys are the same. Cheers!
I'm looking forward to JMAP as well - there are a few of us working on it (I wrote the proxy, which is in serious need of some love)
Great spam detection, no viruses.
Never been out
I'm sure other providers give similar access, but they also make it very easy to create alternative email addresses for the domain I own. Once I started using them, I started setting up a different email address for every company I did business with online and that's cut down the amount of spam I receive dramatically. Basically, when I see a compromised email address, I disable it and that seems to do the trick.
In any case, they've provided me rock solid service and that's worth the $40/year that I pay.
My experience is their people are awesome and their product is awesome. You won't regret signing up.
However there are things I'm really used to in Gmail, for example the unread messages at the top, and seeing messages in several labels. So Fastmail feel a bit less advanced.
search is:unseen works nicely to find them on mobile as well, and you can save it to your sidebar/folders list.
Only ever had one issue (a small bug to do with some very specific domain config) - their support team emailed me back straight away, it was escalated to the devs, and fixed almost immediately.
Now with CalDAV and CardDAV support it's a no-brainer.
It was trivial to IMAP import mail from my old host. Good support for multiple authentication schemes, i.e. I have a 32 random char master password saved in my password manager, and they allow me to add any number of alternative authentications like shorter passwords that are only allowed in combination with Yubikey/Google Authenticator or one time passwords.
As for "app" I just use a generic IMAP client.
Best of luck to the Fastmail team, I hope they are able to weather the storm out.
SMTP/IMAP/etc. are pretty crappy protocols in a lot of ways, but they're what everyone has deployed. They can be proxied like HTTP/HTTPS. There are spam/reputation issues with outgoing traffic, too, which makes this even more annoying.
www.arbor.net
www.radware.com
www.voxility.com
As long as there are services, and as long as those services have finite capacity, there will be DDoSes -- both accidental and intentional.
For a while, the most common way was with botnets of compromised PCs. They still exist, but big attacks with them are less common since Microsoft has gotten better at securing people's computers. The big thing now is "amplification attacks": basically, finding a way to send a small amount of data and get some other host to flood your target with a huge amount of data in response. Search "NTP amplification attack" for details. More recently, China has weaponized the Great Firewall to be yet another DDoS vector: they inject JavaScript into pages that people visit, and that JS floods a target with requests.
As long as there is some way to point a lot of requests somewhere you want, DDoS attacks will be a thing.
What would the AI do?
The problem is your CPUs filter traffic faster than your NICs accept it, thus dropped packets, thus unhappy customers.
In a previous life, I ran physical datacenters, and while the gear wasn't terribly powerful then (we're still worried about running out of memory on core routers, hence why IP blocks don't get sliced up and piecemealed out with the exhaustion of IPv4 space), I'd expect newer hardware to be able to keep up.
The network can remain irrational longer than you can stay online.
I don't know much about this stuff, so I'm extrapolating and pseudo-solving.
You could work to notify the network owners, but it's whack-a-mole; even with strong efforts there are enough DNS and ntp servers out there configured to generate a pretty big reflection.
The reason the attacks are still viable is because little has happened to the Internet itself. We still have the same challenges we had 15 years ago. Some things are slowly getting better, but it's still fundamentally the same. Increase in overall bandwidth and vulnerabilities doesn't help either.
There are exactly two workarounds: Minimize processing of packets that you think are coming from the attackers, and have more servers.
Asking why we haven't solved DDOS is kind of like saying "Why can't we cure decapitation? We've got all of these new antibiotics!"
The firewalls that we are running were easily able to absorb the additional traffic. Neil's metaphor of the post office was quite accurate - even in the heaviest attack, when we didn't have upstream DDoS protection turned on, about 10% of user traffic was getting through just fine - it's just that a random 10% of traffic makes for a very poor TCP experience.
Afaik fixed silicon edge routers peering tier 1/2 networks were the biggest obstacle in filtering good traffic from spoofed/botnet one. Just a year ago we had huge problems when BGP rose to >512K entries, which is an order of magnitude easier.
Did anything change since ~10 years ago? Last time I dabbled in this it was so bad even Tier3 (ISPs) werent filtering spoofed packets.
Of course our growth strategy is quality (people willing to pay for a good service quid-pro-quo) over quantity (free service and monetise later via ads/analytics), so we've had a slow steady growth for the entire 15 years we've been operating rather than the viral growth and sell-out/pivot that unicorns are known for.
People like Ford and Edison used to believe that you could get more quality as volume increased. And in fact, if you wanted to increase quality, then you had to increase volume.
And I do agree to a point. We're very happy to have increasing numbers of users so we can afford to do things like contracting the excellent developer who's working on JMAP support for Calendars in Cyrus IMAP at the moment, as well as hiring people to add new features or improve existing ones.
We do try to stay at a point where we can run comfortably on 50% of our hardware, so we can shut down half our machines at once for maintenance. Redundancy certainly helps - we've blogged a few times about how good it is to be able to shut down any one machine with only a few minutes' warning to move active users off it.
I'm certainly looking forward to spending time again on what I wanted to be doing (Cyrus IMAPd improvements at the moment) rather than battling a DDoS!
- Build defenses against DDoS into its networks
- Have large enough capacity that such attacks are less effective anyway
It would be interesting to see how often people try though.
Google has been buying up 'dark fiber' for years and has thousands of miles of cable connecting their data centers.
They can certainly handle petabit/s levels of traffic inside the datacenter[1], it's not that much of a stretch to think that they can handle double digit terabit/s through their collective external fiber links.
Also, just think about their normal level of operation. Even just all the Android devices feeding data back and forth, let alone analytics, maps, gmail, search etc etc. They've got 36 data centers and co-locate in more than 60 public exchanges (and that was in 2010!), not to mention the Google Global Cache (GGC) servers inside consumer networks across the globe.
Their scale is ridiculously large. I suspect that they actually can't be DDoS'd in the normal 'chuck traffic at them' sense.
[0] http://www.theregister.co.uk/2010/03/17/the_size_of_the_goog...
[1] http://googlecloudplatform.blogspot.co.uk/2015/06/A-Look-Ins...
On a global scale this would mean that the one big DDoS you'd expect to see effectively gets split into many tiny DDoSes, which Google can handle using methods mentioned in the other responses in this thread.