Hacker News Reader

We know that we've dropped the ball here. To be frank, it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent than "they're attacking our webservers", "they're attacking our core routers", etc.

> And it looks like the attackers decided to start things back up again, as Linode.com is unavailable...

They're watching our status page for updates and starting new attacks when we resolve previous ones. There's been an almost 1:1 correlation lately.

I just registered to thank you for the update and to let you know I won't be moving away from Linode as this just gives the attackers what they want.

Indeed, well, keep fighting the good fight. We are cheering for you.

The correlation could also be the fact that you've blocked the attack, they can see that their previous attack is no longer working by testing it themselves - so they'll switch anyway, regardless of a status page update.

Thank you for the update. We really appreciate the information and wish you the best of luck in dealing with this.

> it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent

This always rings hollow to me, and yet I hear it over and over.

A company like Linode surely has at least a dozen people who can be on call in a situation like this, probably much more. All it takes is for one engineer or even a product person... Heck a technically-minded support person could listen in on the war room meetings and get enough information to post something better than "we're fixing it".

5 minutes of blogging every six hours would be plenty.

And yet, people always claim it's impossible and there's no time. Frankly I find it frightening... If you are coding that fast that no one on your team has five minutes to step aside, take a breather after six hours of coding and summarize what the team just spent the last six hours doing, I shudder to think what kind of panicked alarmist interventions you are making.

There's no excuse for silence. There just isn't. It's a gross failure on the part of management to prioritize the responsibilities they have to your customers customers. Full stop. Sure, the engineers can't be expected to remember to tap out to blog. But if that's the extent of the accountability structures you can assemble during a crisis, that is a serious organizational failing, particularly for an organization the size of Linode.

I would say that 5 minutes of blogging every 6 hours is just as likely to cause more harm than good. Wording something badly could lead to far more fear and confusion than the present situation if taken out of context.

[Disclaimer: Linode employee]

Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey, we need to give some clarity to customers" are less available than the people whose time is spent firefighting.

Can you share with us the mitigation techniques you tried so far and what worked better, what did not work at all, etc?

Stopping them from directly hitting our routers has been the most difficult, because our routers have IPs on 1000+ /24's, which means RTBH can't work because no tier 1 will accept 1000+ null routes. The same is true when they attack our upstream provider's networks.

We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.

As a happy linode user, your update is much appreciated.

I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires.

I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.

Stopping DDOS isn't so much about innovation as cost: it's metric tons of data thrown at points you control and don't which have to analyze and react to it somehow. That's assuming it doesn't straight-up fill the pipe where ignoring it changes nothing. Here's a case study showing how expensive DDOS mitigation can be for a smaller firm:

https://protonmail.com/blog/ddos-protection-guide/

I think "whole data center is offline" is the sort of thing you call the boss for. Otherwise why is there a boss at all?

"It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business."

The timing of the DDoS was pretty interesting too, happening when not everyone is available.

This was my first thought when I woke up to a bunch of my server monitor alarms at about 4am on Christmas day....

I've seen Linode getting a lot of flack for not updating customers on what is happening, but what clarity does this announcement provide that their DDoS status page wasn't providing? They were updating the status page regularly.

It gives those of us hosted on Linode a clear place to point our customers.

Push, not pull. Send an email. Don't expect people to go to a status page on their own.

Which they could easily do by enabling SMS/email/webhook notifications in their statuspage.io settings. As it is, you can only subscribe to an RSS feed it seems.

> As someone whose business relies on Linode currently to function

Perhaps its time to consider some failover at another host. Same goes for anyone solely dependent on anyone.

How do you guys re-route traffic for your load balancer to another in another data center? If you change your DNS record it would take x time before it has propagated.

Say I have one server setup on Linode London and one duplicated on DO Amsterdam. How do I quickly re-route the traffic from my two main load balancers London to Amsterdam without much propagation time?

Keep TTL's low (3-5 minutes). Things don't have to happen instantly. Clients understand. You had a bad day on the Internet. Guess what, stuff happens. Remember when "XYZ company down for 2 hours" made headlines? Yeah, that was a long time ago. Downtime is more or less accepted these days. Sure, some hothead customers will flip out and leave. You were probably not going to meet their expectations sometime and lose them eventually. Let them become someone else's nightmare.*

* This is for an answer for the average "Hey I run a website or two" question. If you're asking and you are dealing with financial transactions or ad networks, well, you're asking in the wrong place and you should probably hire someone that knows how BGP anycasting works.

>* This is for an answer for the average "Hey I run a website or two" question. If you're asking and you are dealing with financial transactions or ad networks, well, you're asking in the wrong place and you should probably hire someone that knows how BGP anycasting works.

Eh, usually you just bgp anycast the DNS server. If you anycast just your DNS server, you are pretty much in the same pickle as if you host your dns with someone else who runs a reliable dns server (and have them do the anycast) - the pickle is that there's a minimum ttl.

Now you can set up a redirector server and anycast that, and like 3xx redirect all your requests (or more often just the first request) and that's better, really, 'cause that redirect can be changed quickly and on a per-request (or more often per-session) basis... but while I know a thousand companies that will sell you anycast DNS service, I don't know anyone who will sell someone with a vps customer budget an anycast'd redirector service.

You cab use round-robin DNS ... Keep two DB in sync between different hosts/providers and effectively switch the DNS if host1 or host2 goes down. However this setup is easy for a kind of applications but may not be suited for other kind of apps.

We use a CDN. They provide caching and act as another layer of defense beyond the origin datacenter/provider.

Because they physically route the traffic, you can change backends pretty quickly and have traffic going to the new destination within seconds or minutes. Using DNS for end-users just has too much unpredictable latency, but it works well for routing the backend.

What's the current status for Database failover? MySQL in my case - to mirror that is what's been holding me back.

What happens with 1/10 second's database records not replicated? That could be a successful transaction on one, and no record of it on the other?

> What happens with 1/10 second's database records not replicated?

You can use a multi-master setup and strict consistency checks, though.

Multi-master MySQL is generally regarded to be a very, very bad idea. Single master with multiple GTID replicas and promoting a replica to master upon failure is BCOP.

Certainly what we're in the process of doing.

I'm surprised you haven't completely jumped ship already, considering that there are two other providers in particular that offer very similar specs and prices to Linode. Where I work, we moved our primary infrastructure off of Linode the night after the attacks started.

I've tried a lot over the years, no one can compare to the VPS quality Linode provides is what I found.

All (major) providers suffer from DDoS attacks.

When was the last successful sustained (1 week plus) CloudFlare, GCE, or AWS DDoS attack?

I have no idea.

CloudFlare doesn't offer VPS instances. Google and Amazon are in a league of their own and, last I checked, don't offer the similar services at the same prices with the same level of support and same controls. Amazon might send you a nice bill at the end of the attack depending on what type it was.

DigitalOcean, as an example, routinely receives DDoS attacks and will just nullroute your VPS automatically until the attack ends.

Its a concern of mine with AWS, no downtime but a $30k for the day of fun bill. Got to tie up this loose end before I wake up broke one day.

There are VPS providers that offer DDoS protection in the 10-20Gbps range, but that is a piddle in the ocean these days.

Which ones and protection against what attacks?

Ramnode, Vultr, others.