Hacker News Reader

Comment by mmaunder | Hacker News Reader

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

View on HN

I am not sure there is a magic way to mitigate these attacks permanently other than playing cat and mouse until the attacker gets tired. Linode has in fact mitigated the attacks, only to find them pointing to a different part of their infra or DCs. At this point, it is probably a good sign that they haven't gone completely down (for long). Current tooling and botnets make DDoS-ing fairly easy today, despite countermeasures.

This type of action is what the people behind the DDOS attack are looking for. Its not Linode fault...

Thank you for being so understanding!

Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point. I filed a support ticket trying to get more information about the outage, and the response I got was absolutely worthless. No part of this has made me feel better about Linode at all.

AWS is so massive that they can just sustain most DDOSes, and they write real postmortems after attacks. They're not as fast as Linode, but what good is speed if my site is completely vulnerable to every botnet that decides to look Linode's way?

>> Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point.

Then it will also be the same when you are at XYX ISP and they get DDOS'ed. This isn't a Linode specific problem.

>> AWS is so massive that they can just sustain most DDOSes

No.

http://stackoverflow.com/questions/13244713/does-aws-protect...

"It will not, however, employ security algorithms needed to deflect application layer attacks (e.g., HTTP flood) nor will it help against larger network layer threats."

https://www.cloudflare.com/ddos/

"Layer 3 and 4 attacks are difficult—if not impossible—to mitigate with an on-premise solution. "

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is the attack? Is it targeting a client or Linode itself? And if they're working with their upstream providers, what are they doing?

Edit: In the mean time, I am (sadly) moving docs.wordfence.com back into our data center. I have access to the Dallas server again and so am starting the transfer now. If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.

If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.

I've been a generally happy Linode customer for a long time, but this is my biggest gripe with them. I can almost guarantee you'll never get this. Linode just doesn't do it. Their postmortems on issues (if they ever come) are light on details of the problem, and even lighter on details about their solutions. They seem to do a decent job of addressing most issues, because they rarely have recurring problems, but they never really talk about what they're doing.

Same experience here. I asked Support once about an ongoing outage and was told that "there were some network related issues". Really? I wouldn't have guessed...

Heard of nullroutes?

(Edit: how is it that perfectly valid technical solutions get downvoted?)

I didn't downvote, but I can tell you almost certainly it was because of the snark.

I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it.

If you don't already have it enabled....well, good luck mate.

As of a few years ago Linode got transit from the facilities they are in and almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would be incredibly foolish to operate a hosting provider without it.

Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there.

From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?

Just because they purchase transit doesn't mean they don't have their own AS.

Everyone has to purchase transit at some point. Transit != AS.

Where did I say that purchasing transit relates to an AS? I was speaking out them separately. "House bandwidth" in most colos is pretty garbage, and you don't have much control over it.

I'm a lot less impressed with Linode after this thread than I was before.

That's correct. They use facility transit everywhere except their newest datacenters (Singapore and maybe Frankfurt), where they operate their own AS and use blended transit.

I don't know if I'd outright call it a problem, per se, since in a couple facilities (say, Fremont) it's a good thing given how good the provider (say, Hurricane Electric) is at providing transit. So in the datacenters where the provider is better at transit than facilities (say, Fremont), the equation breaks in their favor.

When I left they also didn't really have the staff to run a proper NOC for a full-on AS; one extremely talented network engineer who has since left mostly owned all the "running a grown-up network" stuff, like running down an ASN, working RTBH with the facilities, well-designed uplink strategy at Linode's edge, and lobbying for anycast infrastructure for DNS and so on. They had just hired another purebred networking wonk as I moved on who probably owns it now, and likely has help. So punting a lot of the typical NOC work upstream to a facility with whom you're already working anyway made a bit of resource-related sense at the time, since staffing a NOC for a network of that scale is a significant challenge and they like to operate lean.

That's changing now (I didn't know this and learned it from agwa's comment), which is probably reflective of growth on the resource front.

Real talk: it's not that hard to run your own network. It sounds like Linode was skirting by on not making the infrastructure investment both in hardware and people. It's finally come back to bite them. No excuses and no pity for them.

Also, I wouldn't run around saying that having HE in house is a great thing.

I would hope it is obvious based on the extent and duration of the attacks that RTBH is not an option in this case.

Attackers shift to a new target on the same net as IP's are null routed, until the entire network has null routed itself which achieves the same result as DDoS'ing the network. So I'm not sure I agree that removing target IP's from your network is a great solution.

I've been with Linode for years and they've fought off more than a few DDOS attacks in the past, so they certainly are competent in that regard. I think this one must be a higher level of severity than previous ones. Generally, they seem ok with this stuff. I don't think there are any DDOS-proof VPS's out there.

Reply on news.ycombinator.com