Hacker News Reader

Comment by kyrra | Hacker News Reader

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

View on HN

That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is the attack? Is it targeting a client or Linode itself? And if they're working with their upstream providers, what are they doing?

Edit: In the mean time, I am (sadly) moving docs.wordfence.com back into our data center. I have access to the Dallas server again and so am starting the transfer now. If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.

If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.

I've been a generally happy Linode customer for a long time, but this is my biggest gripe with them. I can almost guarantee you'll never get this. Linode just doesn't do it. Their postmortems on issues (if they ever come) are light on details of the problem, and even lighter on details about their solutions. They seem to do a decent job of addressing most issues, because they rarely have recurring problems, but they never really talk about what they're doing.

Same experience here. I asked Support once about an ongoing outage and was told that "there were some network related issues". Really? I wouldn't have guessed...

Heard of nullroutes?

(Edit: how is it that perfectly valid technical solutions get downvoted?)

I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it.

If you don't already have it enabled....well, good luck mate.

As of a few years ago Linode got transit from the facilities they are in and almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would be incredibly foolish to operate a hosting provider without it.

Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there.

From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?

I don't know if I'd outright call it a problem, per se, since in a couple facilities (say, Fremont) it's a good thing given how good the provider (say, Hurricane Electric) is at providing transit. So in the datacenters where the provider is better at transit than facilities (say, Fremont), the equation breaks in their favor.

When I left they also didn't really have the staff to run a proper NOC for a full-on AS; one extremely talented network engineer who has since left mostly owned all the "running a grown-up network" stuff, like running down an ASN, working RTBH with the facilities, well-designed uplink strategy at Linode's edge, and lobbying for anycast infrastructure for DNS and so on. They had just hired another purebred networking wonk as I moved on who probably owns it now, and likely has help. So punting a lot of the typical NOC work upstream to a facility with whom you're already working anyway made a bit of resource-related sense at the time, since staffing a NOC for a network of that scale is a significant challenge and they like to operate lean.

That's changing now (I didn't know this and learned it from agwa's comment), which is probably reflective of growth on the resource front.

Real talk: it's not that hard to run your own network. It sounds like Linode was skirting by on not making the infrastructure investment both in hardware and people. It's finally come back to bite them. No excuses and no pity for them.

Also, I wouldn't run around saying that having HE in house is a great thing.

That's correct. They use facility transit everywhere except their newest datacenters (Singapore and maybe Frankfurt), where they operate their own AS and use blended transit.

Just because they purchase transit doesn't mean they don't have their own AS.

Everyone has to purchase transit at some point. Transit != AS.

Where did I say that purchasing transit relates to an AS? I was speaking out them separately. "House bandwidth" in most colos is pretty garbage, and you don't have much control over it.

I'm a lot less impressed with Linode after this thread than I was before.

I didn't downvote, but I can tell you almost certainly it was because of the snark.

Attackers shift to a new target on the same net as IP's are null routed, until the entire network has null routed itself which achieves the same result as DDoS'ing the network. So I'm not sure I agree that removing target IP's from your network is a great solution.

I would hope it is obvious based on the extent and duration of the attacks that RTBH is not an option in this case.

Reply on news.ycombinator.com