The Twelve Days of Crisis – A Retrospective on Linode’s Holiday DDoS Attacks | Hacker News Reader

The Twelve Days of Crisis – A Retrospective on Linode’s Holiday DDoS Attacks

blog.linode.com

234 pointsalexforster145 comments

I strongly believe it's not possible to safely run a site without DDoS protection for all servers anymore. Anyone with $20 can take down anything on Digital Ocean, Linode, Hetzner, and many others. Or they can run up a huge bill for you on AWS. I would love to use Cloudflare but I can't afford $6000/mo for DDoS protection on my servers with the wildcard requirements we need. Linode may have solved their DDoS problems with their own stuff, but what about their customers' VPSes?

I really wish people would start taking DDoS more seriously. It's really not something we can just null route servers for anymore. It's becoming a very serious problem. It's not going away, it's amplifying and getting far worse.

I'm also not sure how effective it would be, but it would be nice to see the FBI, NSA or whomever spend at least as much time fighting these DDoS warlords as they did persecuting whistleblowers and trying to shove backdoors into cryptography.

Of all the DDoS protection solutions you could spend $6000/mo on CloudFlare is probably the worst choice, hell Voxility will give you better service for $600 a month (And on top of that, they don't give the NSA unfettered access to your all of your traffic).

Don't buy CloudFlare for just DDoS protection, their pricing and products are far from competitive.

our you can host your server where DDoS protection is included by default

OVH for example

https://www.ovh.com/us/news/articles/a1171.protection-anti-d...

You get what you pay for, in the case of OVH they'll do a decent job tanking volumetric attacks, but fall short for anything that doesn't involve tons of bandwidth.

Kinda...

I used OVH in the past and they rely on TCP resets/disconnects for large syn floods, making your servers pretty much unusable while being attacked.

I found it better to hide the server behind Sucuri or Incapsula if you are worried about ddos.

I think that effort would be better spent encouraging (see Forcing) ISPs to start dropping forged traffic at their borders.

IMO there should be significant penalties for network operators who do not drop obviously I forged traffic. How long has that rfc been around now and how little adoption has it seen?

>I think that effort would be better spent encouraging (see Forcing) ISPs to start dropping forged traffic at their borders.

The importance of spoofed traffic to attackers is greatly exaggerated, I could personally easily send 500+ Gbit (probably terabit) sized attacks by spending a couple of weeks building a router botnet. No need to spoof IPs and at that point diminishing returns would make amplification attacks useless. Not only that, but most amplified attacks are particularly inexpensive to filter.

>IMO there should be significant penalties for network operators who do not drop obviously I forged traffic. How long has that rfc been around now and how little adoption has it seen?

Who would penalize them? Why?

And I'm not entirely sure if you understand what RFCs are, that RFC (which hasn't even been around for very long) is - most other RFCs - completely meaningless.

In practice, most volumetric attacks have spoofed IPs- amplified UDP reflection attacks and even SYN floods with no amplification.

Getting rid of UDP amplification reflection attacks will get rid of 90% of volumetric attacks.

We had volumetric attacks every day much before reflection attacks became common, the biggest attacks these days aren't reflected but from router nets.

And you simply cannot solve IP spoofing without rebuilding the entire internet, not to mention the fact that it does have legitimate use cases.

Also, if IP spoofing is making filtering difficult for you then you're doing filtering wrong.

How big were the volumetric attacks that you saw that involved real IPs? The amplification factor is 1x for the real IPs. With NTP reflection and DNS reflection, you get 50x amplification, so 1Gbps botnet (trivial bandwidth) will cause a 50Gbps DDos (non-trivial bandwidth). This is why filtering is desirable.

I saw a 500Mpps SYN flood just last week, followed by about 500Gbps of UDP packets. All from real IPs.

Botnets have far surpassed amplification attacks at this point.

>This is why filtering is desirable.

Come up with a way to implement it that actually works and doesn't break legitimate use cases. spamsolutions.txt is starting to seem relevant here.

We work with Verisign that specializes in DDoS mitigations, they have state of the art scrubbing centers on four continents and they are leading mitigation provider to the banks and schools. They told me they almost never see anything above 300gig. You must be special.

Yes, I would imagine I have far more experience dealing with large attacks than verisign.

>they are leading mitigation provider to the banks and schools

We are careful to operate by that security RFC and have BGP filters for every customer to prevent the possibility of spoofed traffic and DDoS originating from my network. It really is the responsibility of the network operators go be diligent in this. TWC - Business Class Fiber

Take a look at Incapsula or Sucuri:

https://incapsula.com

https://sucuri.net/website-firewall/

Both a lot cheaper and do a great job protecting against ddos.

Not really. Incapsula is useless for DDoS mitigation unless you want to pay for their enterprise plans, starting at $15,600 a year and offers 4 Gbps of protection.

http://i.imgur.com/NUnQqlS.jpg

My site was DDoS'd for 2 Gbps, according to Incapsula's charts. Incapsula demanded me sign up for a $15.6k annual contract.

Cloudflare got back to me in 15 minutes and told me they will gladly host me on their $20/mo plan, which includes 'light DDoS protection'. 2 Gbps is light for cloudflare.

I'm curious, why not a "real" DDoS protection service? On Incapsula you'll be screwed if your other infra gets hit.

Also, for the love of god don't use sucuri. They've had a plenty of opportunities to fully ditch linode but apparently still haven't.

They don't use Linode for their DDoS service. I think only some of their blog is in there.

Of course they don't, but it gives a good idea about their approach to security.

If it's <3000 records you could just use the API to generate DNS entries, no need for wildcard.

Layer 7 attacks are the new hotness in DDoS. If you have a big enough botnet (either conventional botnet, or hijacked browsers), you can do them, and they're often quite effective.

Fundamentally, layer 3/4 are usually amplification. Those are still effective, and very efficient for the attacker, but they will someday (5y? 10y?) be blocked by closing up sources amplification. Address spoofing address at layer 3/4 might get addressed by BCP 38, Vixie's good fight, etc., but not holding my breath.

By the time all that happens, attackers will have moved on to layer 7 attacks. Those can target the weakest parts of your stack, and with a large botnet, even the act of blocking the IPs in the wrong place can add enough overhead to hurt. With a huge botnet of hijacked browsers, blocking everyone affected becomes a DoS vector in itself, since some of those are your own legitimate attacks.

The big problem for DDoS mitigation is that this requires much deeper knowledge of the protected application. It's hard to just put a box inline, or an unmodified cloud service, and have it block the attacks. There's both good science and great engineering to be done, by developers, platform vendors, and specialty anti-DDoS providers, to block this emerging kind of attack.

>Layer 7 attacks are the new hotness in DDoS

Maybe 10 years ago.

>Fundamentally, layer 3/4 are usually amplification. Those are still effective, and very efficient for the attacker, but they will someday (5y? 10y?) be blocked by closing up sources amplification. Address spoofing address at layer 3/4 might get addressed by BCP 38, Vixie's good fight, etc., but not holding my breath.

Sending "raw" UDP floods from bots still has several benefits over amplification attacks provided you can amass enough bandwidth, which isn't that difficult these days.

>By the time all that happens, attackers will have moved on to layer 7 attacks. Those can target the weakest parts of your stack, and with a large botnet, even the act of blocking the IPs in the wrong place can add enough overhead to hurt. With a huge botnet of hijacked browsers, blocking everyone affected becomes a DoS vector in itself, since some of those are your own legitimate attacks.

Unlikely, it'll take a fundamental change on how networks work for network layer attacks to become irrelevant. Especially considering volumetric attacks have the added benefit of potentially getting your target kicked off by their hosts and causing added damage in BW bills. And as internet connections become faster, DDoS attacks become bigger.

Browser JS insertion on HTTP is pretty novel; it was done with ad networks a few times, but never at the scale of the GitHub/Great Cannon attack. Using an existing botnet certainly was fine, but for most sites, blocking botnet IPs doesn't cause as much collateral damage as blocking compromised-browser IPs.

If you did a watering hole attack, doing JS injection on a really popular "show HN" post on HN, against HN, you'd be effective in getting HN to block the IPs of a large percentage of real users, which would hurt, even if HN could repel the attack entirely. Blocking 50k random botnet IPs wouldn't really affect many regular HN readers.

>Browser JS insertion on HTTP is pretty novel; it was done with ad networks a few times, but never at the scale of the GitHub/Great Cannon attack. Using an existing botnet certainly was fine, but for most sites, blocking botnet IPs doesn't cause as much collateral damage as blocking compromised-browser IPs.

But the "great cannon attack" was absolutely minuscule compared to the stuff that happens every day, attacks of similar sizes were already reasonably common years ago (See: http://i.imgur.com/0quYBdV.png a graph of an attack on reddit, from 2013).

Today we're talking about Mrps, not Krps.

>If you did a watering hole attack, doing JS injection on a really popular "show HN" post on HN, against HN, you'd be effective in getting HN to block the IPs of a large percentage of real users, which would hurt, even if HN could repel the attack entirely. Blocking 50k random botnet IPs wouldn't really affect many regular HN readers.

That'd just be sloppy filtering, there's no need to drop L7 attacks on IP basis.

I agree and IMO it's basically impossible to fully defend a dynamic site against a well-engineered L7 DDoS, short of operating at top-10-website scale and simply have more resources than the attacker(s). That does create a sort of "survival of the largest" effect which is a concern for smaller startups which anger or frighten a much larger competitor.

Far more concerning to me than this outage were the security incidents (https://news.ycombinator.com/item?id=10845278) that Linode seem to continually have once every year or so. The most recent one seems to have happened in July, but they didn't notify customers or reset passwords for another six (!) months. https://news.ycombinator.com/item?id=10845619

Wow, still a lot of people fighting over whether or not Linode is a good company. It's a shame we don't get to see how <hipster hosting company of the month> copes with 80gbps of DDoS on a single DC.

I'm personally happy with Linode. They have a seriously tough technical issue to deal with —as much working out what's happening as how to stop it— and they seem to be doing a fairly top job at staying afloat. My servers haven't gone down. Any downtime in the last four years has been my fault.

So even if they are targets of some ludicrously powerful botnet, I'd rather stay with them than let the bastards doing this win. The attack isn't hurting my business or my clients and each incident we go through, the lower the chances of it ever being a problem in the future.

On a more serious note, governments keep moaning on about encryption but botnets are still a much greater direct threat to national security.

Uh-oh, the attacks started again a few minutes ago:

http://status.linode.com/incidents/mkcgnmjmnnln

Which is why you do not blog about them and communicate quietly with your customers.

Not really as possible with a consumer scale web service, but you can certainly modify your communications strategy to be less of a red flag for the bull. :(

Where's the Linode founder(s) in all this, and why couldn't they have kept customers informed? It seems like a lone network engineer has been left to deal with a potentially company destroying event.

I was personally in the room, and in agreement, when running a real grown-up AS with carrier transit was proposed to Chris Aker as early as 2010, maybe 2009, to avoid this very scenario and many others like it. It's not really news. I have tremendous respect for the engineer who proposed it and fully believe he could have executed on this when Linode still had four facilities and 360 MB Linodes were the norm. I'm not saying that to toot my own horn (really, I'm not "I told you so" or arrogant like that), but there are very specific reasons that this wasn't done for as long as possible. I lack recent context, but the Linode decision-making culture was for many years completely driven by one individual who worked to spend as little on infrastructure as humanly possible.

Even once growth really took off and revenue started making these big shifts in strategy viable, the mindset was still to be lean and scrappy. The minimal capital expenditure strategy had benefits early on and allowed Linode to maintain an incredible margin and support explosive growth, but they were too slow to start thinking like a grown-up company when it started to matter, and it's coming back to bite both on security (with almost zero investment; just enough to pass PCI-DSS) and things like this.

When I heard they bought the Philadelphia building, for example, I was very surprised because that's not the Chris I knew. We lobbied for a Philadelphia office for years. Could be a good sign regarding decision-making culture for the future, but hard to say.

Don't read me as bad blood or anything, as I wish Linode no ill will (I actually hope they can turn this perceived slump around), it's just educational to see the consequences of choices and mindset catch up with a company. I learned a lot about management style while working there and contrasting with subsequent employers.

> just enough to pass PCI-DSS

nope, the only way Linode "pass" now is being below the self-assessment questionnaire threshold. Once they have to move to an actual external audit they are fucked.

> Our longest outage by far... can be directly attributed to frequent breakdowns in communication

I have direct experience with Linode staff breakdowns in communication because of a security problem before the December attacks.

The problem affected many Linode customers and included risks to confidential information such as billing.

The Linode staff communication was terrible. The problem was severe and ended up with Linode on a blacklist of companies that are not suitable for hosting.

I have to agree with tptacek: do not use Linode for anything, and if you do now, make plans to switch to a new provider.

To end on a happy note, I migrated the project to Rackspace, and the Rackspace staff communication is excellent.

I'd like to learn more about these blacklists so I can factor that in when choosing a vendor, do you have a link? Google is just showing me pages of vendors trying to sell me hosting when I search.

I think this is more of a mental blacklist.

As hard as this may be to believe, tptacek's opinions are not the gospel of technology. I've been with Linode for several years and their staff has been top notch.

>Layer 7 (“400 Bad Request”) attacks toward our public-facing websites

I really wonder what that is supposed to mean, Linode has mentioned it multiple times but not elaborated on what sort of an attack this is.

I personally haven't ever head of a "400 bad request"-attack.

Edit: Yeah, I know what Layer 7 floods are :), but I'm pretty sure "400 bad request" floods are something Linode came up with, so that could use some elaboration by them.

Perhaps they're saying: an attack on OSI Layer 7 (web apps themselves) that result in users receiving HTTP 400 status codes in response to their requests.

OSI Layer 7 would actually refer to all "applications", but the mention of "400 Bad Request" implies specifically web applications were affected.

https://www.trustwave.com/Resources/SpiderLabs-Blog/(Updated...

My guess is strategies like this but it isn't clear. [e.g. Requests designed to slow/increase the processing per-request to create a log jam at the web application level ]

I think it's just another name for Layer 7 DDOS. I.e. crafted HTML-requests, designed to be 'expensive' to compute/process.

Right, it seems pretty clear to me.

There's a lot of different types of Layer 7 attacks, what sort of a payload is a "400 bad request"?

Any HTTP request that causes the webserver to respond with HTTP status 400?

But that's not descriptive at all.

To me what it means is that the attacker figured out some custom call to the application that is probably expensive for the app to deal with and can easily cause a denial of service.

It'd be pretty rare for the application to return error 400, generally that's something that the server would be spitting out when it fails to parse the HTTP request.

That could suggest random urls. But it could be anything really depending on the app. I'm coming around to your pov that it is not descriptive. Something about the layer 7 flood was causing the app to respond with 400s and that's what Linode started calling it. But it doesn't help us understand anything about it.

It wouldn't suggest a flood of random urls, error 400 is generally a response from the webserver when it receives a request it can't understand (e.g not HTTP).

A request like that would never hit the web application as the server wouldn't know what to do with it.

see:

  echo ":P"|nc linode.com 80
  <html>
  <head><title>400 Bad Request</title></head>
  <body bgcolor="white">
  <center><h1>400 Bad Request</h1></center>
  <hr><center>nginx</center>
  </body>
  </html>

What exactly are you asking here? You seem to already have a solid understanding of what type of attack would generate HTTP 400.

Not really, there's a wide variety of different requests that would cause such an error.

Just flooding someone with ":P" shouldn't take them down.

I agree that it's not descriptive, but 400 isn't just for malformed HTTP – it covers malformed requests in general. So there's lots of opportunity for this to get past the basic HTTP processing of the web server and into the application code.

For instance, if they had a web service to reboot a server that takes a numerical ID for an argument, but somebody passed in something other than digits – it's valid HTTP, so it would get past the HTTP level and into the application code.

Presumably the attackers found areas that cause very expensive 400 failures for Linode at the application level. If I remember correctly, their web infrastructure is currently legacy Coldfusion and in the process of being rewritten. They might not have the agility/human resources to patch up these kinds of problems quickly, given the ongoing transition.

Seems counterproductive, even. I would expect a request that causes a 400 response to be rejected early, whereas a request with a valid payload would go through more processing and be more expensive.

I am pretty amazed Linode didn't have their own IP Transit up to this point. Their colo provider is Newark charges some pretty high prices from what i've seen.

My plan is to keep saying this on Linode threads, just in case there are people who have missed it. Take my advice or leave it:

Please don't use Linode. If you are using it now, make immediate plans to switch. If you have friends who have things built on Linode servers, tell them to switch.

Do you say this specifically because of these attacks, or is there something more?

Hey, Thomas. I crossed swords with you when I worked at Linode and we had a brief HN argument about side-channel attacks in Xen (remember that?). I gained respect for your opinion and experience from that, and I think you should blog your perspective on this if you haven't. I, personally, would be very interested in reading it, especially if you've done any kind of security analysis on Linode.

Could you plan on giving reasons why too?

What do you suggest as an alternative to Linode?

digitalocean.com

I'm using it right now, works great

I've been locked out of a droplet and had a friend lose one from a failure without warning. I've switched due to this and the attitude from customer service.

Was working fine for few years before this happened.

What do you use now?

Linode Singapore data centers. Lowest latency for any cloud provider where I live. Been rock solid so far. One time I used Cloudflare for two weeks and had massive service degradation at random times, so not doing that again.

Could you elaborate? We've been their happy customer for 4 years now and while this incident did hit us with several hours of downtime, we are definitively not gonna switch just because of this.

I think http://wptavern.com/linode-confirms-data-security-breach-tha... is a good reason to switch. Linode's security can't be trusted.

https://news.ycombinator.com/item?id=10845278

Yeah, unless you're willing to elaborate on the reasons behind this statement, it's hard to give it a lot of credence. In fact, your reputation is the only reason I'd give it any credence at all, without some explanation.

We've been happy Linode customers for a while now, and definitely prefer Linode to where we were before (Rackspace, via the Slicehost acquisition). I'm not opposed to moving to something like DigitalOcean or other but right now I'm seeing any compelling reason to make a move.

Surely something like this: https://blog.linode.com/2016/01/05/security-notification-and... doesn't inspire confidence?

Sure, but who hasn't been hacked at some point in time? The @tptacek post has an air of suggesting Something Is Really Wrong, like "Linode is a front of the NSA" or something.

Would it be that much harder to say "Don't use Linode, they have a bad history security-wise and just got hacked again"?

Linode has not only been hacked countless times. So many times in fact that it's hard to blame it on just incompetence, but on gross negligence on their part.

Then there's the whole lying to cover up hacks, not investigating clear compromises when reported by customers and generally just avoiding any kind of negative press at all costs.

I know for a fact ~7 providers around the same price range have had the exact same problems and were less honest about it. :p

Hell, I know some in 2016 that are plain text offenders still.

You get what you pay for, and unless you pay about double what Linode charges...you don't get much in the way of added security or protection.

You should name them.

I think you can figure out why I can't pretty easily.

Just so I understand: his claim, which doesn't constitute advice of any kind and doesn't ask for any action on the part of the reader, requires substantiation? But your claim, which does have a suggested course of action for the reader, does not require further elucidation?

I do know who you are and I do have respect for your contributions, but my friend, it's seems like you're flaunting a sense of superiority when you issue a drive-by decree. It's only made worse when you choose to take the time you could have spent helping the readers whose opinions you're hoping to sway, and piss it away with grammar/spelling corrections and sarcasm. We routinely call out people in other public forums who give blatant non-answers, don't we?

I thought your comment was helpful. Until you decided that only those who would take it on faith or on fallible googling would be worthy of your help.

>Just so I understand: his claim, which doesn't constitute advice of any kind and doesn't ask for any action on the part of the reader, requires substantiation? But your claim, which does have a suggested course of action for the reader, does not require further elucidation?

I'm not sure if tptacek implied the claim requires substantiation, but IMO naming and shaming is just the ethical thing to do when someone is covering up hacks.

Fair enough. Tptacek named. "Shaming" strongly implies stating the actual flaw.

FWIW, I am aware of the past issues Linode has had, and hope they do get their act together.

Yes, but not if you found out such from confidential information you agreed to keep confidential.

But in all seriousness, I don't know a single provider that sells cheap VPSs [e.g. At Linode's price point] and actually has something resembling security.

You keep talking about how Linode is cheap, but when I look on their site their prices are absolutely insane. Am I missing something?

>Yes, but not if you found out such from confidential information you agreed to keep confidential.

Debatable.

> You keep talking about how Linode is cheap, but when I look on their site their prices are absolutely insane. Am I missing something?

Its roughly the same as DO, etc. Yes, its more expensive than buying a dedicated server.

I define anything cheaper than AWS/Google Cloud/Azure/etc. as "cheap". I am assuming you are using a reasonable level of bandwidth [e.g. 100GB+] when making such statements.

Really the only places you can find even cheaper VMs is sites like lowendtalk.com and you really don't want to go that cheap if you can avoid it.

> Debatable

Let me put it this way, "contractually obligated to".

@tptacek is some dude on Twitter that spends a lot more time thinking about his abs than I do. I'm @tqbf.

You could've googled "linode", but here you go:

https://news.ycombinator.com/item?id=10845278

Meh... nothing about that is in the first page of Google results when I google Linode. I wasn't aware of the older hack incidents, because we weren't Linode customers then and so there was no particular reason to pay attention to Linode news.

Anyway, if anyone is going to make a blanket statement like "Don't use X", there's no harm in providing at least a high level explanation of why you're saying that. One shouldn't assume other people keep up with the exact same news that they do.

Their wikipedia page, the 3rd google result for me, should provide enough info by itself.

I do agree that some level of explanation would definitely be useful, but this comes up so often that I really don't find it surprising that people are too lazy to explain such statements.

There are times I am comfortable talking publicly about things and contributing context or color commentary.

There are times I am comfortable broadcasting clear, direct advice that might hurt some entity's feelings.

There aren't many times when I am comfortable doing both at the same time.

So you're not comfortable saying:

> Linode has had enough security issues that I wouldn't trust them with my servers.

I am comfortable saying "Don't use Linode."

For all those who want some sort of backup: There is a search function here. If you search for Linode, you'll find a long string of security blunders. You really won't have to try hard. (PS. I'm a current Linode customer)

Just a comment on my experience: the last time one of these threads appeared (after reading several prior), I followed tptacek's advice and shut down the few remaining Linode instances I had. Linode handled it professionally: I removed my nodes, canceled my account online in a couple clicks, and requested a refund of the credit on my account (I had pre-paid for a year, and Linode refunded the time remaining). The payment was issued quickly and without confrontation (though they did ask, by email, for the last six digits of my credit card for "verification") I then deleted my account.

All told, it took a few minutes one evening.

My plan is to keep saying this on Starfighter threads, just in case there are people who have missed it. Take my advice or leave it: Please don't use Starfighter. If you are using it now, make immediate plans to switch. If you have friends who have used Starfighter, tell them to switch.

Feel how dickish this sounds without giving any reason whatsoever?

Please explain why.

There's a pretty clear difference here.

Saying that would make you a dick.

The linode comment doesn't make tptacek a dick.

Why?

Because it's trivial to figure out why he's saying that by simply typing "linode" into google, but as of right now googling starfighter doesn't immediately bring up any reasons to avoid them.

Edit: Didn't mean to imply that parent was a dick.

It's dickish saying that without explanation. Like this it sounds like baseless smearing.

If he argumented the reasons before he could had just put a link. Sincerely, it's too much to ask?

(1) "Argumented" isn't a word.

(2) I'm comfortable with the fact that my suggestion sounded "dickish" to you, whatever that means. It is meant seriously, though.

To be honest, the number of "linode screwed up" posts on hacker news the last few years would be educational to you, and if I remember correctly, ryanlol even got a slap on the wrist due to one of those situations.

At this point, I am bored of people asking for citations on hacker news for things that are should be part of our tribal knowledge.

https://www.google.com/search?q=linode+hacks&ie=utf-8&oe=utf... About 2,810 results (0.34 seconds)

Tribal is the right word here with all the blind faith in medicine men and cargo cultism.

I meant it in the way of shared knowledge, just like we all know how to bypass a NYT filter, or that someone is going to complain about the lack of native scrolling in an article, especially on a Show HN.

I definitely agree that there is a huge amount of that type of thinking on HN (of course), reading the amount of people who used github but didnt know the different between it and git and were commenting today was a personal education.

I changed "linode" to "aws" in your link, very interesting results came up in Google.

Notable is the lack of any mentions of actual instances of AWS getting hacked.

There was a link, you just missed it. It's labeled "tptacek" and links to this page: https://news.ycombinator.com/user?id=tptacek

Of course, you can judge it differently, but following that link convinced me that the claims are probably not "baseless smearing", that it's a well-intentioned advice. Just from the link itself I wouldn't know on what grounds tptacek came to his conclusion, and I wouldn't heed his advice without further research, BUT I'm 99% sure that if I researched I'd find many well-documented arguments in favour of tptacek opinion/advice. I'd even bet on this: you say it's baseless, I say I can easily find the reasoning and arguments behind what tptacek said. Want to bet?

Oh, by the way:

> Sincerely, it's too much to ask?

Let's turn it around: a person with a lot of experience offers an advice on the matter he's experienced with. Is it too much to ask the readers to first, at least, google a bit before commenting? Why do you think you are entitled to receive even more of that person's attention and time?

I don't think it's appropriate to throw around insults and claim jfgi when someone asks for a mordicum of clarification.

If you can't even manage one small teensy link, what respect does that show for your audience?

The only value comes from the ptacek brand, and the trust I have in him through context. But that's generally not a strong foundation to build an argument on.

To be clear: I trust him, but it's not his best post. And op was not out of line for speaking up.

Edit: sorry ryanlol I didnt mean you, but tptacek. It was perhaps too strongly worded. Didn't mean to attack anyone, just wanted to show support for yomism's point.

I consider myself a HN regular, but I dont read everything. Linode posts I subconsciously skip, as they don't interest me. This was honestly news to me. "Incomplete information", I believe that's called ;)

(1) "Mordicum" isn't a word in English, though it should be, and it is in Latin.

(2) Nobody is entitled to detailed comments from anyone on HN, and keeping comments terse simply isn't disrespectful.

I appreciate that it is annoying to have to make decisions with incomplete information, but that's life.

I'm not sure what role spelling critique has in a conversation about technology providers, but to your second point I do think it's reasonable to expect that in public conversations claims be substantiated to a degree relative to their contentiousness. No one expects evidence or citations when you claim that good security is hard, but it's not so crazy to hope for even a respected member of the community to back up a stark claim. At the very least it's reasonable to ask for substantiation - whether or not the original commenter is comfortable discussing details.

It's not a "claim". It's free advice. Take it or don't take it.

Good advices are rarely given, but sold...

I was going to apologize for using "dickish" in my answer before but seeing this you fit the definition perfectly ;-P

http://www.urbandictionary.com/define.php?term=dickish

I have read the horror stories thanks to ryanlol's posts but next time please post a link if you don't want to waste time re-explaining. Let's use the HTML powers!

Who's throwing around insults? I specifically edited my comment to make sure it's not misinterpreted.

There's links elsewhere on this thread and Linodes security fuckups are a recurring subject of discussion on HN

> Because it's trivial to figure out why he's saying that by simply typing "linode" into google, but as of right now googling starfighter doesn't immediately bring up any reasons to avoid them.

This doesn't mean anything. I could get horror stories about any cloud service provider via Google yet we're only being told not to use Linode. Providing some context makes all the difference.

For the love of God don't host your apps on Starfighter.

But think of all the efficiency & security by obscurity I would gain by implementing my apps in AVR assembly! Plus it'd be hosted in the cloud.

Weird machines as a service?

For whatever it's worth, I decided to get linode oh so many years ago because hdm, the security guy, very warmly recommended it to me. Indeed, a lot of very high profile security folks love linode (nmap.org/sectools.org is a good example).

Anyway, personally, I choose to still stick to Linode because their customer support is extraordinarily good. I'm speaking about my experiences in the last 5 years.

Concerning their handling of ddos attacks - I think with this changes made things should be much better.

I might have recommended Linode "oh so many years ago" too.

I think you should ask HD and Fyodor again what they think about Linode in 2016.

The fact that nmap/sectools is still in fact on Linode, right now, the fact that moxie still uses it, as well as jacob and other superstars are using it in 2016 isn't a testament to what they think? And, I think hdm still uses linode for a lot of things. So do many other security folks I hang out with.

I really think that if it was some other VPS, they could not have done much better. You remember the outages that Amazon had? It's just a matter of fact the way I see it, these attacks happen. We learn from it, resilience is built. Until a new type of attack takes place, and then the process repeats. I understand that uptime can't be 100% all the time -- the 1 or 2 days it was down in 2015 was an inconvenience, but not totally unacceptable. I also understand that if you're against very determined attackers, it's pretty tough. How will any of the other VPSs fare when the attacker happens to have an 0day or something?

By the way, I noticed a few years ago that bitcoin-related startups were likely to use Linode. That makes linode a huuuuuuge target. I really don't think that if it was some other VPS in the crosshairs, these determined attackers could have been stopped 3 or 4 years ago with the ferocity and resourcefulness they seemed to be equipped with.

> I also understand that if you're against very determined attackers, it's pretty tough.

You know what makes it even tougher? Using COLDFUSION in 2016.

I ask myself and myself says Linode is the best of all I've tried, which is many. So thanks for your wonderful insights but mine are better for me.

What are we supposed to use? How do I know the host I pick has any better security practices or that they just haven't been owned so far? Which other VPS providers have essential features like private networking?

Please don't say AWS, I have no interest in learning that overcomplicated mess.

Stop using VPSes, start using dedicated servers.

Can you be more specific? Who?

No offense, but while that is your view, many low end providers do the same thing.

Every company around that price point has these problems [and if they don't, they are either burning VC money or lying].

Nah honey, I'm good and Linode is the best.

Slightly related question:

They mention segregating their customers into separate /24s, and consequently having to assign an IP from every one of these subnets to the router for use by the customer as a gateway.

Is there any reason why they couldn't get rid of these by having customers set up a static route to the "primary" IP of the router (migration / configuration issues aside)?

The static route has to be via a gateway that is on a locally connected subnet.

You can have a static route to any local device, this is essentially what subnet membership does.

For example, if I have IP 192.168.0.2/24 assigned to eth0, my routing table will have:

  192.168.0.2/24 dev eth0 proto static scope link

I'm free to add a local route to a device outside 192.168.0.2/24 though:

  192.168.10.1 dev eth0 proto static scope link

This just indicates that I should be able to resolve the MAC address associated with 192.168.10.1 through an ARP query, same as other devices on my subnet.

We are currently getting DDoSed at Hetzner and they are clueless as well.

If you're planning on getting DDoSd you probably should pick a provider that offers DDoS protection.

Not sure why you'd choose a budget provider for production infra anyway.

> If you're planning on getting DDoSd

This, don't do it, far better to not get DDoSd ;-)

Surely any org that depends on internet presence would plan against DDoS attacks. That's like internet 101.

No guess at motive? Did someone ask for ransom before these started? Is one of the Linode subscribers hosting censorship-evasion technologies? Or is this one just some very determined kids having fun over holiday break?

Could it have been Bitcoin related? The timing seems to roughly match up with the Bitcoin XT DDoS attacks mentioned here: https://medium.com/@octskyward/the-resolution-of-the-bitcoin...

> Our nameservers are now protected by Cloudflare

How? I thought CloudFlare only protected HTTP? Can you have it reverse proxy a DNS server or is Linode using CloudFlare as the host for ns1.linode.com now?

They provide white-labeled protection for DNS too, if you're a big service provider.

Yeah, it's called Virtual DNS (vDNS); essentially a DNS application proxy.

(email me if you want more info; it's not really ideal for small sites, it's better to just use cf for hosted DNS then, since it's free, but we're happy to do vDNS for people who can't do hosted. Mainly providers, but also some enterprise customers with special DNS needs. It's a pretty cool technology.)

https://www.cloudflare.com/virtual-dns/

> after some stubborn transit providers finally acknowledged that their infrastructure was under attack and successfully put measures in place to stop the attacks.

Care to elaborate why it took them so long to ack? And name them so I know who to avoid in the future (or route around)!

>he pervasiveness of these types of attacks has caused hundreds of billions of dollars in economic loss globally.

Is it really $100bn+ ? If so we could do with some government funded research / countermeasures.

Thread on IPFS/DDoS: https://news.ycombinator.com/item?id=10329195

> blackholing is a blunt but crucial weapon in our arsenal, giving us the ability to ‘cut off a finger to save the hand’ – that is, to sacrifice the customer who is being attacked in order to keep the others online

There is something very ironic about this. They have a policy which instead of addressing the problem actively assists anyone wanting to attack their customers. No surprise that these customers have been complaining about this practice for a long time. But until now it was Somebody Else's Problem so they didn't bother figuring out some proper (or at least less terrible) solution. Now this lack of preparedness bit them in the ass...

I'd posit that 98% of providers from whom you can acquire budget VPS will do the same thing. The practice is not unique to Linode; why should a network you're paying $20-$100 do everything they can to keep a target online and threaten other customers?

Contrary to popular opinion, if you're getting DoS attacked, you're either (a) popular enough to start thinking about adult-size pants for your transit strategy or (b) inviting the attention by your choice of content or activities. In years of hosting, I started to know the targets of DoS attacks by name. You have to own at least a little bit of responsibility, and mitigate on your own end if you're going to be inviting that kind of attention; IRC and controversial blogs are the usual suspects here, but that's probably changed recently as I've been out of the hosting game for a while.

Linode has few options for reacting other than the one they use. I know that sucks, but it's how it is.

So, if you're big like you say, you're paying more than $20-100 a month. That's the very absolute entry level price point there.

Yes, customers of other budget VPS providers are complaining about this too.

> why should a network you're paying $20-$100 do everything they can to keep a target online and threaten other customers?

I am not a network engineer and I know that this is a very difficult problem. But when the provider doesn't even _seem_ to try, it only encourages further attacks.

OVH has their DDoS system sitting in front of their entire network. VPS and grown-up physical servers...

Does the buck stop with this network admin? Where's the CEO?

Riding his motorcycle across Europe, occasionally sharing photos from vacation with his plebeian workers in #linode-staff who are earning $38k and can't afford to take any vacation.

I wonder what will happen if Linode routes their traffic through CloudFlare...

It wouldn't work. That's not what CloudFlare does (right? they didn't do BGP last I heard). You'd need something like Black Lotus, now owned by Level3, for that.

You're technically correct as of Fri Jan 29 15:48:27 PST 2016.

(email me if you'd like more detail; you seem smart/interesting but don't have email in your hn profile)

Transparent scrubbing would be awesome, have CloudFlare advertise my networks, then CloudFlare on the backend sends the clean traffic to me...

Can do it with peering with CloudFlare directly, or even if that is not possible MPLS on the backend to direct the traffic to where it needs to go.

Reply on news.ycombinator.com