Linode DDoS continues – Atlanta down for 16+ hours
status.linode.com
status.linode.com
This is nothing less than cyber-terrorism, and I hope the FBI is involved.
A problem I see with people throwing Linode under the bus is that only the mega hosting providers (AWS/Google) have the resources to mitigate such attacks. I wound hope that the industry can find a solution out there that allows smaller players (Linode, Digital Ocean, etc) to run a hosting business without the threat of DDoS.
If everyone moves to the big players, it is a loss for us all: feature-wise, quality-wise, security-wise, cost-wise.
I just signed up for a Linode server. Going to use it for part of our offsite backup storage and to monitor some nodes on our primary network.
Huzzah...
AWS and Google have resources in place for their own benefit, and hosting with them means you get these benefits to some degree. But the issue is that this complicated infrastructure is only reachable by giants(amazon, google, rackspace, etc). AWS probably spends enough money, collectively, on DDoS mitigation and fallback that it could acquire another hosting company for that sum of money.
The to consider the shady practices of who ever is trying to sabotage Linode's business, the likely goal is to direct business elsewhere. So you may end up moving to a hosting company who is shady enough to have caused you a loss of profits for their own profits. Somethings up here, and I'm really curious to find out who did this. I'm not leaving a business because they were a victim of espionage.
The Main Twitter account still hasn't announced the Atlanta DoS. https://twitter.com/linode?lang=en
I understand the severity of the issue is different to Slacks', but there should be a bunch of people on the Twitter account replying to people and saving customers. A lot of their customers are angry about lack of communication as much as they are about the downtime.
It's basically the same with the status updates, I have to "poll" for status updates rather than to be notified. They have this mobile app (which is pretty useless as it currently sits) which would be a great platform to receive push notifications on, even if it was to let you know that there is a change on the status page.
There's certainly some missed opportunities here, but then again, I assume they're pretty busy at the moment...
No VPS provider ever has sent out emails. The only hosting company I ever left was DigitalOcean, because they kept having really silly problems and at the time NY1 was the only datacenter with private networking, which is ridiculously silly to think that your infrastructure communicates over the internet. And the private networking was "experimental" for the longest time, and very limited.
I don't think Linode did anything wrong. And moving to another host, I garuntee you that they will not do this either. Chances are if you move hosts you may feed the shitty business who performed this DDoS.
Personally, I use Uptimerobot[3] to monitor my nodes and post any issues into a Slack channel. This lets me know if it's a problem with my VPS or Linode (or any of the other providers I use).
[1]https://ifttt.com/connect/feed/twitter
By making customers aware of the downtime they will have more customers ask for credit. If you suffer 1-2days of downtime and your customer doesn't notice to ask for credit; you win.
It's terrible; but unfortunately cost savings is drilled in from every darned angle.
They should have done a mail out.
Twitter is not that relevant to be honest. I assume many customers are not active on Twitter either. I do not see why it is so important to have activity on the account when there is a dedicated page for informing customers of what is going on.
Also, if you look at the medium post, the tweets are mostly sorry/thank you/everything should be fine soon. No actual information on what is going on. How could you put relevant technical details in a tweet? How does a short, uninformative tweet help you more?
edit: also, there is up-to-date technical discussions taking place on the irc channel #linode on oftc
> How does a short uninformative tweet help you more?
The point I've been trying to get across is that a lot of people just want to have their frustrations acknowledged. An honest, real reply from a human (not a bot like lots of airlines use) can make people feel heard and do a lot to restore goodwill.
They can also link to the status page in replies for more technical details.
They were able to mitigate the attacks quickly, but their bandwidth was saturated.
My comment was with regards to their inability to communicate with customers about on-going attacks and what was being done to mitigate them. There has been (after nearing 2 weeks of DDoS) just one statement on the status page by a network engineer. No company statement, no information on the measures taken, no e-mails, no support tickets. Nada. We have customers in the Middle East that are geo-blocked from accessing nodes in London. We had to find this ourselves. [1]
Furthermore their billing system didn't account for all the downtime. Looking at our latest statement, all instances have been at 100% uptime.
[1] http://www.theregister.co.uk/2016/01/04/linode_back_at_last_...
Not sure whatever happened to the post about "Less negativity" a few months back.
I've had pretty good communication with Linode and I've always had really good customer service. I don't think Linode is nearly as verbose about their internal business affairs and going-ons as the startup world seems to be, but I think competition has driven many companies to do the same, and the rapid evolution in this market makes it a reasonable effort to constantly keep the public in the loop.
As far as my experience, they've never with held information that I needed to know. But I understand why people are suspicious, and I think this suspicion is paranoia. These attacks are clearly espionage, and if they knew who was doing it, they'd have taken legal action by now.
It would be very hard to justify going back to Linode afterwards, even with the best intentions to do so. "... So you seriously want us to go back to this hosting provider that caused us all this mess over Christmas / New Year's??"
However, what are the alternatives? Linode have been dead stable for me the past many years, and delivers what they promise in a transparent way. No overselling of servers. No sudden extra bills.
Linode will come out stronger after this, so it won't be able to happen on this scale again.
The big question is: Who, with a lot of money, would want to hurt Linode's business in this way? This isn't just a "script kiddie" having fun. It's a very well planned and powerful attack requiring buying large botnet capacity for an extensive amount of time.
But the vps market is fairly crowded, I can't imagine what attacking a single competitor would actually accomplish.
I don't know... unless Linode has a large presense in a country where this level of play is expected. I have hard time imagining a large US cloud / hosting company comparible in size or customer base with Linode trying it. There is too high a risk, someone will talkt and reveal it.
BF Skinner, famous behavioral psychologist, wrote about this in "Beyond Freedom and Dignity" -- that much of positive emotions coming from choosing to do the morally "righteous" action exist because of the capacity for evil.
Yes I am :)
It would be easy to engage in morally bad behaviour, yet for the most part we don’t. This is a very good.
Github was attacked by China before I believe for hosting some firewall by-passing software.
Maybe a competitor, but who would bother to go that low? I can't imagine any of the popular ones trying it -- too much risk it will be exposed.
These days such things aren't that hard to come by. See the recent Lizard Squad attacks for example.
Serious question: how much is "a lot of money' in this case? How much does an attack like this cost?
From what they have said however, it seems to me like this is generally a very "smart" attack.
Unfortunately, a smart attack can be done much more cheaply in terms of botnet capacity than a dumb volumetric attack (this still combines a volumetric attack, but, in targeted ways).
I have delt with these kinds of attacks (defending them) and they are, unfortunately, much too cheap to pull off. Botnet capacity is dirt cheap, if you're willing to spend the time to find/compromise your own botnet hosts, its effectively free.
Given the length of time since the first attacks started and the downtime of the most recent attack, the big questions I have are:
1. What are they doing right now?
2. What mistakes were made / Why didn't they realise the risk beforehand? / (how) could this have been avoided?
3. What can they do in future to prevent this?
No notification from them, just a handful of downtime alerts during time with the family. They were completely gone from BGP tables in Newark.
Used backups and moved sites to OVH. Don't know who they pissed off, I suspect another NJ competitor, who is known for taking cheap shots at other VPS companies.
It's a pain in the ass, but at the same time, how is their network so fragile? You would think at least some of the fragile systems being attacked would be firewalled or at least ACL'd off from the public net.
This is what happens when you don't run your own network and rely on other ASN's and uplinks to do the work for you. When it comes to other customers being affected, they will simply null you. Unlike your network ops who would be trying anything they could from OOB to rectify such.
We are also duplicating in OVH, read good things about their built-in DDoS protection on HN.
Linode clearly wins on simplicity and clarity. I guess under the current circumstances, I'd be willing to compromise simplicity for better availability though.
Linode wins on simplicity, agreed. We are in the same boat, OVH has too many offerings. We are looking at their VPS SSD plans [1]. Last thing we want is to be offline again. As such we are also looking for anti-DDoS which is included in the plans. I intent on spinning up a few nodes to try them out first.
OVH has 3 large datacenter PoPs to absorb attacks and do just that, then push the traffic clean back to your server.
They may blow at support and response times, but once I have a dedicated server from them, their Manager is intuitive enough to get going.
Add the fact I can get 64G server on a brand new E5 chassis with 255 free IPs for VPS of my own, and I've been moving more and more sites there as hosts get arbitrarily hit.
Piss off some competitor or skiddie and you get tested. It's ridiculous, but sadly DDoS mitigation is becoming a must.
Good time to leave being a SysAdmin in cloud and go back to web design full time as I watch a lack of best practices and SPOF take over.
Finally, I backup everything to 2 off-site locations and hope for the best.
They have 3 data centers in France and one in Canada, east coast. Status and network map at https://www.ovh.co.uk/community/status/
About the options, I don't know. They updated their products on Q4 2015 and I still didn't have to buy another VPS from them so I didn't investigate.
I mentioned FastNetMon to them, but I just read the status update. They're blocking entire continents by communities... Holy shit. This is not some skiddie, this is likely state sponsored or BTC ransom.
Worrisome how the attacker knows so much of their infrastructure, makes me think ex-employee as he knows where to hit their servers, etc.
So glad I replied 'nope' to taking the cheaper SysAdmin position, after hand feeding them how I did mitigation. They asked me how, and were very interested in why. This was a week before this happened.
It's all making sense now. But even FastNetMon couldn't help this, you need a shitload of bandwidth (OVH size) and thousands (hundreds in cases for arbor) of equipment to match.
They need to GRE their /24's from Voxility or some large ass provider, as this is beyond fucked. I just read the status, they're cutting off parts of the internet to VMs. What in actual fuck.
I've worked in cloud for 10 years, and recently left, and will not be going back. Bare metal and OVH FTW. I can understand the 'going above and beyond' during holidays, but the lawyers I work for just want their 'f email online NOW' (direct quote)
Offloading the responsibility for continuation of your business to Linode (or any other data center provider) is unfair. A history of uptime, verbal promises, or fancy SLA terms should never be interpreted to mean that disasters won't happen. A ten day long DDOS is a disaster and in this case a man-made disaster.
Using Linode (or another provider) instead of building your own data center is more cost effective, but it means you are no longer in direct control of your infrastructure (decreased costs, increased risk).
Designing your application to span multiple availability zones (data centers) can mitigate single points of failure within a single vendor but is more expensive than operating in a single zone.
Designing your application to span multiple vendors can mitigate single vendor failures (or changes in offerings from a single vendor) but is even more expensive.
And still there are ways to mitigate these costs, business interruption insurance can help cover the costs for moving to a new data center or vendor in case of a disaster (such as hiring staff, overtime, etc.). Lost profits can be covered by business interruption insurance also.
Of course it is expensive to operate any business in a hostile environment. A seaside restaurant better be prepared to weather a hurricane. I wonder how much money has been spent on security cameras, guards, metal detectors, and so on since 9/11? The increasing occurrence of targeted DDOS (and other types) of attacks is the physical equivalent of an increasingly hostile environment and is going to be associated with higher costs.
In the longer term, I think we need to find ways to get law enforcement better suited to deal with these problems but ultimately I think we'll need to radically change the way we handle network operations and the technical foundations of the network such as content centric networking (https://en.wikipedia.org/wiki/Content_centric_networking)
Is anyone currently using libcloud or equiv. and able to share details?
The analogy for me is one of roads. If you block a road on purpose then an ambulance might not be able to reach an accident victim in time. The internet is infrastructure, just like roads and purposefully obstructing it wholesale is doing damage to a large number of parties.
What is sad is that these people get away with this stuff over and over again, it is very rare for DDoS organizers to be caught, rarer still (if it even ever happened) for them to be sentenced.
Is it because of only 38 upvotes vs 43 comments? The story is just 3 hours old.
@HN / dang: What's going on with HN sorting algorithm? The #3 story on HN is 1 hour old and has just 9 upvotes and 2 comments: "Churchill and His Money, or Lack of It": https://news.ycombinator.com/item?id=10825575
Screenshot: http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png
Couldn't find it on the 2nd or 3rd page, very odd.
Btw you should email hn@ycombinator.com instead of asking us questions here. It's pretty random whether or not we see the latter.
My take is that it's somewhere in the middle. This type of outage has in the past, and will potentially in future, hit providers at all price points regardless of their SLAs and guarantees.
In my experience (and other comments indicate) Linode are genearlly very reliable. This isn't a mickey mouse, dirt-cheap VPS operation. The communication hasn't been awesome but by keeping an eye on status.linode.com I personally have felt reasonably well informed. If you trust they're working on it and doing the best they can.
Other commenters suggest this is a fairly sophisticated, deliberate and sustained attack. They're putting significant resources toward it. It also seems whoever is behind it has potentially gained inside knowledge of their network topology.
Based that understand, I'm not wasting my time calling them or sitting around hitting F5. I'm working to improve my systems' architecture for resiliency in this type of situation. That involves geo-distributed, multi-site redundancy and fail-over.
My advice: be optimistic and proactive.
I'm asking sincerely. I wouldn't be able to sleep at night if I had a SaaS product with thousands of customers and no way to restore service if my primary provider went down for an extended period of time.
There are SLA's available from hosts that will give the whole month ,1/4 or even year back if they miss even a single month's 99.99%
This is past a joke now this whole period of downtime but still Linode is not a big dollar hosting outfit, need a plan B for when a region or all of Linode goes down.