Linode is down again | Hacker News Reader

Linode is down again

status.linode.com

30 pointsraychancc35 comments

Remember guys & gals: this can happen to you. I am a fanatical Linode customer, and they are, in general, an extremely rock solid service, but that doesn't alleviate my responsibility to ensure my application is available regardless of datacenter outage. Power outages happen to EVERY datacenter. There are some very well documented and understood ways to alleviate this problem.

Care to enumerate a few of them?

More than one server across multiple datacenters is the only way. The use either a load balancer (which brings its own problems) or some kind of status aware DNS scheme to route traffic to live servers.

I wish linode would offer some kind of high availability load balancing like amazon ec2 does, that really is a killer feature.

I wouldn't be so sure that ELB's are actually failing whole IP addresses across availability zones. Given the current ELB architecture is heavily dependent on dynamic DNS resolution, I am pretty suspicious that they are using DNS for failover. The separate set of DNS servers specifically for elb.amazonaws.com and the 60 second TTLs on the CNAMEs they generate seem to indicate this. Of course, there is no documentation to either confirm, or deny this. Amazon simply states in vague terms that it's highly available, but doesn't state how it's designed, or even possible failure outcomes.

Hmmm. Interesting point. However, if that were the case, why would ELBs be restricted to single AWS regions?

Anyway, its a nice feature that it just works without me needing to set it up.

Simple: distribute your application across two isolated datacenters. I'm not trying to downplay the complexity of doing this, as I fully understand the amount of effort required. If the cost of being down for an hour or two isn't significant enough to warrant this level of effort, then a power outage caused by Linode's datacenter vendor probably isn't going to really cost you much either.

DNS

Please change the title to indicate this is just the Fremont datacenter. The current title needlessly scares those of us with linodes in other datacenters.

And as luck would have it this happened right after one of my biggest clients sent out a bunch of holiday coupon codes to all of their customers. Do. Not. Want.

Are there clever solutions for routing traffic to other linode datacenters during one of these outages? I'm moving to AWS if I can't do something akin to elastic ip across datacenters. Linode currently only allows you to swap ip addresses within a dc.

use a DNS roundrobin with low TTLs and use monitoring to remove IPs when they go down. if you have 3 hosts in your production app's roundrobin, that makes it much less likely that they'll hit the dead server and if the TTL is 300, then it's only a few minute window anyway.

That's material for a great blog post. Please write an article wherever you can and share!

Visual Website Optimizer setup is exactly like this. We documented it here http://visualwebsiteoptimizer.com/split-testing-blog/maximum...

Waiting for the mandatory "Goodbye Linode" post. </sarcasm>

Can anyone recommend good documentation on load balancing a LAMP app across multiple independent VPS providers?

We've had a high profile site affected by these Fremont outages so I'm thinking the only real solution is to have VPSs at Linode, Slicehost and EC2 then load balance among them. Of course, then the question is, what do you do when the load balancer goes down?

This is why Linode is better than many other services I've used. Every data center has outages, but at least Linode makes a point to update the server status promptly and (it seems) implement the fixes as quickly as possible.

rbranson is right: the responsibility for redundancy lies with you.

Other discussion: http://news.ycombinator.com/item?id=1934013

My London node was down briefly two days ago .. same happened about a month ago - is this a reason to worry?

From status page: Initial indications are pointing towards another power outage. We are now working on bringing Linodes up as soon as possible. We will continue to post updates as we have more information.

http://status.linode.com

My Linode slice is in Atlanta, not Freemont, and it went down once today and once earlier this week.

I don't think that was the entire Atlanta data center, for what it's worth. Mine's in Atlanta too and I haven't had any recent downtime.

And to think how much time I spent last month recommending these guys, especially the Facebook-friendly Fremont data center. Awkkkk-ward.

Why? Because there are other services with no downtime?

Even Amazon and Rackspace go down...

Even Google has had outages...

This is really annoying... Down 2 times in 3 days!

Have they broken your SLA?

Linode has a 99.99% uptime guarantee, which amounts to 43 minutes and 12 seconds offline in a 30 day month. While I don't have a Linode in Freemont, their status page appears to indicate, at minimum, 1 hour of downtime. As repayment for the inconvenience, users get the lost time added to their balance.

Actually their SLA is only 99.9% uptime. Thats 525 minutes.

A 99.99% uptime guarantee would be 4.32 minutes per month. You calculated for a 99.9% uptime guarantee.

I'm going to take a long break from ever doing math again on HN.

With 99.99% uptime:

.0001 * 30 * 24 * 60 = 4.32 minutes

With 99.9% uptime:

.001 * 30 * 24 * 60 = 43.2 minutes

I think he just typo'd the %. From the terms[1]:

    Linode.com provides a 99.9% uptime guarantee on all Linode hardware,
    and on network connectivity. In any given month, if your Linode is
    down for more than 0.1%, you may request a pro-rated credit for the
    down-time.

[1]: http://www.linode.com/tos.cfm

Wait so the only backlash they get should they break their guarantee is that they credit me for the down time? This doesn't seem like a motivating factor to keep uptime. Is this standard practice in the industry? I know prgmr has refunded me the entire month when they broke their SLA.

All that said, I won't be getting rid of my Linode box in Fremont simply because it's one of the best vps's I've had performance wise. It just seems like an odd SLA.

It's a term of service (TOS) agrrement, not a professional SLA.

Most of us don't pay Linode anything important enough to warrant an SLA.

You don't get very strict 99.9% SLAs for $20 and $80 a month anyways...

They were down for us nearly 6 hours Saturday night, 1 hour this morning. They broke their SLA by a mile. Moving to EC2 asap.

What is EC2's SLA, and what cases of downtime have they experienced over the past couple of years? I'm assuming you wouldn't change providers without first finding this out?

Reply on news.ycombinator.com