Digital Ocean NYC2 Droplets Down | Hacker News Reader

Digital Ocean NYC2 Droplets Down

52 pointsheavymark68 comments

And counting. Second time in about a week and during prime time hours. If it doesn't get up quickly or if happens again to NYC2 in the next few weeks may be time to reconsider DO for hosting production sites. I love them, and glitches happen, but twice in a row clients have a hard time swallowing.

View on HN

Why do people have their eggs in one basket? More specifically, whenever hosting comes up on HN, it's always DO vs Linode. Why not Zoidberg^Wboth?

In the age of DevOps, it's not hard to configure different providers. I have been using Linode and DO for a while now. I noticed DO go down last week, checked my site, and it was UP because it was being served via Linode.

I flinch every time I hear someone say to me that their service has redundancy because they use multiple Amazon regions. Multiple providers people! And preferably multiple providers in multiple states, paid for by mutually exclusive credit cards!

Redundancy doesn't mean having multiple hosts by the same provider, within the same data center, within the same rack, provided by the same power strip.

tl;dr: Plan for outages.

It is always DO vs. Linode because both are relatively stable, relatively good for their price points. Tbh, the only real reason I use Linode over DO is the fact I've seen higher restarts on my DO nodes and the fact I've attracted DDoS's at a size Linode shrugs about.

How do you do this on the DNS layer? Very short TTL and switching on failure detection? Do you use both providers at the same time when both are up via DNS round robin or similar?

There's nothing stopping you from adding multiple A records for a domain. In fact, that's what a lot of people do:

  dig a www.google.com

  www.google.com.         300     IN      A       74.125.237.178
  www.google.com.         300     IN      A       74.125.237.177
  www.google.com.         300     IN      A       74.125.237.176
  www.google.com.         300     IN      A       74.125.237.179
  www.google.com.         300     IN      A       74.125.237.180

Browsers these days will resolve and try each one, on a first-come-first-served basis. If one fails, it will try the request on the next server.

If you have your own IP-space you can use BGP to route IPs to different locations.

If you're relying on DNS then short TTLs are pretty reliable, but there can be extended outages due to erroneous client-caching.

I'm a big fan of Linode. Have had one of their mid-range servers for about 2 years now and have had no troubles. When it came time to upgrading, or any other questions I had, their support staff is 1) quick, 2) knowledgable, 3) thorough & helpful.

I've used DO in the past for quick spin-ups, and they're great for that. But for long running applications I'd stay away. Was actually considering DO for a new project, but this kind of tipped the scales the other way.

If you're looking to switch, I think Linode gives you the best bang and reliability for your buck. Have also used Rackspace for a previous startup. A bit more pricey, but also a great service if you're considering switching.

Linode can't silo resources on a per VPS basis and constantly will harass you for over-utilizing your share of arbitrarily enforced limits. Just a heads up if you actually utilize your instances.

Are you talking about the "Linode Alert" emails? Those are user-configurable — in the email, they specifically state "This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds based on your own individual needs."

If that's not what you're talking about, I apologize, but I've been using Linode for years (often with very compute-intensive projects) and I haven't seen what you're talking about.

Just out of pure laziness, I use the CPU and disk access emails as makeshift notifications for when my backups run.

Really? I find that odd. I do load testing on their instances all the time. which maxes out cpu/disk to give me an idea of performance changes for stuff I have high throughput on.

Hell, I've gotten DDoS'd for like 2 Gbps and they were like "Oh, ya, its fine."

So, can you explain more about what happened?

Have run a web-app with 30K+ users. Only been 'harassed' when I've been warned that I'm running the CPU at 350% of max rate and I should be concerned. That's ok in my book.

I find that surprising. Linode support never contacted me regarding resource usage. Not even when one of the servers has been under DDOS. I contacted them first notifying of such event. And that was it.

I agree with harpastum. I think you're confusing automatic email notifications based on your own settings with Linode support. I feel like you have to seriously abuse their infrastructure to get them to contact you.

Disclaimer: has been a happy Linode customer for over 4 years now.

http://i.imgur.com/UCPWdBq.png

I was not confusing automated emails with direct contact from Linode support. I fully understand and can appreciate the need to be a "nice neighbor" in a shared environment but what baffles me is:

1) Linode doesn't have the capability to actually enforce any sort of quota or limit. It's 2014, there's been numerous advancements in the area of virtualized servers in the past decade. cgroups is a very easy way that works across xen, kvm and lxc.

2) Linode doesn't publish these arbitrary limits nor give guidance on which node size comes with what acceptable usage levels.

This is ridiculous. Would suggest you move out of Linode right away.

It's their job to limit resources on your node, not your job to self-limit. Asking you to limit resources because it's "affecting other customers' performance" is not only outrageous, but also unfair with the other customers being impacted.

This reminds me of the late 90's [1], when web hosting meant sharing the same box with hundreds of other customers, and very limited tools to enforce limits. When you had a resource hog, the only option was really to get them to reduce usage, or move them to a separate box (which wasn't as easy as today).

[1] ps: I know this well; used to have a web hosting company at the time. We had to frequently move customers around, and in some cases to terminate their contracts - e.g., porn sites. But this was a good 10 years before Xen become commonplace.

Yup. That's the conclusion we reached and also why I suggested approaching Linode with caution. They're attempting to compete with the "cloud" providers but running their business the same as they did a decade ago.

Thanks for explaining. :)

Anything I have that uses that level of disk i/o I don't run in a VPS which would explain why I've never noticed before.

Thanks for the answer.

1) I don't know enough to talk about resource limiting that Linode might do. There might be a reason they don't want to or can't use it with their infrastructure.

2) I agree, they could publish some guidelines for resource usage, but this is hardly baffling. I'd guess most of the customers just don't hit this problem, so there's no need to list all of the limitations.

If you'd used their servers for a bit longer, you'd have plenty of trouble. Linode is notorious for dreadful security practices. Their main ColdFusion app (yes, really) has been hacked on numerous occasions, including at least once incident that involved customer data loss. Linode dodged the issue and didn't tell their customers until badgered.

I can agree that DO isn't suited for production apps, but Linode isn't either. They're both VPS playgrounds.

Is there a better alternative you would suggest? While Linode has done better than DO, that fiasco left me feeling somewhat uneasy as well.

There are a great many VPS and server providers out there. I've had good experiences with gandi.net, Rackspace, moln.is and pgrmr. I've seen Ramnode and Lithium Hosting get a lot of praise. You could also use AWS if you don't have any ethical objections to using Amazon products.

Same question as antsar. What would you suggest then?

I use DO because they have servers in Amsterdam, which happens to be very close to my country.

I never had problems with DO that weren't resolved promptly.

Just a note, Linode is very US-centric anyway... (and that one London location doesn't make it international very much)

Their status page indicates "We have identified a slow connection to our NYC2 data center". Assuming they have redundant connections, this seems like a big understatement for being down for 1 hour and going. Even their control panel is down for me. ( http://i.imgur.com/3ltg00o.png )

we've had an ongoing issue with a juniper fiber core in nyc2 it seems. We've engaged juniper to help us, I don't expect this to be an issue again.

Linode's new offering probably the better choice these days. I get the feeling that DO is just overwhelmed with problems related to their hyper growth - 10% per month on a huge base.

http://trends.netcraft.com/www.digitalocean.com

They've done so well to this point keeping up with the growth curve. Hope they don't become crippled by success.

Having previously left Linode for Digital Ocean due to lower pricing and SSDs, I can safely say Linode both dropped their prices and matched the DO plans with SSDs included. If you want to jump ship, that's the route I'd go for a comparable.

I still love DO, but downtime is a no-no when it comes to VPSes.

In the process of switching over now.

When you charge so little and the SLA is just credit during the time of the outtage (pennies), downtime is pretty much a given. It always surprises me that this surprises people.

You should probably reconsider hosting production sites on super-cheap VPSes with a 1:1 SLA refund. They're going to go down. And, when they do, you'll get the 7/10ths of a cent refund per hour promised for your $5 instance (1.4 cents per hour for $10, 2.8 cents per hour for $20, etc). But if you depend on a site for your business, ask yourself how much it costs you per hour of downtime and plan appropriately.

For context: I'm a happy Digital Ocean customer (as well as Rackspace and others). I like their services and consider them a great value for the money. I've also gotten multiple developer friends into using them. But I don't let them be a single point of failure for anything critical.

downtime is an issue no matter how much you pay your infrastructure provider. You should account for downtime in your application and your infra by taking advantage of low-hanging high availability fruit.

Things like saltstack (saltstack.org) make this mega-easy.

It's easier for apps since you can build in failover support based on multiple servers pretty easily (I just implemented this in our 'live' installers recently on Windows). It's a bit harder at the website level for an average user, though. I know far too many people that have a single point of failure website on a super-cheap VPS that's just down whenever the VPS is down. Most of them would have no idea how to implement a load balancer and failover support. Though I could probably get them into something like CloudFare or similar.

I am getting pretty sick of it, too. I might go back to Linode. I was already upset that they dropped Arch Linux support.

Linode has price matched them so its not a price issue anymore. [e.g. $10 Linode is roughly the same as $10 DO]

Glad to see the ProjectWonderful ads providing something, even if its small btw. :)

It definitely helps. The biggest source of income has been (surprisingly) Gittip, though.

Ya, I am surprised by that too. I'd create one but I'd feel guilty. My ads cover my costs.

Er, no.

My droplets are up. Never went down today.

Also, please don't post that things are down here on HN. This is not a support forum. Not for DO, not for AWS.

You came here after it was fixed. It is fine now. Also, it was just NYC2, maybe only part of it.

Mine are on NYC2. Terminal sessions up all day. On three of them.

Check http://downforeveryoneorjustme.com instead of HN.

Or you could check the official status page[1] and see that there was indeed an outage.

[1] https://status.digitalocean.com/

Even further reason not to post it here!

Here's their status panel: https://status.digitalocean.com/

https://cloud.digitalocean.com/ now is unavailable too.

"but twice in a row clients have a hard time swallowing."

May I ask what your clients are paying?

Maybe the answer is to charge them more instead of giving them a low price (an assumption on my part of course which is why I ask what you are getting for this) and some implicit guarantee of uptime.

For example we had some customers some time ago who were pooled on a shared IP address for sending SMTP service. Occasionally they would experience problems with mail reputation on that shared IP. If they complained about that would simply explain that that was the drawback of a shared IP and that they could always upgrade to a service level where they got a unique IP and could establish their own reputation.

Folks, I strongly, strongly suggest if you use Cloud/VPS providers to look into something like Cassandra, Galera, Mongo, etc. that can do WAN replication and using 3 DCs.

Yes it costs money but things like this are avoidable. Every single Datacenter on the planet has downtime and/or performance issues from time to time. Hell, one of the places I have a dedi at had its network performance drop by 50% for about 15 minutes while they did emergency maintenance.

Things like this are going to happen as long as you depend on one provider.

I've had great experiences with them. I recommended DO to everyone.

Now I'm tempted to drive to New York and punch them each in the face.

PEAK downtime.

EDIT: $160 refund. Maybe just a Charlie horse.

I just created a droplet today, and was trying for the life of me to figure out what I would have done so early on in setting it up to cause it to become unresponsive. Glad to see it's not just me. It's not really a great sign though, as I was hoping to use this fairly frequently. Is this a common occurrence, or do I just have bad luck? If so, I'll definitely have to find another solution.

I'm not too qualified to comment on the hosting market, but I do want to vouch for DigitalOcean as I've never had uptime issues with them until now. They also have always had timely and helpful support and their pricing is obviously competitive.

That being said, near 100% uptime is really desired in this space and I've been scratching my head for the past hour or so wondering why I couldn't connect to my mongo process running on my droplet...maybe I should have tested DO sooner, but the fact that only mongo seemed to be failing (and not postgres for example) made me think I had client and not server issues.

I've been using Linode for many years, and DO for some time now (a year? not sure, since I can't even login to their control panel right now and check). I've seen issues on both providers, and dealt with support of both not too infrequently.

I'd still say Linode is miles ahead in terms of expertise and responsiveness. Most of my servers are on Linode for this reason, and only a few on DO.

As a customer I enjoy this competition and would like to keep both players compete - because I'm benefiting from it. So I'm likely to stick to both. But I would still recommend Linode more easily than I would DO.

Thanks for your input. I might be switching over as a result of the recent downtime and the support Linode has been receiving on this thread. Unfortunately, especially for non-hobbyist apps and services, this is an area demanding fickleness.

The big issue here is NYC2 droplets went down only 7 days ago. This time its even longer at over an a half. When it happened last week it was unfortunate, but things happen. When it happens twice in a week that is a much bigger problem.

Agreed. I think what worries me most is that I wasn't aware of their downtime a week ago..

They sent out an email last week with an explanation.

I'd love to know what's going on. My website hosted on DO, michaelbreedlove.net, is unavailable. I also have a few services running on subdomains of the same DO VPS instance; they're all accessible. I also have SSH access. They seem to be having some serious HTTP routing issues.

Yeah , NYC2 seems affected https://twitter.com/digitalocean/status/494170340973887488

It seems to be http (perhaps load balancers) rather than the instances being down - I'm ssh-ed into a droplet and everything is working fine, except remote HTTP connections.

I ssh-ed into some droplets, but others are completely inacessible outside DO. The internal network works fine, so it is just a several network outage. Again.

I'm struggling to ssh into droplets, but it seemed like when I was ssh'd in, I didn't have an issue at all.

It seems to be intermittently up and down. So your connections all die, and you can't reconnect for a while, then it comes back and everything looks good for a bit until it dies again.

Same here. Since 1:24p, Pingdom has reported our sites going up and down about 16 times.

What happens when you try to execute outbound web requests from the instance?

http://netalyzr.icsi.berkeley.edu/cli.html

I've been unable to SSH in for an hour and a half. But I expected this when I signed up (cheap = cheap) so have backups running that were switched to.

We're getting the same thing. I don't care how cheap it is or how nice the customer service is, but if they are not up 99% of the time they will not be my web host.

I'm back up as of 2:59pm EDT, so I was down for about 1:35. This is the only outtage longer than a few minutes I've had with DO thusfar.

One of my instances on NY2 is fine and the other one is sporadically accessible via SSH. Hope they figure this one out soon.

This is a little ridiculous. I hope not to have this problem again. Second strike her. Unfortunately.

its working now , atleast for me .

I have to agree with you on this one.

Reply on news.ycombinator.com