Do not use Digital Ocean for production

>I guess I got what I paid for.

News flash, you won't get any better customer service from AWS or Azure, etc. (unless your dropping lots of cash). With AWS you have web forms to fill out to create a support ticket, good luck ever getting a real person on the phone.

This is all of course subject to the size of your account. Sure you can get a dedicated AWS account manager if you throw enough money around. But running one small managed k8s cluster won't cut it.

You can just pay for premium AWS support, and it isn't even that expensive.

I had a real person on the phone from AWS today. They actually called me after I opened a ticket yesterday and hadn't responded to their latest ticket update.

I have had an AWS support person call me a few minutes after I got locked out of my account (<$10/mo spend) when I broke my 2FA device.

You can sign up for a paid AWS Business support plan, immediately use it to file a priority ticket, and cancel it after 30 days. The business support plan includes 24/7 phone support. For small peanuts users, this is $100.

How do you sign up when you can't log in like the OP?

AWS doesn't send you codes via email like DigitalOcean does. The situation as stated doesn't exist at all.

If you forgot your password AND lost access to your email so you can't use the reset link, you can contact them at https://aws.amazon.com/contact-us/ without having to log in. They have a special system for lost login support requests.

We were on a $400 google gold support plan and got shafted regardless https://talktotheduck.dev/production-horrors-handling-disast...

The fact that they have some person answer the phone doesn't mean they have to fix the problem.

If your bill is >$500 / mo, you get their "Business" support level which lets you use their customer success managers.

If you're not spending this much, you won't get good support from them or any cloud vendor. AWS, Azure, and GCP all push you to the forums or submitted a ticket which may or may not get read.

True though with a lot of the European ones you get good support and/or can pay explicitly for a level of support.

The email disappearing with no trace sounds like Gmail hating DO address blocks though.. Gmail sends only a fraction of things it thinks are spam to the spam folder.

Just found a blog post with a similar issue - seems like DO silently blacklists your email in certain cases.

https://qertoip.medium.com/you-will-randomly-lose-access-to-...

And on the other hand, only a fraction of things it sends to the spam folder are actually spam! The universe is in balance.

I don't know how DO's 2FA works, but when you signed up were you not given a number of backup codes for a contingency? I wish more vendors did this.

Also I think this isn't so much a caution about DO as it is poor implementations of 2FA without good customer service to back it up.

Digitalocean offers TOTP for 2fa, with backup codes! No email needed. If memory serves well they only offer a TOTP and SMS based 2fa so i'm not sure why an email would be sent for this.

Edit: Their docs mention the 2fa modes that are supported: https://docs.digitalocean.com/products/accounts/security/2fa...

I never really activated 2FA - the email verification code routine is something they do when one logs in from a new browser, or when one deleted cookies it seems. Unfortunately there are no backup codes for that.

I've never gotten an email like that from them, but I also have 2FA set up.

I guess once you have 2FA setup it's not necessary anymore. I just set up a new account for testing purposes, and I get their 6 digit code email out of the box.

https://docs.digitalocean.com/products/accounts/security/2fa...

Looks like they provide backup options.

We ran our backup server on Digital Ocean. Needed to restore our database backups but were bound by very low I/O read rates from the Digital Ocean disks.

Another issue we had is that they decided we were being DDoS'd, despite a server running no externally accessible ports apart from SSH (which was not being hit). Their solution was to disable the networking, therefore preventing our services on the machine from running. They couldn't provide any further information on the DDoS. Their suggested solution was to "put Cloudflare in front of it", but of course with no services to put Cloudflare in front of, this was useless advice. They did not understand this point.

They're easy, and not bad for a first pass/MVP, but not ideal for production.

The grass often looks greener. I've never had good results with any cloud company's customer support with the only exception of Google, but that was because the individual I was assigned actually put in the effort themselves. Azure/Microsoft for example didn't even call me back when I requested a windows VM quote for a client I was working with.

yeah I company I worked for had terrible customer service from google and great customer service from amazon, so I think it's just a roll of the roulette wheel as to who you get as your CSM (once you pay for it).

Welcome to the new normal. Simple Bank attempted the "no customer service" stuff with banking products and fortunately it failed. Robinhood is attempting it as well, and, well idiots keep pouring money into it. Your email provider went that way a long time ago. It's profitable not to provide customer service, and companies are in a place where it's hard to sue them.

If they are using Google Workspace they will get pretty decent super from Google. You can also check the reports to see if the email even made it as far as the mail server - some items are blocked before they even hit the spam folder.

This reason exactly is why i'm terrified of using email based two factor. I find the OTP from 1Password more reliable because nothing has to be sent or received.

One of our suppliers went bankrupt and left us hanging without proper access to their Digital Ocean account where they were hosting some of the services they had provided for us. I think our account was already configured to have access but only as billing contact or something like that, so we couldn’t do anything besides paying the bills.

I wrote DO and explained them the situation. It took a few days and a bunch of documents back and forth before they managed to let our account have an admin role.

All in all they provided us with pretty good support and we were not a big customer - a few hundred euros a month, I believe.

Cloud is too large and usually too cheap for most people to really get meaningful customer support. I have heard of the same challenges from Azure and AWS too.

The exact same thing will happen with AWS, GCP, and Azure unless you are spending a minimum of about $1000/month or purchase elevated support.

AWS does not care about your spend - you can always add support and this is a % of your spend with a minimum. You can add it and remove support at will - https://aws.amazon.com/premiumsupport/plans/business/

You broke your production system, and didn’t have 2FA set up, or a recovery plan. None of that is Digital Ocean’s fault. Slow response to tickets is unfortunately a problem with many services, a problem of scale.

I’ve hosted sites on DO for several years, never had any problems with them. On the few occasions I had to contact their support I got a reply right away.

Use OTP 2FA. Log in to your hosting provider console before doing critical deployments or upgrades. Use a staging server and have a recovery plan you’ve tested.

No hosting provider is perfect, some are more reliable and responsive than others. But instead of blaming this on DO maybe title your post “Don’t use my service in production.”

> You broke your production system, and didn’t have 2FA set up, or a recovery plan. None of that is Digital Ocean’s fault.

That their emails and logins don't work, is their fault though. Those things are so basic and essential that they go way beyond "No hosting provider is perfect". And then they offer no assistance in rectifying their buggy software.

I find it hard to understand that you blame the user when a product is so broken that one cannot login.

That I didn't have a recovery plan is an untrue assumption of yours.

Luckily I was able to mitigate this "outage" of digital ocean and host my app elsewhere. But this costs me work and energy I would have liked to invest elsewhere, and hence I would appreciate a more stable provider, where the most basic functionality reliably works.

The product may be broken for you. I have used DO for years and never had this problem. They offer OTP 2FA specifically so you don’t have to rely on emails. Emails can fail in many places, you shouldn’t rely on that.

Next time make sure you can log in to your hosting provider’s console before you make possibly breaking changes.

I get that you had a bad experience but that doesn’t imply that DO is broken or not worthy of trust.

2FA is typically implemented by using an authenticator app that generates time based codes. Looking at DO's docs, they do this: https://docs.digitalocean.com/products/accounts/security/2fa...

2FA by sending an email is kindof strange. I wonder if that's the method of security they use if you don't actually have 2FA setup?

p.s. For anybody reading this, DO docs talk of 2FA via SMS codes. Don't do that - it's generally considered insecure.

DO's email thing is not really 2FA - they do that when one logs in from a new browser or after deleting cookies it seems.

I was thinking the same thing 1-2 days ago when I wasn't getting the 6 digit code.

It wasn't anything urgent, and it arrived after ~5-10 minutes. But if it were something urgent then it would have been a huge problem.

That's what happens when you run your infrastructure on $5 servers, even if paying more, the company is optimized for people paying $5 and running mostly side projects.

If you're doing anything serious, go with gcloud or AWS. Their support might suck as well, but 99,9% of the time the problem is your fault and not their service, because it is so widely used and so many businesses depending on it that they'll notice even the slightest issue on their platform.

I unfortunately had a similar issue about 1-2 years ago also on DOKS. We upgraded some k8s infrastructure on our side, but something about the "managed-ness" of their k8s solution caused our entire cluster to break. I believe it was due to some sort of admission webhook controller that had a failurePolicy of Fail, causing dependent pods to break, preventing everything else from starting.

Some email providers block email from networks they consider spammy. Just drop them on connect, never accept the delivery. I can't get email from Linode to Outlook 365 for love nor money.

I wouldn't be entirely surprised if DO had the same problem from time to time, if they deliver direct from their own network.

Linode has always had great customer support for me. Responds in a few hours, friendly, helpful.

It took GitHub a month to not fix my 2FA problems. I think we rely on a lot of systems that for solo devs (where a colleague can’t fix your credential issues), it’s a terrible experience.

Anyone got data points from Linode?

I was going to post a separate top-level comment asking about Linode as well, but you beat me to it!

I've never used Linode for my own projects before, or for anything production, but a company I've worked for in the past has used it's k8s offering for non-production/lab usage and I found it very slick and easy to use from the UX point of view. It was stable, performed well, and generally looked like a great product.

We did run into one issue, though, where some internal and undisclosed account limit on I think the number of containers you could run had been hit because we had some automation starting a bunch of containers in there and then killing them off real quick (we were testing something if memory serves). All of a sudden we start seeing these unusual failures going on, coming from Linode's API, that made zero sense. So a manager contacted their support folks, who resolved it the next day with the first response to the support ticket. It wasn't production and we didn't file it as an emergency priority thing (even though it caused us to miss a deadline by like, a day I think), so I was perfectly OK with "next business day" turn-around.

That's my very limited experience with Linode, and based on that alone, I'd certainly be willing to give them a shot for a small-scale production environment, at least. And if that didn't go south very quickly, I'd probably be willing to do more.

A co-worker of mine is ambivalent to Linode, but swears by vultr. Says he loves them to death. I have no experience there, just wanted to throw out the name for anyone looking to see what options exist, so you know what to evaluate.

Why was this flagged? It seems like a valid reason to post it on HN.

they provide backup codes for this very reason

They do that for proper 2FA - but the email flow is not 2FA. They do that when you use a new browser or deleted cookies.