Hacker News Reader

Comment by paukiatwee | Hacker News Reader

Recently one of my production node landed on a bad physical host, DO reboot my node twice within 24 hours, I tweeted about it and DO support open a ticket for me.

Since the physical node have problem, I plan to take snapshot then spin up a new node using the image. However, as the physical node have problem, the snapshot take more than 7 hours and still failed to snapshot (No way to cancel snapshot task).

DO support told me the snapshot is running, but I seriously think their support not honest about it and I ask they provide proof to show the snapshot is moving, they failed to do so. (From UI I can see JSON response show progress stuck at 3%, they lied)

Yesterday, DO reboot again my node.

Just my two cent, never ever use DO for serious production usage.

View on HN

Alternative perspective - Recognize any time you use a cloud provider, you need to be aware that not only are the nodes ephemeral, but the zone/data center you are deploying on may go down as well.

Plan for your server to go down at any time, and, if you have actual $$$ in play, also plan for the zone to go down.

Amazon encourages this thinking by having very low-latency links between AZ (Availability Zones) - and they also recommend deploying in two geographical regions should a regional disaster occur.

I know that all. The point is, DO support lied. After reboot, they still put my node in bad physical host. Also, DO reboot without notice (AWS have EC2 events).

Amazon has a robust ecosystem of supporting services and APIs to enable you to orchestrate as you described. DigitalOcean does not.

Same happened to one of my production nodes as well. I had to take a snapshot and migrate to another physical node as suggested by the support team. Had about 6 reboots without warning in the last 2 months.

I love DO, but sometimes the reboots without warning can be annoying.

Unfortunately, VM storage isn't a solved problem at scale. You can use remote storage (like EBS) to avoid being dependent on any particular server, but in practice it also has outages and performance anomalies.

Why would you tweet about it before opening a support ticket yourself?

it works fine for production. although one time one of my none essential droplet had trojan and started launchign DDOS attacks and it was shut down. I don't know how it happened but the password was super weak.

Strong passwords and always use a private key when SSHing.

Or no password and only private key :)

isn't the whole purpose of private key to avoid having to type in password?

You don't want passwords, at all, so that you can't be bruteforced.

On the otherside, your private key, on your local machine, should be password protected. But the node itself should not accept passwords over SSH.

Reply on news.ycombinator.com