If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png
They gave lots of warnings and also sent an email with the date the account was going to be closed a week in advance.
Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/
Having multiple people that receive billing notifications is strictly better than having a single person - what happens if that person is on vacation, sick, overloaded? No one even has a chance of catching this. If at least one other person receives those increasingly important emails, they at least have a chance to catch this before it breaks.
It seems to work—at this moment, anyway—even in private mode for me now though. ¯\_(ツ)_/¯
The real kicker is that they refused to tell us what terms we had allegedly violated because it’s “critical to maintaining our security systems.”
And they had “just conducted a manual review” and confirmed the suspension.
Then, after escalating further and a few choice words about how much I’ve spent on their platform, sure enough they conducted another “manual review” and found no issues.
I’ve really tried to avoid moving to AWS over the years, but I’m really reconsidering.
Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case.
Though I'll agree that weekends should be considered off-limits regardless.
Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.
I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...
They proceeded to DDoS my droplet.
Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy.
As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.
Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto.
Essentially the hosting company is offering a cash equivalent for unreliable credit.
This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose.
This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target!
The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash.
Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks.
So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops.
Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1]
An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale.
Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine.
Axed.
[1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.
Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.
When my account started spending 300-400 USD monthly, I was contacted by an account manager, who was at my office for a meeting two weeks later.
It doesn't matter. Google operates at scale. As long as they save more money by not bothering to handle such incidents better, they will see it as a perfectly good business policy.
Incidents like this turn customers off for life.
Small customers are completely interchangeable for larger service providers. You can easily get new customers by throwing around some free credits to college kids and posting a few tech ads masqueraded as tutorials.