Hacker News Reader

Comment by tnolet | Hacker News Reader

Just started using the new Kubernetes offering on Digital Ocean. Still in Beta but works pretty good. If this is a sign for how they will do databases, I'm all ears. Add CI/CD and you have a Heroku competitor.

View on HN

You can do custom agents with Azure DevOps (formerly VSTS) or GitLab pretty easily. Haven't played with the new Github stuff.

How are you liking the K8s on DO? I just got access a couple days ago but haven't touched it yet, I seem so out of the loop.

Thanks for mentioning GitLab. You can find more information about Auto Devops and creating Kubernetes clusters from withing GitLab at this doc https://docs.gitlab.com/ee/topics/autodevops/quick_start_gui...

Glad you are having fun working on your after hours side project with DO. But please DO NOT host with DO if you plan to run a real production systems, build a company, hire people etc. I have never found a better company to spun off a server and play with some settings but when their algorithm decides there is something fishy, bye bye your account, servers, backups, you never going to see any of this ever again.

The last startup I heavily pushed with switching to OVH or even Rackspace was exact sample of what happens when DO algorithm decides you are not genuine. That's it. No explanation, no phone number to call, nothing. These people blindly decided to believe their algorithm and never wanted even discuss resuming the account or even get us backup of data. There was nothing shady going on I assure you. Funny part is as of today DigitalOcean is still in violation of GPPR, as we have requested to hopefully find out what was wrong with our account by filing request for info. Nope, zit, nada, totally ignored. We have filed complain with proper authority and also notified Attorney General in NY and continue await result.

Build all you want on DigitalOcean, but please understand not people but their weird algorithm is in charge of the future of your startup, future of your company and future salary or lack therefore when you are forced to fire team of people because you fall behind with payroll. In other words: be warned and build at your own risk.

This is frightening. Can anyone from DO chime in?

Just making sure to reply to each person that raised a concern. There is a lot of fraud that comes into every single cloud provider as root access to a virtual server can be used for a lot of malicious activity. As a result every cloud provider has automatic and manual processes that they run to find these accounts and flag them.

In the case of DigitalOcean when an account is flagged it is locked, which simply prevents a user from creating more resources and an email notification goes out to the user to establish a line of communication.

There is no service interruption, and certainly the account, it's droplets, and other resources are not deleted, and never deleted automatically when the account is initially flagged.

There are numerous communications that go out even if a user is unresponsive.

This sounds terrible. I never had any issues with DO. They actually refunded me once when I actually made a mistake. Would love to hear the DO side on this.

Our team is currently migrating our whole production system from Linode and GCP to Digital Ocean. Your comment is raising warnings, I'd like to hear more from DO itself.

Out for curiosity but asking honestly: what made you chose DO from all providers out there? It sounds like you have a serious setup that most likely is production and probably makes money. You do know that DO does not provide over the phone help, right? So if your server goes down you open the ticket and you wait... Yes they do offer SLA because I was in heavy email chain back and forth when moving one of my clients from Rackspace and they were sold on DO being the right choice, but again I was explained no phone support - only that your emails to customer support are prioritized.

I'd love to chat with you. If you have some time, send an email over to jdonnell@digitalocean.com and let's talk. I promise nothing but honesty, transparency, ideas, and maybe a few laughs :)

Any chance at an invite to the k8s beta for a large-ish open source project?

Toss me an email, I'll see what I can do :)

This is bad, but your data should have also resided somewhere outside DO.

I was only technical adviser for a while, but I know they did offsite backups but its hard to have these up to a minute. So they tried to recover the latest version of their DB so that truly no records are missing.

Edit: my original post got downvoted severely in few minutes from posting. Hello DigitalOcean staff and/or owners!

I think more background information would be more appropriate than conspiracy theories of what happened to you. Usually there are two sides to a story.

How did you figure out that DigitalOcean staff and/or owners were the downvoters?

Don't participate on HN a lot, but I couldn't resist responding to the GDPR violation in the same line as engaging with the NY Attorney General.

I downvoted your original comment because it was a bunch of unsupported conspiracy BS and at least one of your claims was directly refuted by the company itself.

If you have a real complaint against DO, please provide specific allegations and support for those allegations.

Chiming in with everyone else. I would like more info from OP and DO since I host all my sites there.

These are some serious allegations. If this is true then DogitalOcean needs to explain themselves.

I guess, one should also have backups off site. Relying on just one provider constitutes a single point of failure.

I suspect that any cloud provider has enormous amounts of fraud on it and no tooling is 100% error free. It's also important to note that GDPR has specific exemptions for companies responding suspected security/fraud related issues.

I love DO because the performance for the price is great. Also have no issue supporting emerging tech companies with cultures I connect with.

No question fraud exists on all these platforms, however, if anyone is wrongly flagged by an algorithm, reaching out to the company must be followed by a prompt and timely response, so any misfortune can be remediated. I would assume fraudsters wouldn't typically reach out to have their accounts reinstated.

In the digital age, startups and businesses rely on cloud providers for their livelihood. These providers must be reliable and trustworthy, otherwise they shouldn't get a penny.

Also, I'm not intimately familiar with OP's situation, my comments are just common sense generalizations; I think!

We have an entire fraud and safety team whose sole purpose is to deal with these situations. Every account that is flagged is notified. Every account is communicated with and there are always replies sent. Unless a droplet is actively being malicious, such as sending out a DDoS attack, or performing some other sort of determined malicious activity, there is absolutely no interruption in service. The account is locked so that the account can not create more resources, but there is no disruption to the underlying running resources such as droplets and otherwise. The intent here is to establish a dialogue with the user and determine if the activity is fraudulent or otherwise.

Thanks for chiming in. I’m very pleased to see you engaged with your customers/potential customers; especially in a high stakes community such as HN.

I’m not going to comment any more on this situation as I don’t know all the details, I do hope I won’t see comments of unhappy DO customers on HN in the future, as that would be a sign you guys are not up to expectations.

P.S. I have been a happy DO customer for years; thus far (:

GDPR does not have specific exemptions for fraud. It is often possible to process personal data for anti fraud purposes but it requires a full legal assessment in the same way as any other processing activity would.

Hey friends! My name is Jarland and I'm on the support team at DigitalOcean. We do have a number of fraud and abuse algorithms, and when we are alerted to potentially fraudulent activity, we take appropriate action, which includes notifying and communicating with individual users. I also want to confirm that we are fully compliant with GDPR.

Thanks @jarland for chiming into this thread. I love how DO has evolved over the past several years and want to comment about that.

I use DO for production and have gradually migrated my infrastructure away from AWS and Linode to Digital Ocean as the platform improved.

Just a quick question: If the algorithm is triggered (regardless if it is a false positive or not) and the user is notified, what happens with the droplets in the meantime? Is there a grace period for the user to act before DO takes action? And is the whole account frozen or just the offending droplets?

It seems the major concern amongst commenters here is the sudden lost of service.

Thanks for the great service, and I look forward to your insight on this.

Yeah, I want to know if the execution is before or after the trial. Part of DO's appeal to me is the simplicity and predictable (low) cost. It would be really great if they published well defined account termination procedures. Do I get a phone call? An email? Do I get to respond before being disconnected? Is there an appeals process?

As anyone who runs a service that provides full root access to servers understands there is a tremendous amount of opportunity for potential abuse. It becomes a game of cat and mouse to catch the abusers and prevent them from creating numerous accounts which ultimately impact system performance and can lead to potential problems for real legitimate customers.

Those guidelines aren't published specifically because if they were, then the abusers would immediately begin to route around them, so it's meant to be opaque for a reason, but that is against fraudulent use, not legitimate use.

@raiyu, this is excellent and thank you. It’s reassuring to see DO’s leadership come out and explain things.

Depending on which items are flagged the account is put into a locked state, which means that access is limited. However, the droplets for that account and other services are not affected at all.

The account is also notified about the action and a dialogue is opened, to determine what the situation is.

There is no sudden loss of service. There is no loss of service without communication. If after multiple rounds of communication it is determined that the account is fraudulent, even then there is no loss of service that isn't communicated well in advance of the situation.

The answer depends on a variety of factors, but in general, when we're alerted to something that could be a violation of our Terms of Service, we attempt to engage with customers. In some cases, we may take actions against the resources running against an account and a vast majority of the time, there is a grace period before any permanent action is taken. If you have questions about specific cases, we recommend contacting our support team directly.

It may be worthwhile to have a blog post with a (LOT) more details (maybe any correspondence with DO included) and include a link to that.

This is exactly why i'm ok with paying 10x more for AWS. You can speak with them in few minutes, you have a grace period before service shutdown unlike ovh, DO and most others

I've had an experience like this but with less consequence. I was running a server years ago that scraped information from various cryptocurrency sites and it was flagged as fraudulent. Support refused to say what the issue was or offer temporary access to the server for data retrieval. This wasn't some cowboy unbounded crawler either, it was making very specific requests, around 1000 a day for around two weeks, probably less than an average day of web browsing for the majority of people.

I'll never use DO for anything, even testing and mucking around. At the very least you would have to provide excessive evidence of identity if you get flagged by their magic algorithm, and if they restore your service it could be after days of wrangling with support (it took support over a day to answer my support tickets at the time).

I glanced at the DO terms of service, and crawling is prohibited.

I just had my account locked with no warning or explanation. All 8 droplets were turned off. Account was unlocked 40 minutes later (also with no notification), and I could go in and turn droplets back on.

Over 2 hours later and still no response to my support ticket asking why and how it happened.

I'll be interested in the response I get. Unless there's a good reason why, or a plan to prevent the how from happening again, I'll be shifting anything critical away from DO, and go back to just using it for spinning things up to play with or test on.

Ufortunately nobody here is going to listen to you. A flock of DO fanboys downvoted my comments into obl livion despite I was just merely posting my experience. Some ambulance chaser even turned out to be a psychic reader because he knows better than me myself whether we been provided with GDPR related response or not, because - well, some DO CS worker told so on this forum.

I guess getting your droplets cutoff in the middle of business day and waiting 4 days for customer support copy and paste template answer has to happen to everyone before they themselves realize how crucial over the phone support is when it comes to hosting a production website. Its all good anyways.

Reply on news.ycombinator.com