In Digital Ocean, S3-like space keys can access all your buckets | Hacker News Reader

In Digital Ocean, S3-like space keys can access all your buckets

ideas.digitalocean.com

127 pointssergioisidoro98 comments

The lack of ACLs or comparable permissions is by far the biggest thing that prevents me from recommending DO for production workloads. This kind of thing is absolutely essential. You can't even separate dev resources from prod resources, every API key has godmode on your whole account. This is a security disaster.

For a simple example, I'm running externaldns on a kubernetes cluster. For production use, I'd want to at least have an API key that can only access the DNS domains and nothing else. As is, the key can create compute resources or delete anything (including object storage buckets!). Again with no dev/prod isolation, so a leaked dev key that looks innocuous means your prod resources are now compromised.

Some second tier providers do better at this, OVH and scaleway both have some concept of ACLs, but DO just don't even try.

Apologies for the slight rant but it's frustrating to see something so basic overlooked by a company the size of DO.

I agree, probably the biggest point Digitalocean is lacking. I think the only workaround is creating several projects? But that probably becomes annoying and hard to manage quite easily if you want a lot of separation.

IIRC projects only group resources under the same account. API keys are still account-wide, so projects provide no isolation there.

There is a other mechanism DO has to allow multiple accounts under one billing context (I think it might be called 'teams'?) but sadly this is still extremely coarse grained, and doesn't allow you to lock a key down to a particular resource, or even a class of resources.

Teams have separate billing per team. Other platforms would call them ‘organization’

Agreed. We had solved it at PhishDeck back in the day through the use of multiple projects, i.e., Platform Production, Platform Staging, Tooling Production, Tooling Staging etc.

Still not ideal, but it minimised security risk and kept things organised.

You've described exactly why my company moved from DigitalOcean to AWS a while ago.

Whilst AWS is more expensive and perhaps over-engineered for our use-case, it was absolutely essential to have some kind of ACLs.

I mean, I can't exactly give my junior dev access to DO if it means he can also delete the entire production database and all backups with a misclick.

S3 (and similar storages) have caused plenty of security issues in several occasions (usually because of misconfigured buckets, making all contents available to the public). Given this, it would be expected that companies would pay a bit more attention to the security of these data storage methods.

However, in Digital Ocean, by design, you can't restrict keys to certain buckets. Once you issue a key, it can access all buckets within your project, with all operations (list, read, delete) files.

The issue has been reported and is known for a long time. It's even the top voted "idea" in the company portal.

Posting this here in hopes to bring awareness of this issue.

Things like these are why I stopped using digitalocean.

They had issues with DNS PTR records too for years. Same for initial auth tokens being global. I had hopes for them and used them exclusively at 1 time.

Really? I have had no issues using them for basic hosting.

My read on this situation is that DO is holding out until they have a comprehensive IAM solution. If you need IAM, don’t use DO. If you can live without IAM, DO is pretty nice.

Isn't this usually solved through some form of IAM policies? Does DO not have this?

No such luck there, but this is kind of the MO for DigitalOcean, sadly.

I think DO is doing fine addressing their target market. I also wish they had IAM but you have to admit IAM is huge and messy and is often more about managing your employees than your infrastructure. I’m okay waiting for them to get it right rather than rushing out a quick half baked idea that ruins their entire product stack err platform.

I don't care for the full on flexibility of IAM, but I'd kill for a simple "this key is for this bucket" feature. That doesn't seem so far fetched, I don't need per-object permissions within a single bucket!

Since the bucket namespace is global you can always just use different accounts. But I agree it doesn't seem that hard to add this small feature.

As a user, you can fix this by having a proxy that fits in front and manages ACL's.

Obviously you have to pay for resources for that proxy, and it's probably going to want a large bandwidth allocation. Lucky because DO doesn't charge for bandwidth.

If I have to implement and manage an ACL myself (such a basic feature!!), it's better to go ahead and implement something like min.io and ditch DO entirely.

This is mostly true, but there is a case where you use your transfer pool.

From their product docs:

Droplets have their own transfer allowance, independent of Spaces. Traffic from Droplets to Spaces does not count against your Spaces transfer allowance (because inbound bandwidth to Spaces is free), but does currently count against your Droplets’ outbound transfer allowance.

https://docs.digitalocean.com/products/spaces/details/pricin...

I'm afraid you broke the site guidelines by using the submission title to editorialize. Please see https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait; don't editorialize."

If you want to say what you think is important about a page, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so.... Alternatively, you could have made a text post to communicate what you think is important about the issue, and then your title would have been fine.

What's not ok on HN is to use the title field as a way to comment on an issue, because then the submitter gets a privileged position relative to other commenters. Since the title is by far the largest influence on a thread, we want to avoid that. Being the submitter of a page shouldn't confer any extra special power over the discussion.

Well, it used to be same for Cloudflare R2 , which was just fixed less than two months ago [1]. This was an unexpected showstopper for me.

I honestly find it hard to understand why this kind of basic security feature wasn't in GA.

[1]: https://community.cloudflare.com/t/r2-token-per-bucket/38906...

And DO consistently ignores requests to add this. It prevents me from seriously using Spaces and I think many others, too.

DO has had a terrible attitude to security for as long as they've been around. I reported a major data leak vulnerability to them and they told me it was operating as intended, so I published it, and then they accused me of irresponsible disclosure, while simultaneously claiming that there was no security issue. This was a dozen years ago and the founders were involved in the response.

They are at the top of the list of companies with which I will never do business.

Could you provide more details?

On delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning.

https://github.com/fog/fog/issues/2525

https://news.ycombinator.com/item?id=6983097

https://news.ycombinator.com/item?id=6983520

Some of the comments by DO on that GitHub thread, are really surprising, and disappointing to say the least...

I wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.

I do find the claim that "there's nothing irresponsible about full immediate disclosure" to be interesting.

https://github.com/fog/fog/issues/2525#issuecomment-31336855

The fact that you think it should be the customers’ responsibility to make sure that DO doesn’t accidentally give your private data away to unknown third parties is an absolutely minblowing take on this matter.

I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data.

If anyone can suggest any alternatives that are not Hetzner, I would be interested.

Well first, I've not worked for DigitalOcean in a long time, I left in 2015. I don't really wanna re-hash something from 2013, but again: when you deleted a VM you could click the scrub button or pass the scrub flag in the API when you issued a delete call. It was very well documented and the scrub button wasn't hidden anywhere in the interface. The fog platform wasn't using the scrub flag in their app for whatever reason. As I recall, we did change it to scrub by default shortly after. From what I remember, the reason it wasn't originally default was the zero'ing process tried up a lot of resource on the box and it was hard on the drives and we were very much in startup mode at that time with limited capacity, so it was better to only zero if the customer requests it.

You still don't appear to get it though. The fact that there even exists an option (never mind a default) which does not scrub storage between use by different tenants is completely unacceptable in a multi-tenant environment.

Imagine if your bank told you they only close the vault when a customer asks.

And they give you an excuse that it’s time consuming for them and you should respect their time.

Would you bank with them?

This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center.

It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)

I was always very clear at that time "this is a startup" - "do not use DO for mission critical work" - "the product is not very mature yet" - we had no sales team, nothing in our marketing was gear towards business and we primarily catered to students/personal sites/etc. I'd accept your analogy except we regularly said "use AWS for important stuff, not us".

Even startups have to follow the law.

Before the GDPR the EU had the 1995 Data Protection Directive, which you would have to comply with to have EU customers.

You can’t just decide to not follow the law because you’re a startup.

What law would cover this??

I just told you in my previous comment..

In the event your google is broken:

----

https://en.wikipedia.org/wiki/Data_Protection_Directive

>the controller must implement appropriate technical and organizational measures to protect personal data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure or access, in particular where the processing involves the transmission of data over a network, and against all other unlawful forms of processing.

https://en.wikipedia.org/wiki/Personal_data

Are you confused about what PII is?

Should I explain this too?

> tried (sic) up a lot of resources on the box

Do you know what else ties up a lot of resources? Civil action, negligence claims, PR disasters.

This specific GitHub issue is one of several why I will never use DO: language, official response, and attitude all matter. I'd be curious to know how much revenue DO has lost because of this one issue.

Wow, this just confirms my belief.

There is precisely zero circumstance where it is ok to give private customer data to another customer.

The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it.

The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.

Unless I misunderstood something and one has to explicitly opt out of scrub for this to occur, I agree 100%. This is surprising. And they were mad that a customer thinks this is undesirable?

I was using DO. It is not a serious project but I think I'll be moving on nonetheless.

For what it's worth, neom has moved on from DO a while back.

Yeah, I read a bit further and it was not scrub by default but had to be opt-in. And today is the first I heard that all API keys have ridiculous access. I'm done.

Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.

Looks like OVH does managed k8s, and they're pretty solid at this point for the most part. https://us.ovhcloud.com/public-cloud/kubernetes/

Thank you, I didn't know they did. That puts them top-of-list for me.

Why on Earth would someone have to opt-in to scrubbing with a flag vs. opting out? That's surprising. Is the flag on by default and they specifically opted out before complaining about it?

The scrub flag was default false, apparently because scrubbing was slow and caused wear.

> there isn't actually a security issue, the fog app wasn't passing the scrub flag

At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.

You're right. I hadn't really thought about it that way. My response at the time actually came from confusion on my part (that is my own fault I suppose). I got woken up at 4am on a Sunday by a flood of calls saying there is a post on the top of HN about DO leaking data between VMs. I got out of bed to see if I needed to wake up the engineering team, checked the support system and the disclosure emails because my first thought was shit someone has pwned the LXC/escaped the jails or something. Then when I read the github, I thought the scrub flag wasn't being respected when it was passed. That was the type of security issue I had originally taken it to mean, both I would have woken up engineering for. It was a someone is in the hypervisor type of security issue I thought we were dealing with. Once I realized it was fog didn't pass the scrub flag, I wish I'd just gone back to bed and dealt with it on Monday instead of trying to do it in real time, that was stupid of me. I also accept your point that an unexpected behaviour can in itself be a security issue.

> there isn't actually a security issue

The official blog from DigitalOcean is here:

https://www.digitalocean.com/blog/transparency-regarding-dat...

While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem.

> 10 years later and I still remember how pissed off I was that day, hah.

I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.

Consider the multiple S3 leaks that caused AWS to change their defaults to block public access and disable ACLs. You could say that since buckets were always default private, any misconfiguration was simply user error and users complaining were doing so in bad faith. Or you could understand the implications and help your customers protect their data.

I actually came here to add this very issue and then noticed this comment.

This issue is the reason why I will never use DO or recommend it to a customer.

Holy shit that’s bad.

I wonder how one could test for that “automatically” on other providers.

Don’t forget the time they issued the same sshd keys to every instance because they didn’t scrub them from the images.

Reminds me of a hosting company in my country. When you use their spaces thing, they generate a password for you (not being able to set your own is already a red flag for me, but ok). It was "hidden" in the UI to look secure. However, the password itself was passed as an argument that was part of the URL. So everyone who logs access urls could see the password.

I reported it, they fixed it quickly, but never acknowledged that it was a "security" issue.

Yeah I was unaware of this, and will no longer consider DO for anything beyond basic VPS hosting, if even that.

See also this old comment on security issues with their managed Kubernetes offering.

https://news.ycombinator.com/item?id=22490390

Wait till you hear that when you generate a DO container registry key it includes a key that gives you access to all of the resources, not only the registry.

Are you saying you have to issue a "god" level API access token to access a single docker (private) image in the Digital Ocean container registry?

The permissions options on the API key are read or write for the whole project.

There is basically no granularity.

It's not about the API key. It's about the container registry credentials. Which you would expect to be able to only interact with the container registry. This is not the case. If you go to the container registry and click to download credentials you will get JSON like:

    {"auths":{"registry.digitalocean.com":{"auth":"<base64-encoded-credentials>"}}}

If you decode the base64 encoded credentials it will be a string like "<token>:<token>" with either a read only or read/write token to all of the resources on DO.

> It's not about the API key. It's about the container registry credentials. Which you would expect to be able to only interact with the container registry.

think my reply shortness maybe has you misunderstanding me?

The fact that an API token can only have read/write project wide basically results in everything you said.

No RBAC on services. No RBAC on specific API actions. Anyone with read+write can do/nuke everything.

That’s why I’m migrating $COMPANY off to AWS as fast as I humanly can by myself.

Ok, I see what you mean. Still, even knowing that general access tokens are all or nothing it was surprising that exactly the same thing was used for containers. Like, if you generate a separate credentials for containers it strongly suggests there are for containers only.

Yeah I use doctl, which has a big note on the docs saying

> you can simulate what `doctl registry login` does by using an API token string as the username and password when calling `docker login`

https://docs.digitalocean.com/products/container-registry/ho...

Yes. And when you generate the container registry token it doesn't mention any of it and it's hard to notice, cause docker auth is base64 encoded. But after you decode the credentials you get a regular DO token that you can use with the API

I haven't used DO Spaces for a good while as I've shifted to B2 as cheaper S3-compatible storage that suits my archival workload.

However I absolutely would be using Spaces as storage alongside any apps running on DO, and would have assumed that all the usual per-bucket permissions I'm used to elsewhere were present, so thanks for the heads up!

https://buckets.grayhatwarfare.com/

Is this post somehow related to the recent Ceph CVE-2023-43040 ("Improperly verified POST keys"), where Digital Ocean was also credited?

https://www.openwall.com/lists/oss-security/2023/09/26/10

---

A flaw was found in Ceph RGW. An unprivileged user can write to any bucket(s) accessible by a given key if a POST's form-data contains a key called 'bucket' with a value matching the name of the bucket used to sign the request.

The result of this is that a user could actually upload to any bucket accessible by the specified access key as long as the bucket in the POST policy matches the bucket in said POST form part.

We have assigned it a CVE of CVE-2023-43040 and the patch is attached.

Credits to Lucas Henry of Digital Ocean.

Nope, this post is from 2021 pretty much begging that DO provide per-bucket level access keys for Spaces. Otherwise, right now, all keys created for an account can access all buckets on that account.

This is the same for Linode, OVH, etc. One key can see ALL buckets, but access can be limited.

I unfortunately had to ditch DigitalOcean because of a number of small problems like this (not taking permissions seriously is a symptom of a larger problem). Other issues included how you can't assign a floating IP to a load balancer, the managed kubernetes is embarassing (to resize nodes, you must destroy and recreate the whole cluster), paltry volumes IOPS, minimal support for DNS stuff, and more.

AFAIK, you don't need to destroy the cluster and recreate it, just add the nodes you want and destroy the old ones.

Another reason DigitalOcean should not be used for production use, or the sole network.

I am looking into blackblaze and other providers.

As seriously, Digital Ocean contains a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.

Aside from this, their Spaces service has had multiple outages, the service seems to be pretty unreliable in general (definitely compared to S3). A quick glance of their NYC3 status suggests they may have finally improved this recently, but I don't know what use case I would have for a version of S3 that goes down multiple times a year.

Indeed - I used to use Spaces for a project but found not just the uptime but teh overall reliability and performance was, at the time at least, markedly worse than AWS.

I don't know if they've improved it, but 5 years ago I basically couldn't delete a DO Space because they had no way of doing a mass deletion. I even got technical support who basically told me I was doing it the best way possible. My delete operation took several months to run.

That is a limitation that comes along with having to be S3 compatible; the best you can do is to combine ListObjectsV2[1] + DeleteObjects[2] (1000 objects in a single API call).

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

[2] https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteOb...

While S3 doesn’t have a REST API to do it, you can trigger it through the console and it applies pretty quickly if you don’t have a lot of objects (hundreds of thousands if I recall correctly) and will apply in the background otherwise. The way to do it via the s3 API is to install a lifecycle policy to delete everything.

At the time, the lifecycle policy didn't seem to be an option on DO unfortunately.

Lifecycle requests are a good call, but being an asynchronous process, it may take some time to be processed, even when using S3. On a particularly large bucket, I've observed it taking over a week in one instance.

The console isn't magic however, it makes the same ListObjectsV2 + DeleteObjects calls in parallel.

DigitalOcean should not be used for production use, or the sole network.

There is a critical flaw in their operations management that will delete all of your long standing droplets with a valid credit card on file and zero issue with the conduct of the droplets or with the credit card.

I used to use DO Spaces and Backblaze, but then I found Wasabi and have switched everything over since. It pretty much has full S3 compatibility, and doesn’t use Minio or Ceph. Mostly just keep adding data, very rarely delete data.

Same here, very happy with wasabi so far.

Why the smaller players can't figure this out? Are they inventing their own S3-compatible storage or using a open source project underneath if?

They are all using Ceph, which is the limitation.

Is this true? That's horrifying

Do you mean Ceph Object gateway? I am not familiar with Ceph but does it have serious limitations?

This is correct, Ceph's RadosGW doesn't support this feature.

For people trying to figure out if their provider is using Ceph, make a request for https://<s3-endpoint>/test and see if the x-amz-request-id header begins with "tx000", if it does, then they're using Ceph.

$ curl -sSi https://nyc3.digitaloceanspaces.com/test/ | grep x-amz-request-id

x-amz-request-id: tx00000947bc21a401c0f02-00651981d1-4b6a0-nyc3d

Painful, slow, costly.

Access request needs to be created, validated, logged, processed, expired or revoked.

Or just validated, processed, and logged.

Minio can do this, so I don't see why RadosGW can't.

Spaces is annoying.

Tailscale uses it as their main host for the repos, snd its down right annoying with no CDN and slowdowns occuring often.

Spaces has a CDN

Well a crappy one then. Because tailscales repo is clearly in the US.

Click saver: this is about access keys, not bucket keys. (Which would have been in the original title...)

The point is that there are no bucket keys. Only Access keys for all buckets. There is no way to issue bucket specific keys. I'm not sure I follow how the title could be misleading.

We're still talking about different things: In S3, for an object known as s3://foo/bar, "bar" is the key. See eg https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...

So the title is ambiguous, it could mean that keys have to be unique across buckets since they share the same namespace.

Hmm, the only issue I've had with spaces is not being able to trivially serve a static site.

What part isn't trivial? It has a CDN + public access. What am I missing here?

That is what Apps Platform is meant for. You can pick static as an option which, if i understand correctly, should do the same thing.

Reply on news.ycombinator.com