AWS Support able to access any S3 object due to permission change
twitter.com
twitter.com
Posting to get ahead of some of the comment here on security posture.
If you don't trust them, don't use them. Provide evidence to your leadership of malicious intent and provide an alternative and they'll back you.
Executives make decisions based on a small amount of evidence and go with their gut from there. So far AWS has a good reputation but if they're not careful the mood could change quickly. I don't think something like this shows any malice and I have to just assume any cloud provider _can_ access all of my data but chooses not to in order to preserve their reputation which ultimately generates more revenue long term than any single piece of data.
Why would AWS risk breaking the money printer?
If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.
If having all customer’s private S3 data being accessible by all AWS support is happening due to a wrong checkbox being selected, it definitely hints at a bigger problem and erodes my trust in that vendor.
And how they respond to this incident could restore that trust, or continue to erode it even further.
This is never possible if you're using KMS / server side encryption or no encryption at all. Your data on these services is always visible if they were to try to read it, whether that be forging KMS requests to decrypt data or passively snapshotting VM memory for inspection. Technically this might be solved via recent datacenter CPU security improvements but there will always be flaws and 0-days that people with money and a will can use to bypass these protections.
Use rclone with encryption for S3 and consider anything else potentially visible.
It’s not entirely unlikely that AWS was hacked / socially engineered in order to get this privilege in there, perhaps because some specific s3 buckets were being targeted.
As an organization, you need to have a high enough level of protection that these things are just not possible.
So maybe this whole idea of managed IAM policies being installed in all customers’ accounts is just a fundamentally bad idea, and should just be given on a case by case basis.
If your data leaves your network and it’s supposed to be private, encrypt it in such a way only the people who should be able to read it can read it.
https://docs.aws.amazon.com/firehose/latest/dev/controlling-...
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...
Rolls off the tongue
* KMS Encrypted objects would not be accessible because the support personnel would need permission policies that grant `kms:decrypot` permissions to encrypted objects. The only way this could wind up happening is if you are granting the AWS Support principal access in the KMS Key Policy.
* Objects with a default-deny bucket policy could not have been circumvented with the support team's escalated privilege. So if you have a policy that looks something like this, that data was not exposed:
{
"Action": "Deny",
"NotPrincipal": [...]
* Internal Checks. AWS has a lot of protections and checks in place to prevent their support personnel from accessing metadata about S3 objects. They don't have tools to fetch the actual objects unless your really high up the food-chain. Think like, people with a legal or security related reason to need to review data.
Nevertheless, I'll share some nuggets of wisdom I've accrued over the years, in a hopes to save y'all some time:
If you have an NDA with AWS, I'd recommend reaching out to your TAM and asking them about what the potential exposure was; and make sure to ask about the internal access control mechanisms.
But everyone who's concerned and DIDN'T set up data access logging already: 1) Consider turning that on to trace potential disclosures in the future. 2) Open up a case with Premium Support under the CloudTrail, state that you have a security incident and you need to retrieve data events for the time of 2021-12-23 to 2021-12-22. If granted to you, save that sucker in S3 and query it for requests coming from `support.amazonaws.com` in Athena. [0]
Hopefully this helps some of y'all.
[0] https://aws.amazon.com/premiumsupport/knowledge-center/analy...
Any time the wrong permissions are assigned and confidentiality is potentially breached, I think you have to have an incident. Arguably in some jurisdictions, it's a legal requirement to ensure you have a near miss not an actual breach.
Declaring an incident doesn't mean sending out a breach report or anything particularly dramatic, though I can see how as an outsider it may sound that way.
An incident does mean something bad has happened, and requires action. An example of an action can be to investigate the impact, or to shut something down, or to patch something.
This is only true if you are using SSE-KMS *and* are creating/managing the CMKs used to encrypt objects. If you’re using SSE-S3 or SSE-KMS with the default AWS S3 key (aws/s3) there is no key policy to manage.
Of course SSE-C or 100% customer-managed crypto would be immune as well, but under different mechanics.
> Objects with a default-deny bucket policy could not have been circumvented with the support team's escalated privilege.
I would wager this is done for a vanishingly small percentage of buckets used in production. Less than one percent for sure.
The general point you’re making seems to be that if you had a comprehensive, defense-in-depth security strategy for your cloud computing environment then this would have had no effect, and i do agree with that. I just think that in reality this would have provided access to wide swaths of customer data.
Thus, I imagine with AWS-managed KMS keys, in some cases permission to KMS service is not needed to access data.
KMS-CMKs are better in this respect, can be controlled and audited.
Service accounts are not constrained by customer bucket policies. In fact, not even SCP's are restricted by service-linked roles:
"SCPs do not affect any service-linked role. Service-linked roles enable other AWS services to integrate with AWS Organizations and can't be restricted by SCPs."
https://docs.aws.amazon.com/organizations/latest/userguide/o...
Also reminds me of the (hyped?) "outrage" when a former facebook developer stated that they used to have a "default password" that allowed fb devs to log into every account and the media were like "omg they could have logged in and seen your photos". I mean... yeah they're the developers they could always do that even without the password
Not really. Obviously facebook the company can always access your data. Weather or not an individual developer can do the same, which developers can do it, how they can do it, and under what level of supervision this would be is a design choice.
It is possible to design a system with very high level of security and ones with none too. As with any design considerations it has trade-offs. A super secure system might introduce dev and operational frictions which the company might deem unnacceptable. But even with that consideration the question is a lot more complicated than a simple “yeah they’re the developers”.
Unless a business is heavily regulated and checked for compliance, the burden and friction introduced by developer access controls to the data of the software they write is anecdotally not seen as a positive investment in any company I've seen the internals of.
There are so many zero days in regular consumer software, just imagine how many are in closed source public facing Amazon services.
Now multiply that by 100 to get the number of zero days that probably exist in Amazon's closed source dev only back end environment.
Wise rule to live by. I would certainly advise everyone to assume that.
On the other hand geek_at was talking about a slightly different thing. They were talking about how the media criticised FB for having too lax controls on private information. geek_at even called it an "outrage".
We can and should absolutely ask platforms to do better while at the same time playing it safe ourselves with the data we control. There is no contradiction there.
There is an other layer in which it feels we are talking by each other. You mention zero days, and yes those are a thing and yes an insider is in an excellent position to find them and exploit them. Finding them and patching them is a good idea for sure. (For many reasons.) But the FB thing mentioned wasn't about an exploited zero day. It was a company sanctified system and associated work practices. We can demand that a company develop better practices (where not every engineer needs this high of a level of access to do their job) without expecting them to find and patch every single vulnerability.
This was early-early facebook. Like under 10 developers back in the day. But obviously today a facebook frontend developer should not and probably has no access to the database
Maybe AWS should have another object storage product that's specific to sensitive material. I know that would flip Corey Quinn's lid because it would be yet another AWS product (h/t to him for actually have a valuable twitter post and not just snark) but I honestly don't care - I'd rather have extra level of confidence. For this there would be absolutely no data access.
Tangentially, I wonder if this role has been deployed to GovCloud?
I get the idea, but also realize this is fundamentally incompatible with using the range of services at AWS. Fringe-future tech aside, you need unencrypted data to process it and use it. AWS isn't just S3, it's lambda, it's hosting and data science and databases.
Having just read the tweet, as weird as it is to give AWS the benefit of the doubt, I agree with others in that thread. If you KMS encrypt your data, then the engineers will likely only have the ability to see encrypted data. My guess is that there are processes and monitoring in place to ensure this is only used as a break-glass.
Have you ever dealt with automation of AWS resources? There are definitely issues where, by putting incorrect permissions on a KMS key or an S3 bucket, not even root can get your data back. This is likely what this is for. Customers would rather have a non-removable AWS break-glass than their own root account.
All that's changed is that this role is now visible in IAMS and any API calls by Amazon support tools will be logged in AWS CloudTrail - I don't think AWS have any more access now than they did before.
It's obvious to anyone who has had problems with any AWS services (lambda functions that couldn't be deleted, non properly propagating/operational services) that support has access to them via their tools.
https://docs.aws.amazon.com/awssupport/latest/user/using-ser...
"SCPs do not affect any service-linked role. Service-linked roles enable other AWS services to integrate with AWS Organizations and can't be restricted by SCPs."
https://docs.aws.amazon.com/organizations/latest/userguide/o...
Edit: Based on what I know, I'm pretty sure support will not be able see any of the customers data.
Or how broken is the tooling for IAM + S3 + other services (for example Athena and Glue).
Several times I had to explain to support that we do not want s3:* anywhere in our infra because they insisted that is the easiest solution so they do not need to waste their precious (paid by us) time on figuring out which exact permission is missing that I as a customer have no way of figuring out.
Many of us working on cloud infra for 10+ years and we still struggle some times to set up especially new services.
I really like how you conclude that this is somehow the customer's fault. I find it entertaining how the decent support staff of amazon admits that the tooling is subpar, because they got a different system internally to check out why S3 throwing a 403. As a customer we do not have anything just the API.
And no, this is not because the customers are dumb. I can't wait the moment when AWS has to actually compete with other cloud providers because this arrogance has to go.
It happened several times with Glue mentioned by the user two replies above (usually schema registry which requires *s in resource element of the policies to work).
Maybe a more constructive way to look at this would be that people simply do "dumb" things. In customer support where you only see those moments, it might not always seem that way, but dealing with people's simple mistakes is also educating them to do better next time.
If you've ever worked in CS, or known anyone that works in CS, you know that there are an absolute fucking shitload of these people. Often in roles they are unqualified for and with privileges and power no sane person would ever give them.
I find this insulting as a customer. Is AWS usually contemptuous of its customers?
I don't think I've ever called my customer "dumb", and working as a consultant I've seen all kinds of interesting things.
People make mistakes. They're always in a hurry. They may have a hard time understanding ambiguous, complex or incomplete documentation. The interface may be confusing and lead them to bad solutions. Come on, support is there to help.
Take it easy, shall we?
Oh come off it. We've all seen the idiotic things that "users" can do. Someone complains something isn't working. Then you go through the steps to see what they have done, and you think "why would you ever do that?" We've all been there, and if you haven't been there then you just haven't had much interaction with "users".
"Take it easy, shall we?"
As a developer for a SaaS, I have to spend some time on support every day, including for devs who have refused to read our documentation for a particular service we provide (and the only one these devs use). It's frustrating, I know. You should assume that the developers who are your customers are unlikely to be stupid, and are instead just not informed about something or haven't read the docs (maybe they didn't know where to look, or like many, they are too busy to justify spending a day reading the docs for lambda). Best thing to do is direct them to the relevant parts of the documentation and do your best to help those people.
1. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-contex...
But, I have sent off helpdesk requests for things that turn out to be me being very stupid.
You shouldn't compare consulting vs tech support. Especially when your billing rate is noticeably higher.
(I'm not tech support; I'm one of the ones asking said questions of them.)
Everyone who works with customers knows how dumb they can be and how much extra work goes into supporting them.
This sort of personal attack is unwarranted and extremely unfair. AWS is renowned for it's byzantine and ever-changing and expanding nature, to the point it's outright practically impossible to know extremely basic things such as what are you paying for and how much you are paying.
Suggestion, if you want to secure your data don’t put it on other peoples computers, and for fucks sake don’t store your crypto keys on someone else’s computer.
You can also securely storage your data on other servers by using client-side encryption.
Not every business/person has the means or knowledge to have their own datacenter.
Hey but you have audits, contracts, why would you need that? You are effectively saying the same thing that parent comment is. You're just offering a more practical solution.
A national law of the country explicitly tells the company to do so, or a company you are in contract with asks of you to do so. The key that S3 can provide is not good enough for your internal usage,...
Stop looking at everything pure technically because that is not how the real world works.
The stress is on could. AWS is too big to be allowed to fail. Has Facebook seen such severe consequences because of known misconduct? And AWS is in a much more critical role for many businesses.
But it could impact them if there came an issue out of it (someone can prove that AWS downloaded some of their files). AWS doesn't want to go in the news that they look at their customer data as that would impact the decisions of future and current deals of hosting their data on AWS.
I've worked for some big financial institutions and the longest part of the contract with AWS was all lawyers going over what is happening with the data, how AWS has access to it and especially how it doesn't have access to it.
The rest of the points save for keeping the keys on your own hardware is orthogonal to whether Amazon with physical access to your data could access it.
I think we are both in agreement that in most cases the data isn’t worth accessing which is the real world protection most data on Amazon has.
But even in the world you are imagining where AWS is peeking at customer's data willy-nilly, I have to imagine you don't believe that every tech support representative should have default access to every AWS customer's storage data, do you?
Even a dishonest unethical company that created backdoors for its employees would surely gate their backdoors.
This change (a mistaken one that was rolled back immediately) would have given the keys through the front door to presumably thousand low-level employees.
BTW, AWS spends a long time talking about how verifiably they do not have access to customer data. If you're interested in crypto (otherwise not sure why you are referencing it here), this kind of thing should be right up your alley: https://www.youtube.com/watch?v=4J8REvs7zaY
They also have regions in the US where they verifiably DO have access to your data.
Both points of access are verifiable by their compliance with the law in those countries ensuring that the government can access that data.
If you use their CA or EU locations it’s conceivable that they’ve developed separate software that actually protects your data but I would hazard a guess that they use the same backdoored software there once it has been sufficiently beta tested in us-east-1
It’s hard to decipher the AWS Data Privacy policy:
https://aws.amazon.com/compliance/data-privacy-faq/
In section Who Owns Customer Content, it’s implied that AWS doesn’t access customers’ data:
“As a customer, you maintain ownership of your content, and you select which AWS services can process, store, and host your content. We do not access or use your content for any purpose without your agreement. We never use customer content or derive information from it for marketing or advertising.”
Here, the statement is we don’t access customer’s data without customer’s agreement. But customers do agree with ToS in which different statements might be included.
In another place it’s mentioned, if governments request, AWS will comply. Considering that governments obviously request access from cloud providers, and obviously require them not to disclose the backdoor, I came to the conclusion that governments have unconditional access to data held in data centers of companies such as Amazon, Google and Microsoft.
As for non-government access, there are claims that Amazon uses data or metadata to launch competing products. If I recall correctly, AWS may collect some data presumably to maintain and secure the platform and fight abuse. The counter argument is that, AWS will not risk a profitable business; but AWS is too big to be easily impacted and might act on good opportunities.
It would be great if people working in AWS or similar platforms could chime in.
Amazon's product people also don't look at third party seller statistics to decide which products to sell themselves. Until they got caught doing just that.
To assume that they don't look at data feels naive.
A German super market chain with online ambitions has a rule that nothing touching their pipeline can be hosted on AWS. Want to sell them SaaS? You can't run your nodes on AWS. I consider that to be reasonable.
That's completely different and separate issue from AWS customer data (for example someone running e-commerce software on a linux VM).
3rd party seller sales pay Amazon commission on each sale - deciding which products are selling well is just a matter of Amazon doing a SQL call on their own sales database (much like a physical retailer may see what brands/products are selling well)
We built are service in similar vain. Nothing can reside under authority on non-EU entity, and nothing can be hosted on servers owned by non-EU entity. This effectively removed AWS, Azure, GCP and Alibaba.
And still, we had plenty of choice.
We specifically picked “boring” cloud provider. No thrills cloud vendor which has core business in building infrastructure rather than snooping on customers.
Would having a 3rd party host the services in the EU meet your requirements? Or having data residency restrictions with strict key management, EU based support, and access transparency/approvals?
IMO Google is also taking this seriously, but I am genuinely curious if any off the above would meet your requirements.
Which one? I would like to support non-AWS-companies.
I was a stakeholder in an early cloud negotiation back in 2010-11, and one of the key issues was the support personnel. Ultimately, they have access to your data.
If all of the above is true, is it time to panic? Perhaps maybe that why the cloud is somewhat of a scam anyways?
Of course there are also risks with self hosting, depending on your competence and the service.
I have a Vm on the cloud I use for running nightly nmaps against my network range, when that breaks it’s not the end of the world, if the VM provider hikes the price, or goes bust, I can migrate it in an hour by running a script, thetr no sensitive information on there, it makes perfect sense to run on a VM.
Would I entrust my journalist’s files when they are investigating corruption in Amazon? No.
- SSE-KMS
- SSE-S3
- SSE-C
Without having an AWS support person test each type and report back, one must assume that the only bulletproof s3 encryption methods are client-side (where you handle encryption and decryption yourself and they just store the blob) and SSE-C (where AWS don't store your keys, you send them in every bucket API request). But even that latter method has other caveats:
- What does the S3 service log? Who can access those logs?
- Where does TLS for your S3 https request get terminated? Who can view the traffic?
I'm assuming that this isn't just a regional issue, and that any AWS Support person globally could access buckets in any region. If so, then that's a big deal. If you're in Europe and your bank or healthcare provider is an AWS customer, how much trouble could you cause them (and by extension, AWS) right now?
Furthermore, with the antiwork movement and backlash amongst employees for their treatment of warehouse workers, one cannot guarantee that an AWS worker wouldn't do something to hurt the company.
Amazon need to head this of with a very thorough explanation of what happened and what was exposed directly and indirectly.
Since this is symmetrical encryption we're talking about, let's just be completely aware that the technical possibility to also store the encryption key definitely exists. It would violate the terms of service, of course.
For those who don't know how SSE-C works, it's that you send both the unencrypted data and a key in a request. AWS will encrypt the data with the key, and store it encrypted. To get your data back, you supply the same key in your subsequent request. AWS will decrypt the data using the key, and send the unencrypted data back to you.
During both those times when you gave AWS your key, you entirely trust that they will not also happen to store it for their own use.
It is normal practice to have a 3rd party access to your technical infrastructure (for example for purpose of support/maintenance). I was once contracted to maintain database for another company. You sign NDAs, you sign penalties, you sign your children to slavery and the right of first night with your wife. You know, standard business practice.
But if you care enough that you would not have contracted 3rd party access to the data, client-side is the only solution assuming the client is under your sole control.
Of course Amazon could just give themselves access to decrypt with your KMS keys too, but that didn’t happen here.