But, if you see something, say something. This crap continues because there are too many folks that are happy to help support immoral business practices for some extra scratch. This isn't all on you in particular but when google folks started raising hell about Chinese censorship the company was forced to move. We all have the power to withdraw consent over how our labour will be used and, as software developers, we've got a strong enough employment market that we have real power to help make companies behave better - power that folks working in the warehouse are absolutely deprived of.
Amazon needs to be properly taxed so that this crap doesn't happen anymore.
The idea that they shouldn't pay taxes simply because they're large should absolutely enrage everyone.
https://www.cnbc.com/2019/04/03/why-amazon-paid-no-federal-i...
There's specific credits/exemptions in the tax code that they are able to exploit (and perhaps they can only exploit some of them _because_ they are a big company), but it really isn't about their size.
Can we please stop arguing like influencing is only true if it is done in the most direct way (similar to the quid pro quo debate). Obviously if big company lobbyists try to get tax law in their favour they are not pushing for "please write a law which exempts companies over N employees from taxes." They push for laws that sound innocent but only they will be able to take advantage of, just like it is at the moment. The outcome is still the same they pay less or zero taxes.
You're a fish, eat other fish and evolve into a shark, you eat other sharks and become a whale shark, you start eating everything and then become godzilla.
A whistle blower isn't going to fix this. This is the system. The system MAKES godzilla sharks like this.
Oh yeah sure a whistle blower will do what? Get amazon fined for how much? Then they just change tactics. Outsource. Make agreements and partnerships and farm out doing the same thing just with different proxies. I mean come on man this is a company that can buy other countries.
And let's not forget Microsoft was pulling the same shit until they got put under the same charges and then all of a sudden years later after Bill got tired of stabilizing his empire and making sure it would live without him he became a saint all of a sudden. Cuz like yeah if I was richer than 99% of the people on the planet yeah I could start being a nicer person and shit too.
I think taxes aren't really a solution anyways - fines might be but taxes would hurt honest players just as much as dishonest ones. What they did is (AFAIK) illegal and needs to be punished, if it isn't then there is no incentive for them to correct their action.
But there's probably other superficial business data that's helpful to evaluate that.
2. What measures that you know of is Amazon implementing to make sure no employees across all teams are having access to said resources?
Basically is preferable to get a bullet in the head than to ever reveal or tamper with customer’s data.
I cannot answer your question about who has access or not but I’m telling you what’s the culture when it comes to customer’s data.
At the end of the day I was just another IC doing menial work so probably not a good reference, but that was my experience
If someone has access to customer’s data for their work they have to do a bunch of extra training and do other stuff. Potentially sign some things and there’s probably a different way to authenticate. I really don’t know because I never had to do that and nobody I knew had that type of access but I heard when you do you have to put with more things.
i don't see anybody claiming that amazon is harvesting data from inside their customer's infrastructure. amazon has a lot of data that's "amazon's data" that would tell them about businesses that are operating on AWS that might be ripe for competition.
For example, they know what your AWS bill is, and how it's been trending. If you pay a huge bandwidth bill and it goes up 50% each month, they know you've got a business model that's working and that they can undercut you on one of your big expenses.
However, metrics like AMI popularity is Amazon's data... and that definitely informs first-class AWS product development. Once the company identifies a business opportunity, different teams often investigate "build" and "buy" options simultaneously.
Same goes for retail - Amazon works backwards from high-margin categories to identify opportunities, then pursues investment in existing brands versus spinning up products under the company brands.
This all feels very monopolistic to me, but regardless it's worlds apart from the accusation of stealing private information through faux investment offerings.
From a technical standpoint, that statement is false.
Every employee might not have the credentials to, but for AWS to function as it does, SOMEONE inside the company has to have those credentials.
If you change 'cannot' to 'don't', well then we've just gotta take you at your word, which is where we started anyway.
I worked with a DO on an technical issue, and they were steadfastly against me granting them temporary access to our servers even though it would have made the issue easier to diagnose. Cloud provider that verifiably get caught doing this will quickly lose the trust of all their large customers
- https://news.ycombinator.com/item?id=23117660
- https://news.ycombinator.com/item?id=20064169https://www.zdnet.com/article/aws-error-exposed-godaddy-serv...
Those people are jealous of AWS.
https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...?
They recognized that their processes were too mechanistic and inhuman, and introduced a lot more compassion and open communication into them—and even chose to spend more money on hiring people to reduce ticket queue wait times.
I'd say that speaks volumes in DigitalOcean's favour.
Source: Worked at AWS for several years including working on systems that had audit requirements for [secret project where I could not know the name of the customer because I don't have TOP SECRET security clearance].
And there are lots of things that many folks at the big cloud providers don't know about their internal threat management and monitoring. Source: Audited most of them for that customer you weren't allowed to know the name of. :)
This is one area where AWS takes things MUCH more seriously than it's competition, and they don't talk about it enough publicly.
That's not necessary unless SOMEONE includes computer programs.
Yes, when things go very seriously wrong, I believe AWS can have literal people override that permission, which will leave a mile long audit trail and likely accompanied by an internet scale outage.
I don’t want to devolve into audit logs and permissions and multi user key signing and wether they actually do or not.
The statement that ‘they can’t’ is 100% false, full stop. That’s all I’m trying to get across.
It’s probably not a good idea to make a system with no human fallback, but it IS possible with current, non-magic technology.
Amazon does take privacy and security very seriously, but these systems are run by people. Attacks like the recent Twitter attack could work for various AWS services.
Source: I used to work in EC2 Networking.
and you def cant see in s3 buckets or instances. hell if a customer sends you a link to an object in their s3 youre not supposed to open it
The raw billing information, oh motherfucking yes.
Technically, its absolutely possible. Most likely you'll just need a support ticket or bug, and then you can troll around as engineer.
Also, security teams also usually have access to stuff when things get interesting.
Better to say that access is strictly on a case by case basis and monitored thoroughly.
Ideally customer is notified each time it happens - that would be cool, but likely technically not possible since data ends up in so many systems (like logs, SIEM, telemetry, debug files, backups, data scientist desktops,....)
You're underestimating the investments that AWS (and Amazon at large) make in to security, confidentiality, and auditing. You're also missing a fundamental implication of building AWS on AWS primitives.
As a relevant example there is only one AWS IAM and one CloudTrail. It's a core tenant of AWS IAM to put that control and root of trust in to the customers control. That means when developer support is helping with your ticket they do so via your accounts AWSServiceRoleForSupport role. That means you can control whether that role exists, which principals can assume it, the capabilities it has, and you can see those same API calls in your CloudTrail logs. Although it would make support difficult you're welcome to delete that service linked role and prevent support.amazonaws.com from assuming said role in your account.
https://docs.aws.amazon.com/awssupport/latest/user/using-ser...
I'm not talking about Amazon SSH into your EC2 instance - but of course they can do that also - at will, without you authorizing it.
Lower level disks, logs, hypervisor, telemetry, etc.. are accessible beyond your control.
Of course there are lower level primitives. And if the public documentation and observed behavior is insufficient I encourage you to inquire more about the various compliance, certification, and third party auditing programs in place https://aws.amazon.com/compliance/programs/. However at some point this approaches solipsism and I can’t prove a negative in a HN thread.
> I'm not talking about Amazon SSH into your EC2 instance - but of course they can do that also - at will, without you authorizing it.
No. Extraordinary claims need evidence. Either you have serious non public information counter to many AWS statements ... or you misunderstand some fundamentals of SSH and public key cryptography.
> Lower level disks, logs, hypervisor, telemetry, etc.. are accessible beyond your control
I would encourage you to read the AWS data privacy statements https://aws.amazon.com/compliance/data-privacy-faq/. Particularly the definitions of “customer content” and the “shared responsibility model.”
Depending on how the infrastructure is built, or what the particular service set up, it may not even be possible to gain access to specific data without extraordinary means, possibly involving replacing physical hardware.
https://www.bloomberg.com/news/articles/2019-07-29/capital-o...
Quote:
Capital One Financial Corp. said data from about 100 million people in the U.S. was illegally accessed after prosecutors accused a Seattle woman identified by Amazon.com Inc. as one of its former cloud service employees of breaking into the bank’s server.
While the complaint doesn’t identify the cloud provider that stored the allegedly stolen data, the charging papers mention information stored in S3, a reference to Simple Storage Service, Amazon Web Services’ popular data storage software.
AWS customers that want to avoid this vulnerability should disable IMDSv1 as per https://aws.amazon.com/blogs/security/defense-in-depth-open-...
The EC2 instance credentials via the metadata url is public documented functionality. Its how things like the SDK “just work.”
The S3 bucket policy, instance creds, and (inferred) overly permissive IAM policy is all public documented functionality. This looks like a simple case of an initial intrusion being escalated via permissive configuration and controls. There would be no story if the suspect had not been employed by AWS in the past.
Disclaimer: Im a Principal jn AWS but have no direct or inside knowledge of this incident. Everything I know or have stated here is public record (eg the indictment) or public AWS docs.
There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.
Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it.
Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it.
Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited.
Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.
What the technical implications are is moot, the process that hands out these credentials should not be accessible to anybody but the customer. It implies that AWS personnel can impersonate customer representatives or processes run on behalf of those customers. That's a serious problem.
In all the years that I've been co-locating I do not remember a single instance where a representative of the hosting facilities that I've used gained access to our data or hardware without my very explicit permission.
As for audit logs: they are only as useful as those inspecting them, and more often than not are entirely passive until required for evidentiary purposes.
but no 1 can export the private key itself. and key policy changes are vry heavily audited by aws (and can be by the customer, too). this is all proven by the 3rd party audits aws receives
Somebody can access the key hardware but they can't extract the actual key out of that. However, I've never met anyone with that level of access - and AFAIK you have to go through various security clearance and approval before such human intervention is permitted.
There's no such thing as perfect security - but KMS is as solid as I can see with centralized key management at the moment. And customer can roll out their own key server as well that is managed in your own data center.
In short IAM controls everything, there is no “back door” or universal admin access, and KMS is used to perform sensitive operations NOT handing secrets to arbitrary (internal or external) consumers.
Rather than being a serious problem I think it's more on an obvious fact. AWS personnel build services that specifically exist to act on the customer's behalf with delegated credentials. Any time you configure a managed service to run with an IAM role, that service assumes the role and acts with the credentials granted to the role. AWS personnel have access for emergencies to the systems running their services, and by their very nature those services are in possession of customer credential sets for the IAM roles that the service is configured to use.
For example, a Lambda Function can be configured to run with a particular role. When the Lambda service goes to run the function, it fetches the role credentials from IAM and makes them available to the running Function. It could not be otherwise, because the purpose of a managed service like Lambda is to carry out actions on behalf of the customer. The role's credential set is as much a piece of data as the code of the function to be executed.
But leaving all of this aside, of course AWS personnel can access any and all data you store in their systems. They are legally obligated to turn whatever you have stored over to the courts in response to a warrant. So not only could they gather up your data by this roundabout method of misappropriating credential sets, they must have a way to simply access all of the data directly in a way that doesn't appear in audit trails. I assume for simplicity that the IAM service simply has an endpoint accessible to the company's lawyers that will serve up forged customer credentials on demand.
KMS is very clear about it's usage and what it involves. It's obvious that with Symmetrical Encryption AWS obviously needs to know the other end of the key at some point so that it can decrypt the data.
However, as customers can't even export these keys and the whole system is based on using KMS to actually perform the decrypt operations it is a non-starter. It's a lot more secure than most infrastructure which probably encrypts locally but is stored in a broom cupboard with a $10 lock.
Its worth noting that even symmetric keys dont imply direct access to the secret itself. You can instead use the highly controlled secret material to derive less sensitive material. For example a hash derived from a known input + the secret. A third party can use this to prove that two other parties both have/had access to the shared secret. But the third party never needs to access the secret itself.
Theres a great example of this in the chained hashes that make up an AWS sigv4 API request signature. https://docs.aws.amazon.com/general/latest/gr/sigv4-calculat...
In order for AWS to comply with LEO's they must have some way of accessing data, that is NOT to say they do this for business purposes.
And of course, you're always vulnerable to someone with access to the physical host of an EC2 instance where your workload is running. Only GCP AFAIK offers an encrypted-in-processing compute service, and it's like a week old.
https://cloud.google.com/blog/products/identity-security/int...
We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money.
No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.
Even if the customer had a misconfigured S3 bucket that was exposed to the public, it would still constitute as accessing customer data you're not meant to see.
As other users have provided insight on, everything you do as an Amazon employee basically leaves a trail with your employee ID, even if you had access to private information (which you wouldn't basically because it's locked behind several layers of security). Fireable and sueable offense which Amazon would definitely not allow, let alone endorse.
That might be true in retail, but it wasn't anywhere close to true in AWS. When I left most engineers still had SSH access to the production hosts (and a not-insignificant portion of operations relied on that fact).
There are many easy mechanisms to audit and monitor SSH sessions. So... no?
Before going into our AWS production S3 buckets, looking at our databases for customer lists AWS seems to be pretty careful to get an OK.
Now we are being told that production customer data was normal to trawl? How in the HELL are they passing all their certs with all production data so wide open. I do customer managed keys - I mean, this is a HUGE backdoor.
Either Amazon is lying about AWS security (and has fooled a bunch of others) or routinely trawling AWS customer production workloads for data is a false statement.
How are they trawling through all our buckets and databases without codepaths for access?
Again, they aren't talking about amazon data (ie, billing, support inquiries etc). They are talking about customer production data.
This isn't amazon billing data etc (obviously I expect they analyze that carefully given they bring in billions from billing). To ROUTINELY go through AWS customer production datasets is beyond all reason.
Don't ask someone to admit to felonies over email. Tech employers have a LOT of power to investigate their employees' digital behavior.
How about this instead: https://www.nytimes.com/tips
But it would be helpful if you broke that down a little more than 'trawling customer data', because at the most innocuous, if they're just looking at what's publicly selling on Amazon, what goes into sales rank, that seems acceptable, to me anyway.
In this case, tech investing and online retailing are not the same industry. Amazon is using a dominance in one to fund the other, which then it uses to either drive valuations of potential competitors down or to simply outcompete them.
And that's a plausible antitrust problem.
I'm normally not in the Amazon haters camp. Most of the time I'll defend them against the typical charges of unfair competition. Not this time. This is sketchy.
https://www.gnu.org/philosophy/who-does-that-server-really-s...
It's not. And there are plenty of trainings inside of Amazon to make you aware of that. It is your fault, in the end, to not report your team. I have been on several teams at Amazon and this would always be an absolute no-go. It's already difficult to even get basic ideas about customer data, things that you would consider "essential" to improving the customer experience.
Talk about all time gaslighting. It's the managers/directors job to ensure compliance, not normal employees.
That is totally false.
Conspiracy requires two elements: an agreement to commit a crime, and an act in furtherance of said crime. There is nothing unlawful about looking the other way. You might be a scumbag, but that's a different problem.
The elements of criminal accessory require one to harbor, conceal, or act in such a way as to help someone avoid or escape arrest or punishment (CA law here, other states may be different). Again, merely "looking the other way" is not an act. Otherwise, anyone who merely witnessed a crime could be charged with criminal accessory.
That said, corporate policy might be quite different. If I look the other way while a colleague violates customer security policies (and I'm aware of such violation), I can justifiably be fired.
*Not giving legal advice, seek licensed counsel in your jurisdiciton.