Hacker News Reader

Comment by TimSchumann | Hacker News Reader

> I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances.

From a technical standpoint, that statement is false.

Every employee might not have the credentials to, but for AWS to function as it does, SOMEONE inside the company has to have those credentials.

If you change 'cannot' to 'don't', well then we've just gotta take you at your word, which is where we started anyway.

View on HN

> SOMEONE

That's not necessary unless SOMEONE includes computer programs.

Yes, when things go very seriously wrong, I believe AWS can have literal people override that permission, which will leave a mile long audit trail and likely accompanied by an internet scale outage.

The point I’m trying to get across is that the default viewpoint of many knowledgeable developers I know is ‘Of course AWS can’t see inside my EC2 instance because X’ — where X is some magical technology that doesn't exist.

I don’t want to devolve into audit logs and permissions and multi user key signing and wether they actually do or not.

The statement that ‘they can’t’ is 100% false, full stop. That’s all I’m trying to get across.

The technology to do it does exist likely on hardware you possess. The trusted computed platform lets you build a signed OS that encrypts its data using keys on the TPM. Using this, you could build an S3 implementation that stores customer data, but doesn’t let you access it.

It’s probably not a good idea to make a system with no human fallback, but it IS possible with current, non-magic technology.

The reality is that groups of people inside AWS have access to your stuff. A given person might only be on the S3 or EC2 team... but each of those teams can ssh to hosts in production, or has other access that could be used to compromise your data.

Amazon does take privacy and security very seriously, but these systems are run by people. Attacks like the recent Twitter attack could work for various AWS services.

Source: I used to work in EC2 Networking.

Are you sure about that? Most of the aws provided S3 sdks include the option of client side encryption. Not to mention that there are plenty of third party options for that as well. AWS could I guess look at your s3 data, but it will just look like gibberish.

I think it’s pretty clear the person you are responding to is not suggesting AWS can magically break encryption, but rather that they “have access to your stuff” that is actually on AWS. There are plenty of AWS customers running data through, or storing data on, AWS that is sensitive in the form it is in on AWS. If you have an rdbms (database) actively running on AWS for example it is not e2e encrypted. If you are terminating a customer TLS connection on an ec2 hosted web server their web form upload is exposed to that machine. Etc etc.

Except they get audited by 3rd parties on statements like that, and have controls tested. It's not like they're just ... digital ocean or somebody.

Do you have evidence of this claim re DO?

I worked with a DO on an technical issue, and they were steadfastly against me granting them temporary access to our servers even though it would have made the issue easier to diagnose. Cloud provider that verifiably get caught doing this will quickly lose the trust of all their large customers

DO doesn't have a great track record for customer trust. I run personal workload but couldn't recommend it over AWS to a larger company.

  - https://news.ycombinator.com/item?id=23117660
  - https://news.ycombinator.com/item?id=20064169

Sales != Engineering (in regards to the first one), AWS have had similar issues. The second one wasn't good.

https://www.zdnet.com/article/aws-error-exposed-godaddy-serv...

There comes a point where your pricing is so opaque and confusing that it's indistinguishable from lying.

Those people are jealous of AWS.

Reading through that second one, while the inciting incident was certainly pretty bad, their eventual response was, to my mind, all that could be hoped from a company in this day and age:

https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...?

They recognized that their processes were too mechanistic and inhuman, and introduced a lot more compassion and open communication into them—and even chose to spend more money on hiring people to reduce ticket queue wait times.

I'd say that speaks volumes in DigitalOcean's favour.

The audits check that controls are in place, not that the controls are technically bulletproof or people-proof.

Source: Worked at AWS for several years including working on systems that had audit requirements for [secret project where I could not know the name of the customer because I don't have TOP SECRET security clearance].

Nobody said things were perfect or bullet proof. But that they are there, and it's not just 'trust us'. And it's not just single technical controls - the control regimes include mitigations against technical failure and requirements for ways to catch collusion and actions taken outside of authority.

And there are lots of things that many folks at the big cloud providers don't know about their internal threat management and monitoring. Source: Audited most of them for that customer you weren't allowed to know the name of. :)

Yeah. True. I guess what I meant is that just a handful of employees have access to that and they need to have legitimate reasons.

Also, it is possible to build systems such that, no, there isn't a 'root' or 'unlimited permission' or whatever. Or that there is, but it's a multi-person credential.

This is one area where AWS takes things MUCH more seriously than it's competition, and they don't talk about it enough publicly.

The critical factor here is whether there are controls in place to prevent it. Sure, somebody probably could, but what to what lengths must that person go to do it, and what happens when it is discovered? Most things are not technically impossible, after all.

for its faults aws takes data privacy super serious. if you are in support you cant even see attachments customers put on cases without providing auditable justification

and you def cant see in s3 buckets or instances. hell if a customer sends you a link to an object in their s3 youre not supposed to open it

Some group of people on the S3 team likely have root access to the machines where your objects are stored. If you don't have encryption turned on...

You keep making factually incorrect statements. I'm not going to go into detail to refute them, because I don't feel comfortable sharing internal design details and security mechanisms, but your comfort in confidently asserting falsehoods is disconcerting, to say the least.

If you work in AWS security, then you of all people know about the litany of service teams who don't meet their security goals every year.

I find it funny that none of the people here arguing really understand what data is important from a strategic sales point from view and what's not. The customers databases and other crap they store on the cloud. Not really important.

The raw billing information, oh motherfucking yes.

Agree. The billing data gets explicitly or implicitly discussed when various orgs talk about their successes, annual planning etc.

Reply on news.ycombinator.com