Amazon S3 now automatically encrypts all new objects | Hacker News Reader

Amazon S3 now automatically encrypts all new objects

aws.amazon.com

73 pointsredbell33 comments

I'm so impressed at how S3 can roll out changes like this on a regularly frequent basis that impact an unimaginably large number of servers and stored data.

This is an incredible piece of engineering. Almost as impressive as when they made S3 read-after-write consistent a couple of years ago: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

The only thing impressive about this is how long it’s taken them to make this the default. This has been available since 2011 and other cloud providers have had encryption at rest by default for years.

The read-after-write consistency was definitely an impressive change. This however, not so much.

Encryption enabled by default at different S3 compatible providers:

- [x] Microsoft Azure (https://learn.microsoft.com/en-us/azure/storage/common/stora...)

- [x] Google Cloud Storage (https://cloud.google.com/storage/docs/encryption)

- [x] Backblaze B2 (https://help.backblaze.com/hc/en-us/articles/217973398-Encry...)

- [x] Wasabi (https://wasabi-support.zendesk.com/hc/en-us/articles/1150016...)

- [ ] DigitalOcean (https://www.digitalocean.com/community/tutorials/how-to-crea...)

- [ ] DreamHost (https://help.dreamhost.com/hc/en-us/articles/360001089163-En...)

- [ ] Linode (https://www.linode.com/docs/guides/server-side-encryption/)

Is Amazon-managed encryption (SSE-S3) mostly a checklist/regulatory thing? Like, is it mainly protecting against somebody walking out of a DC with a storage device?

Yeah this question. I'm not well versed in Amazon lingo to understand the article

Or some other kind of hack/data theft. That's what it looks like to me.

I don't see it as just a checklist thing: if Amazon ever gets hacked, you wouldn't want your data compromised as a result. Of course, if they manage to steal the keys too, it would still be.

Yes. This is really a non-announcement. S3 has had configurable encryption since at least 2011. And Google cloud storage has had default encryption at rest since 2013. For a decent explanation of encryption at rest check out https://cloud.google.com/docs/security/encryption/default-en...

I would have assumed that's always the case for all storage devices? Or am I naive? I would expected any cloud provider to do that, even if it's just to make disposing old drives easier.

What does encryption at rest protect against with a Cloud provider?

I assume the read credentials must ipso facto decrypt the object. So encryption at rest protects you against an inside job at AWS (smash and grab drives) or government confiscation. Am I thinking about this correctly?

Yeah - physical drive theft I think. I assume that's well locked down (and a crap shoot whose data someone'd get if they stole some drives) but nice to be protected I guess.

I am thinking that it lowers their cost for recycling drives. If everything is encrypted at rest, you don’t need to go to crazy lengths to wipe the drives.

This is it. I help maintain the object storage for a private cloud. Because we have LUKS, preparing servers for decommissioning is done in a matter of minutes (once the servers have been removed from the cluster configuration, of course). We only have to stop all server processes, unmount the disks, overwrite the LUKS headers with /dev/random. Then server management kicks the nodes out of the Kubernetes cluster, then data center management takes care of the hardware.

When I started working for AWS as an SDE, I was hoping it’d be possible to visit a datacenter. I was surprised to find out that I’d require L11 (!) approval to so so! The only L11 in my reporting chain is Adam Selipsky.

I’m told the AWS data centers has red zones, which no harddrive can be taken out of, without being mechanically and violently destroyed first.

How it's protects from government confiscation if key is also stored at AWS (most likely on a different server)? Please explain.

Yes, if AWS has possession of keys and drives and the govt wants the data, I cannot imagine how encryption at rest does anything for you.

Encryption keys are kept physically and logically separate the majority of the time. Generally on HSMs that you can’t really read the private key from. It’s a huge improvement in security if one of their layers of defenses fails.

I read in the past, yet it's still unclear to me. It is a symmetric encryption, right? And as I understand keys are stored at AWS as well. Am I missing something?

That’s awesome. Zero cost and performance drop, too.

S3 is frankly amazing technology; probably the most robust and well rounded piece of cloud tech that exists.

Some light previous discussion, although a lot of it was because that specific article was hot garbage: https://news.ycombinator.com/item?id=34276986

Did files hit the disk unencrypted before this or there were full disk encryption?

Does anyone have rough numbers on s3 scale? Most storage classes have redundant copies as well so 3x things maybe from that. YouTube / S3 - a few services seem like they must have crazy storage scale

If you're building one of these object storages for yourself (say, using Ceph or OpenStack Swift), a plausible configuration is 12 drives with 8 TB each, or 96 TB per node [1]. Let's round that to 100 TB for simplicity. You obviously want to spread your replicas across multiple datacenters, so let's just say you'll put 10 of these servers (i.e. around half a rack full) each into three different datacenters. That's a very modest deployment in terms of floor space, and yet already has a capacity in the low petabyte range (3 PB raw capacity, 1 PB net capacity).

On the AWS scale, object storage servers will probably be filling entire datacenters (or at least an appreciable portion of each datacenter), so it's very easy to see them being well into the exabyte (= million terabyte) scale, though probably not zettabyte (= billion terabyte) yet.

[1] Source: I help maintain an object storage for a private cloud. Disclaimer: These are not the exact numbers for our nodes, but in the same ballpark.

Erasure coding drops the drive requirements massively; they talked about it at reinvent 2021.

There is a talk by the S3 VP on Youtube which mentions some rough numbers, I think it’s from re:invent 2019. Also, they mention 100 trillion objects here https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it...

Any insight into how this is done without impacting performance for reads as they now need to decrypt the data?

I’d guess that the top tier of S3 physical storage is all SSD, and hardware-driven encryption on such devices is virtually free, as well as desirable for lifecycle reasons.

To allow this to double as ”user level” encryption you need to coordinate & manage the keys used vs. just picking something random when the drive is formatted. This is how Apple’s and others’ full-disk encryption has worked for years.

I really would love to know how s3 is implemented. Is it maybe just using a backed up file system and some api?

I read that DynamoDB and S3 share many similarities in terms of architecture.

There's some good discussion of S3 internals in this thread from 2021: https://news.ycombinator.com/item?id=26466338

Thank you

What they won't tell you is that this will use the default KMS key, which cannot be shared across accounts. So your S3 bucket objects cannot be shared across AWS accounts by default, unless you setup a custom KMS key and "copy" everything in place (a standard AWS organisation is made of multiple accounts).

The first line of the article reads otherwise, that it's using the AES256 option, which does work cross-account without KMS permissions.

> Amazon S3 now automatically applies S3 managed server-side encryption (SSE-S3) as a base level of encryption to all new objects added to S3

That's not correct, this isn't defaulting to SSE-KMS. and it's trivial to share KMS encrypted objects cross-account.

Reply on news.ycombinator.com