Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects | Hacker News Reader

Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

aws.amazon.com

150 pointsyarapavan77 comments

If you're curious about some "S3 internals", this is a talk I gave in 2009 [0], based on a ton of interesting stuff that S3's then-GM Alyssa Henry [1] shared with me.

[0]: https://vimeo.com/7330740

[1]: http://itc.conversationsnetwork.org/shows/detail5273.html

This recent interview with Werner Vogels[1] touches on S3 design priorities as well:

https://cacm.acm.org/magazines/2021/3/250706-a-second-conver...

https://news.ycombinator.com/item?id=26365873

[1] which you commented on, but others might not have seen.

>Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled

I have a tough time taking him seriously reading this - I'm not sure if he's being intentionally ignorant or intentionally misleading.

I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.

Their goal is 5-9's of UPTIME - the expectation is 100% data durability. You would get fired if you architected a solution keeping two copies of data in one datacenter.

Object storage absolutely has a place and I'm happy amazon was able to push a quasi-standard for the industry. Object storage prior to S3 was a hodge-podge of proprietary plays (EMC Centera, Bytecast, etc) and sort-of standards that nobody really used (CDMI). I just wish they did a better job of fairly representing it vs. turning every opportunity into a sales pitch spreading FUD about the alternatives.

Giving him the benefit of the doubt that he knows his own customers here.

He said:

> Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center

You said:

> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters

His customers don't equal your enterprise customers. You are both right most likely.

Given that part of what I do is help enterprise customers with cloud adoption, almost all of whom are fortune 500 (and on his radar), with an AWS presence, I strongly doubt it.

> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.

Longtime consultant here with BCP/DR insight into 20+ large F500 companies... I think you would be seriously surprised at how common it is for even enterprise customers to not bother with multi-DC or even offsite backups. And even among the ones that do, many of the ones that are non-cloud-based are so immature at it that I would not put money on their backups being restorable if needed. And this is even more true for your “we’re a startup, we don’t have time to worry about backing up our data!” companies, which are in abundance.

For a small peak, go look at how many people were freaking out about losing their entire business due to the loss of a single OVH data center.

Of course, as a consultant I do naturally skew towards customers that need help with this stuff, so my perspective is probably biased towards the companies that are worse off in this regard. But they’re definitely out there.

You snipped out the part of the quote that answers your question.

"Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled. And it trumps everything."

> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters...

Except, in AWS' case, each AZs (Availability Zones) is made up of upto 8 DCs (Data Centers), and each full-region has at least 3 AZs and 2 Transit Centers. Amazon S3 replicates data to 3 different AZs (which, I am guessing, is in addition to replicating it across DCs in a single AZ for 'eleven 9s').

With S3 cross-region replication durability may shoot up to 'sixteen 9s'? 100% durability, if it exists, is something the major Cloud providers are yet to offer?

ref: https://maisonbisson.com/post/object-storage-prior-art-and-l...

>You snipped out the part of the quote that answers your question.

Replicating across two datacenters isn't "really good" though, that's considered table stakes.

>Except, in AWS' case, each AZs (Availability Zones) is made up of upto 8 DCs (Data Centers), and each full-region has at least 3 AZs and 2 Transit Centers. Amazon S3 replicates data to 3 different AZs (which, I am guessing, is in addition to replicating it across DCs in a single AZ for 'eleven 9s').

In AWS' case, the 8 DCs in an AZ are directly adjacent which isn't really useful for anything beyond metro (active/active) availability. I haven't seen a DR plan that doesn't have a hard requirement for the secondary copy of data to be outside of the metro loop/blast radius generally in a different state.

> That’s almost 13,000 objects for each person in the world, or 50 objects for every one of the roughly two trillion galaxies (according to this 2021 estimate) in the Universe.

That’s mind boggling to think about. I wonder how much paper would’ve been needed to store 100T objects on paper. It’s like the new Library of Alexandria.

> It’s like the new Library of Alexandria.

Presumably the Library of Alexandria didn’t have access control lists

And look what happened...

There's a joke in here about OVH...

For sure ;) but I also imagine that the number of objects with a public-read ACL (potentially mis-configured) would still dwarf Alexandria.

If it did then Caesar wouldn’t have had permission to burn it.

> It’s like the new Library of Alexandria.

Would be very unfortunate if the S3 story ends like the Library of Alexandria.

What’s way more mind boggling to me is that there are at least 260 galaxies per person in the world. I am not shocked at all that S3 is home to any number of cat pictures many of which are probably duplicates.

Found the IPFS user.

I don’t know, but to me 260 per person seems less than what I would guess.

> S3 is designed to provide “11 9’s” (99.999999999%) of durability

Has anyone lost data on S3 or know anyone who has?

There's different storage tiers, with different redundancy. So if anyone has an anecdote, would be curious to know which storage tier was used (e.g. single availability zone)

If it works as designed I guess 1000 have; but how would you know. Did your perfect code running on an perfect CPU and RAM connected though a perfect network upload the file with 11 9's certainty?

This is a real problem, but it is partially accounted for:

> To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error. Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value.

It’s not as obvious with multipart uploads where each chunk is (iirc) individually checksumd and stored, but only a single md5 is returned in the “etag”

Not to me, but it’s happened.

https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanent...

I imagine that in most cases it's a giant archive - data that no one really cares about (I mean they do in principal). Then there's probably like 0.1% that's probably operationally important data - which is actually huge.

This is just a guess based on my own usage of it over the last 15 years at different companies.

Yes. Once provided extensive documentation and debugging info they did refund portions of bills too.

S3 may be my favorite AWS product. It's the epitome of "It just works" design.

One of the tenets for the S3 team is/was "indistinguishable from perfect." I guess they're living up to it.

https://www.allthingsdistributed.com/2010/05/amazon_s3_reduc...

One of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3.

   aws s3 sync ...

And you are in business.

When people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer.

But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)

Minio and other S3-compatibles make moving off of AWS that much simpler, though obviously it might be difficult if you're using Amazon-only S3 features extensively.

Especially if you layer on additional functionality, like triggering a lambda function to resize an image when it hits the S3 bucket.

Yeah, even if you don't use any vendor-specific features, there's data gravity—It's hard to move your data from one location to another. AWS and others increase this cost even further by making egress so expensive.

Precisely - vendor lock-in generally happens at the data level. For example, AWS finally turned off their last Oracle DB server only a year or two ago.

> tens of millions of requests per second

How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware?

Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?

I’m wildly guessing here, but I assume that there are many tiers of load balancing to turn your initial request into the message that reaches the thing that gives back your file data. They also likely do have specialized hardware in various parts of the AWS infrastructure.

A service that can handle tens of millions of requests per second can probably outscale most DDoS attacks without further engineering.

> At some point isn't contacting this service really just contacting a handful of IP addresses?

"Handful". S3 has 283 subnets allocated to it, biggest one being a /15, so there are some few hunderd thousand IP addresses reserved for S3. While impossible to say how many of them are actively in use, I imagine large fraction of them.

The AWS IP ranges are documented in https://ip-ranges.amazonaws.com/ip-ranges.json

I am worried.

We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing.

It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.

At this rate, won't AWS run out of storage space before they run out of UUIDs?

or the waste heat boils the oceans... or something

Is this a real problem? It should be possible for Amazon to update their code to generate IDs a different way, and for them to update their client libraries well ahead of a time when this becomes a problem. Yes, code that relies on the ID being a particular format will break. But users have to deal with deprecated client library features all the time. It's reasonable to give a few year grace period before old APIs are shut down.

UUIDs are usually not simply random numbers. So it's possible to tell when a particular standard of UUID is used. Amazon's code can branch based on this detection, or even something simpler like "UUID-type", a new field that would be initialized by default to the old UUID type.

I simply don't see this as a problem for anyone.

All AWS need to do is add something like a namespace. Shazzam! problem solved for your lifetime.

Is there any specific reason one could not just put two UUID-4's alongside one another?

Also as another commenter mentioned, if they are tied to a namespace prefix, that delays collisions.

> All of those decisions remain valid and code written on launch day will still work just fine today.

And this is why GCP will never come close to AWS.

S3 is a great service but it's a double edge sword. It's easy to start with and "cheap" until you scale and then you come to realize you pay a ton for writes and you have a latency of 100ms+ to retrieve data. Even S3 select which on face value seems great is absurdly slow. Then they offer you Fsx or elastic cache as a proxy.

I wonder how many objects have been lost and how many have been corrupted.

Do they de-dupe at all?

oops, I think you found their secret sauce for cost control and economies of scale.

There is no way

prove me wrong.

You have the burden of proof for your claim.

It's obviously technically possible.

"Technically possible" that they do it, sure. But I would disagree it is possible that it is S3's "secret sauce".

Because?

... this is on you to prove it is their secret sauce. Throwing out some random feature and saying it is their secret sauce and putting the burden of proof on somebody else makes no sense.

But it would be because 1) they probably don't have many duped objects to begin with 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth it

> 1) they probably don't have many duped objects to begin with

That's crap reasoning. You have no way of knowing that.

> 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth

Would it really be that complex? I kinda doubt that.

Read latencies wouldn't be affected - except being improved.

Write latencies - the deduplication can happen after the write has been confirmed, so no extra latency added.

Whether or not it's worth it - I think it's very hard to to estimate the deduplication factor. My guess is 1.5x. At that point it would save 33% storage. That would instanly make a whole lot of extra complexity worth it.

It's like you are living in an alternate reality

...because?

I dunno, I kinda think you might be out of your depth on this one, based on that response.

Also: none of your few previous posts are tech-related. They're all about personal finance.

So just to sum up your argument: You think 33% of objects in S3 are dupes and that removing those dupes would be S3's secret sauce?

Doubtful since they have an option for at-rest encryption.

It's just that, an option, not the default. People tend to use the defaults.

Yes. The contents of zero-byte objects are deduplicated. (The metadata, of course, are not.)

I wonder how much of ceph inspired s3...

Very little, I'm sure. S3 and Ceph were both introduced to the world in 2006, which means they had been in development at the same time - or perhaps S3 even a bit before, since it was in a far more mature state by then. Also, while S3 and Ceph's RADOS both supposedly handle "objects" they're very different kinds of objects with very different APIs. And of course the metadata layers are completely different to serve different needs. It's practically certain that they both drew on the same common ancestors, but it's more likely that lessons learned from S3 informed subsequent development on Ceph than the other way around.

Wow, has it really been 15 years?

I am really getting afraid of this centralization of data.

At the same time, the S3 api is so ubiquitous that there implementations for it in multiple clouds, it frequently comes with hardware storage vendors as well, and there are projects like minio that provide a solution to host your own s3 compatible data architecture.

All things considered, I’m much more concerned about the centralization of eg browser technologies than this.

Eh… the thing about cloud storage is that it can be efficient in ways that smaller players cannot. A system like Amazon S3 can pack its hard drives to the brim with data and pay a fairly small overhead for replicas & unused drives.

If you try to decentralize the data, or keep your data yourself, you end up with something that uses more resources, has worse durability, or is worse along some other axis.

I am not so worried about centralization yet. There is still healthy competition between different cloud storage vendors. My impression is that cheap storage is used to make other cloud services more attractive—which means that Amazon’s incentives here are somewhat aligned with mine (as a customer). Personally I am much more worried about the centralization of the shopping experience.

I think it’s fine as long as people know what they’re getting themselves into. I’m sure I would’ve found whatever was happening on Parler abhorrent, but in the same way Merkel and Macron were concerned by the tech deplatforming that followed the capitol riots, so am I.

I don’t think filecoin will ever demolish a fantastic business like S3 based on stealing away customers. The interesting value proposition from the protocol is probably more about censorship resistance and adjacent issues.

Can we update the title? 100T reads as "100 terabytes" instead of "100 trillion" (as per the source).

100 terabytes is 100TB. The T prefix without a unit means 100 trillion, just as 100TB means 100 trillion bytes.

I think it's clear as-is.

I read it as 100 trillion. 100TB would be the 'right' suffix for terabytes IMHO.

> 100T reads as "100 terabytes"

No, it doesn't. Not more than it reads terameters or teragrams. SI prefixes were not invented specifically for computer memory.

100 terabytes would be '100TB', not '100T'.

Further, '100 Terabyte Objects' wouldn't really make sense as a sentence anyways.

It should be read as "100 tera-objects", meaning 100 trillion objects. Writing "K" seems to be so widespread these days (Youtube likes/dislikes are written as 1.5K, for example), I'm surprised there is any confusion here.

Reply on news.ycombinator.com