Amazon storing more than 449 billion objects in S3

gigaom.com

37 points·bbr··10 comments

At what point does S3 break?

Very happy user of S3 and much of the rest of AWS here, but I can't help but ask (mostly rhetorically).

S3 is presumably the largest system of its kind ever; it's not clear to me what its failure modes, thresholds, or weak links might be as it continues to grow. Anything from a breakdown in the dynamo architecture to hard infrastructure limits to failures in operations and management strike me as plausible. What will we see first: data loss, increases in latency, repeated catastrophic outages, or "other"?

Getting some feedback/insight on this topic seemed interesting enough to me that I wrote a slightly-expanded post separately:

http://cemerick.com/2011/07/20/at-what-point-does-s3-break/

I'm curious what their storage capacity is.

Assuming an average object size of 100kb that would amount to 45 petabytes (plus redundancy). But I have no idea what their average object size is...

We have a few million objects stored in S3 and our average object size is only ~40k.

Plenty of people use cloudfront as a streaming CDN, though. Those assets would be far larger, obviously.

Previous link to the actual source (posted here by an Amazon employee): http://news.ycombinator.com/item?id=2783595

If DropBox continue to grow, so will S3

I am calling it here. 1 trillion objects by 2013.

That's a linear extrapolation from the last two data points on the graph, but it seems to be growing super-linearly.

I work for Amazon (and wrote the blog post referenced by GigaOm), and I certainly won't speculate (or place a bet). However, note that the growth from 262 billion to 449 billion objects took place in 6 months.

I'm calling 1T+ objects for Q2 2012. The S3 object count is growing by between 150% and 200% every year.

Reply on news.ycombinator.com