Amazon S3 - Bigger and Busier Than Ever | Hacker News Reader

Amazon S3 - Bigger and Busier Than Ever

aws.typepad.com

44 pointsjeffbarr15 comments

I'm pretty certain that I store almost a billion objects into S3 every month (yes: the requests for that cost a lot)... does that make me a noticeable percentage of S3's object count? Seriously now?

Don't keep us in suspense, what are you storing?

I wish it was interesting, or even sane... one day my website got like a million more users /that day/ and every database mechanism I had to do some of the user statistics I was keeping totally failed: the website ground to a halt. So, rather than decide what I really wanted to keep, I frantically developed a little library that stored all of that data into S3 as a giant key/value store, with my own little makeshift inverted indexes. Needless to say, this was stupid, and when I next checked my S3 bills I noticed I was doing a billion puts per month (which costs like $10,000). Lukily, this has only been a few months, and I'm now fixing the situation during a giant cost-cotti g endeavor... but yeah: omg that was stupid.

(typed on my iPhone and therefore a little shortened: hacker news comments suck on the iPhone)

Interesting, thanks for sharing. I'm a packrat myself, but I guess something like S3 where you actively pay for storage and transport shows you very clearly the cost of being a packrat--there may be value in what you keep around, but it's not free to keep around, and it's not free to move it to storage.

Is it just me, or does this post leave you a bit confused. What does 292 billion objects mean? I know an object is a file stored in a bucket.

But rly? If Amazon wants to convince developers to use their platform they have to speak in terms that ordinary people can understand.

Tell us how much money developers have saved since they switched - how many individual apps are using S3, etc.

Looking at it from a bit more of a critical view than their graph would probably appreciate, that means that from the end of 2007 on they've done a little better than doubling yearly.

That's it? They're an enormous company and seem to me to be by far the most-mentioned storage option. I'm surprised they're not a heck of a lot higher, especially given that a number of very popular cloud services sit right on top of S3 and store many gigabytes and hammer it with requests.

Oh well. If they maintain that growth, they'll own everything soon enough. I don't mean to imply yearly-doubling is bad in any way, just that I would've expected more by now.

Sure, depending on the definition of little better: Yearly growth (# of objects): 2.9 > 14 > 40 > 102 > 262 Yearly growth (as a ratio): 4.8 > 2.85 > 2.55 > 2.56

Also - this is just the number of objects stored (not put in ever) Obviously because storage costs, folks have purging solutions. It'd be interesting to see a similar trend for requests/sec.

Is cloud storage the factor that is driving companies to AWS? I'm sure that's huge for NetFlix, but they are an outlier. My bet would be on EC2 usage as being a better growth metric.

Well, to pick a high-profile AWS-user here: all of Dropbox runs on S3, and apparently a bit of EC2, last I saw. Not that I think they account for a truly significant amount of Amazon's traffic, but they're not small-time either.

Off-topic: I've always wondered who 'Jeff' is on the AWS blog…I like to think it's Bezos, but is it?

It's Jeff Barr, Lead AWS Evangelist.

http://news.ycombinator.com/user?id=jeffbarr (the OP). http://www.jeff-barr.com/?page_id=670

edit: Barr, not Bar :(

Yes, that's me, but with two r's.

When I started writing the AWS blog in late 2004 I wasn't sure how personal I should or could be, and didn't know if it was ok to brand it with my full name. I had adopted "-- Jeff;" as my signature a long time ago, when one of my friends had "++md;"

Nope, It's Jeff Barr. http://www.jeff-barr.com/

[[edit]] that'll teach me not to refresh before commenting.

wow the growth is phenomenal! Everything is indeed going to be in the cloud

It is kind of funny, right? I mean, our servers are already "the cloud". So it's a really cloud within a cloud.

Reply on news.ycombinator.com