Google Storage Now Available To Developers | Hacker News Reader

Google Storage Now Available To Developers

code.google.com

221 pointsASUmusicMAN60 comments

5 years, 1 month and 28 days after S3 is publicly launched. It is just a reminder that the threat by incumbents to startups is hardly worth worrying about.

Amazon was hardly a startup 5 years ago. And we're talking about a service and a space that's difficult to replicate and get into. S3 had resources most startups won't have to get that 5-year lead.

The push by Amazon into cloud services was very different from the business they were pursuing up to then. OK, so they were essentially just wringing some money out of all the infrastructure they already had in place but it was quite a radical change for an incumbent.

Amazon wasn't the "startup" in his example. His point was the 5 year wait time for a "giant"* to get a product out the door...

* who, even worse for their case, already had the infrastructure, resources, engineers, etc. to get it done.

The point still stands; this isn't an example of why startups shouldn't fear giants/incumbents. Just because Google couldn't/chose not to get a product like Amazon S3 out the door to compete with Amazon, doesn't mean incumbents can't replicate and dominate a bootstrapped start-up's low-cost, innovative web app or mobile app.

The statement should say that startups with innovative, hard-to-reproduce products have nothing to fear; you'll either produce a product that they'll choose not to compete with/can't compete with (Twitter, Facebook) or get bought by them.

How many startups get killed because they can't compete with the big guys, though? Here's an example: http://family.go.com/assets/bubbleshare/

I'm trying to innovate in the same field that Bubbleshare was in. I only learned about them recently via reading old techcrunch posts. What killed them? Did they go through a prolonged period of difficulty?

Founder greed and bad timing.

Ah can you elaborate please? To me it looks like Bubbleshare was the best thing ever---till it suddenly shut down.

I find glowing reports about it from TechCrunch, then.... nothing, it just dies.

So what happened?

Disclaimer: I was not involved, but several people that were are friends. These friends did not inform my answer here, which is just my opinion based on many sources.

BubbleShare was a Toronto tech darling started by the always brilliant Albert Lai. They were WAY ahead of their time.

All I'm comfortable sharing here is that they received buyout Offer A, which Albert rejected in favour of doubling down for a better deal. Unfortunately fortune did not smile and BubbleShare ultimately accepted Offer B, which was significantly smaller than Offer A.

The site was sold to Kaboose, which is a Canadian family content company. As usually happens when startups are acquired, the key talent left (Albert started Kontagent) and innovation on the site halted.

I'm not an analyst, but these things usually distill down to "too early / too early / too late". I'm not sure that people were ever lining up to pay them money to use the service, and that might give you pause before going down the same road. Many products are cool but aren't solving a problem causing paying customers real pain.

Wait, who's the incumbent and who's the startup? Amazon was founded four years before Google...

I didn't mean it as an analogy, but there are plenty of examples if you are looking for Google vs startup. Twitter/Buzz, Groupon/Google Offers, Facebook/(Whatever Google's Social Strategy is).

5 years to clone a product that Google already has the infrastructure for and is within Google's core mission seems excessively long.

How is this within Google's core mission? Their mission is not to provide IT services to other companies...

I'll spell it out very clearly:

Google's mission is "to organize the world‘s information and make it universally accessible and useful."

Gmail helps my company organize our email and communication, and makes it accessible from any computer anywhere in the world.

S3 helps my company organize our information and distribute it around the world.

S3 helps dropbox organize their users information and make it accessible from any device.

An S3 clone is clearly within Google mission.

Gmail is a product, S3 is a piece of infrastructure used by other products.

Google's mission is also to make money, companies make money by focusing on their core strengths. I would rather have Google not clone other people's products, Microsoft did that and they failed over and over again.

S3 may help you organize your information, but that information is probably not public, and if it is then Google's services can access it anyway - they also aren't a charity - if you're looking for charity (i.e. products released for the betterment of the world only) then you need to search for non-profits.

S3 is most definitely a product. They charge for it, don't they?

The pricing is far less interesting to me than the differences between GS and S3. In some significant ways, GS appears to be a much more technically sophisticated product. And there are some arguably less used functionality like BitTorrent which are removed from the picture.

The ACL scheme is significantly more flexible on GS. In fact, one of the two major problems I have with S3 is a non-issue on GS:

1. All files on S3 are not world-readable when they are first uploaded. You cannot change the default permission for a file uploaded to a bucket. On GS you can set the default ACL on a bucket to world-readable.

2. For me, the most incredible thing GS could do right now is add a callback API. I want it to notify my application when my bucket is updated, webhooks style.

With both 1 & 2 in place, you can build storage driven applications in the cloud that don't require constant polling. Man, that would sure be something.

A compelling reason for our use case to switch from S3 is their support for lots of buckets coupled with CNAME support:

http://code.google.com/apis/storage/docs/reference-uris.html

Due to Amazon's limitation of 100 buckets per account and the coupling between bucket name and CNAME, hosting files for our clients and supporting custom CNAMEs has not been possible for us. If we were to move to Google Storage, it would be.

I just wish either one would support SSL for CNAMEd buckets.

That is a limitation of SSL.

If you don't like it, ask your browser vendor to support section 3.1 of RFC 3546.

It should be possible via a combination of Amazon's elastic IPs and the SSL offloading that their ELBs already do.

Shouldn't that be ask your user's browser vendor to support section 3.1 of RFC 3546?

Another good reason for certain use cases is its support for resumable uploads:

http://code.google.com/apis/storage/docs/developer-guide.htm...

Afaik S3 can't do that.

No, S3 (essentially) can as well:

http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-uploa...

Multipart uploads only sort-of address this. For one thing, the minimum "part" size is 5MB, so you can only resume at 5MB boundaries (or whatever part size you use). You also have to manage more state yourself.

It's really nice to see that boto (python AWS library) supports S3 and Google Storage side-by-side. Being able to pick and choose providers behind the same API is how the cloud should be.

It probably wouldn't have worked out that way if Google didnt clone S3's API.

To be fair to Google, most of the S3 REST API is pretty obvious. Even if you'd never seen S3, you'd likely come up with an API that was 80% similar.

not the openstack storage api..

What do you mean? I haven't looked into Openstack in great deal, but my impression of the storage API was that it was very similar to S3, aside from how it handles authentication.

Cloned but with some required new fields like "x-goog-api-version", so existing S3 libraries won't quite work without modification, it seems.

Can someone explain why I would choose this over S3, when S3's starting rate is 0.14 / GB and goes down from there?

At Google's scale I would have guessed that they would have had no difficulty in matching and even beating Amazon's pricing. This is surprising. Why so?

Even if they could, why should they, if they don't have to?

Perhaps they're looking to grow the service slowly initially? If they announced pricing significantly lower than amazon's right at launch they'd get a flood of new traffic that could overload however much resources they provisioned for launch. By starting out with a slightly higher pricing model, they can test out the service with production customers, and slowly reduce prices to attract more business when they're confident they can handle the load.

No more invites required to use, but at first glance it appears to be more expensive than S3.

Cost of storing data:

S3: $0.140 per GB (at the most expensive rate) vs. Google: $0.17 per GB (at their only listed rate)

The rest seems to be the same though.

Fascinating, so a 1TB sata drive is $70 [1] these days and that equates to about 7 cents per GB, if you made three copies to insure your backups had a 'good' copy that would be 21 cents/gB so its cheaper than using your own hard drives.

The whole 'we've decided to kill that product, you've got 30 days to get your stuff back' and the 'we lost a switch or something and we'll be back online day after tomorrow' kinds of things are still concerns of course.

I've always been impressed at the prices Amazon could charge for S3 and keep it a going concern. I'd love to see the breakout on that revenue but I'm sure that isn't going to happen any time soon.

[1] http://www.newegg.com/Product/ProductList.aspx?Submit=ENE...

Your comparison assumes that the average lifetime of a hard drive is 1 month. I sure hope that's not the case!

Also, don't forget bandwidth charges if e.g. some of your s3 access does not come from ec2.

Oh it gets much worse if you assume that you replace your hard drives every 3 years (their warranty period) which would add a $1.94/month depreciation cost.

You could add into the cost / GB of bandwidth to access the files, not as easy to factor into a TCO model in the personal use case.

It gets better, not worse. At steady state you just account for the depreciation cost, though, right? So that's $1.94/TB/MO/Drive, or $0.0019/GB/MO/Drive or $0.01/GB/MO for 3-way redundancy.

The way I was accounting for it you have to replace the drive in 3 years, you have 3 drives. So you've got a cost of $210 that recurs every three years (purchasing the drives). If you distribute that cost across the 36 months that is a $5.83/month fixed cost (doesn't vary by storage usage because you have to replace the entire drive). Unlike Google which can amortize depreciation on a per-GB basis because they are spreading out the replacements amongst many thousands of drives, as an individual you're on the hook for your own drives regardless.

right, but the point being made was that the storage fee Google/S3 charge is not a one-time thing - it recurs monthly.

Google's pricing is pretty close to Amazon's for low-end users, but high-end users will still find S3 to be much cheaper. I was hoping Google would be the competitor that would pressure Amazon to reduce the cost of S3. S3's prices have been pretty constant over the last few years even though I would assume that Amazon's expenses have decreased due to lower storage and hardware costs.

Download data: $0.15 to Americas, Europe, the Middle East, and Africa $0.30 to Asia-Pacific

So if a bunch of people from Japan decide to download from your app, you are screwed.

You're screwed if a bunch of people download from your app from anywhere, for sufficiently large values of 'bunch'.

Not if they paid you a sufficiently large sum of money for the privilege of downloading your app. :-)

Truth. My comment was quite negatively off-base, considering lots of downloads is what everybody strives for. I almost deleted it. Funny it got so many upvotes, should have been downvoted to oblivion.

A bucket in Latin America would be a killer feature. Doesn't Google already have a data center in São Paulo?

I am a little confused about Google Storage Manager. Is this supposed to be a consumer facing product? AKA dropbox killer?

No, it's an Amazon S3 competitor.

Then what is the point of the frontend app?

S3 has a web based access application as well. It is nice if you need to just manipulate a few files, check that a backup actually did make it to S3, etc.

Dropbox isn't really a storage company (they're built on top of S3) their secret sauce is in the syncing.

Only 5 years, 1 month, and 28 days after Amazon made S3 available to developers.

meh, in 3 years they'll probably just announce a new pricing structure where every reasonable use of this ends up with 1000%-2000% price hike (GAE user here, grumble grumble)

Damn, I was hoping this was going to spur Amazon to cut prices.

Anyone know how to copy an S3 bucket to Google Storage?

The quickest/cheapest thing would probably be to startup an ec2 instance, install the GSUtils [1] and then pull from S3 and push to GS.

1 - http://code.google.com/apis/storage/docs/gsutil.html#install

does anyone know if s3 is profitable?

Only the same people who know how many Kindles they've sold. Amazon is pretty tight-lipped about their financials, at least at a granular level.

i thought this was just an auxiliary service to other google services.. not really a competitor to s3?

Reply on news.ycombinator.com