Hacker News Reader

Comment by stuckagain | Hacker News Reader

"We have committed to spend $2 billion with Google Cloud over the next five years and have built our software and computer systems to use computing, storage capabilities, bandwidth, and other services provided by Google, some of which do not have an alternative in the market."

There's some numbers for you that Google wouldn't have provided (as far as I have seen).

View on HN

Is it not possible to build your own "alternative in the market" for $2 billion?

There is also the opportunity cost of investing in hardware

i doubt it would involve hundreds of millions let alone billions in hardware costs.

It is absolutely possible to build a stellar internal alternative to Google's offerings in 5 years with 2 billion dollars.

edit: I should point out that this doesn't account for ramp-up. It would obviously take awhile before things would be to a point where it would be even comparable.

Just to clarify your answer, here is another thing Google won't tell you: once you build yourself in their cloud you will have to find something else to do with all the time you used to spend on latency.

Why won't Google tell you that?

But by the time you matched them, they likely would have moved way beyond, with all kinds of new features...

Oracle is spending way more than 2 billion dollars on its cloud team in an attempt to catch up to AWS, GCP, and Azure.

Would you consider Oracle's cloud endeavor a fair test of how able a company is to reproduce GCP's offering with a fistful of money?

Oracle is offering external services, that is a huge differentiating factor

What I can't figure out is why Oracle didn't start down this road 10-20 years ago. They more than anyone should have some ideas about scaling infrastructure for databases.

sales led, not strategic, and it would have cannibalised their existing money printing machine. imagine if they started offering their customers PAYG elastic accounts rather than multi year mega provisioning deals - their numbers would have been destroyed

This, and related topics, always makes me laugh a bit. Mostly because people and companies will defend either move, with terms such as "focusing on the core business" or, opposite that statement, as "vertical integration".

From my mostly uninformed POV, a company can do whatever they want with respect to this and be on good ground, making it an arbitrary decision with little basis in actual objectivity.

That both directions can seem like a good idea to a speculating outsider doesn't mean the decision is an arbitrary one. It means the deciding factors aren't something we're privy to.

... but might still be wishy washy things like personal opinion or background of key stakeholders, company culture, etc.

Things scale differently in different markets and products... It might be trivial to spin up extra cloud capacity within Google's system and benefit from all the robustness and redundancy they've already designed out. Meanwhile, Amazon is building their own UPS competitor because their infrastructure needs in the real world are specialized enough to be able to benefit.

Each case is different and the details drive the decision.

I think Amazon is building their own UPS because UPS hasn't scaled for them.

In this case, scale counts as a specialization IMO.

TLDR (for my comment) : Amazon are building their own UPS because they can. Any othe arguments in favor of it be it efficiency or costs are secondary.

Longer version: Not sure how Amazon deliveries by UPS are in the US but in the U.K. there are carriers which offered far better delivery service than what is known as Amazon Logistics here in the U.K. Once Amazon started using Amazon Logistics, the level of service dropped significantly. False deivery attempts were common and Prime Next Day were not happening next day 3 out of 5 times. I cancelled my Prime membership.

After about a couple of years I have resubscribed to Prime since it is now much better value AND Amazon Logistics have noticably improved. Next day and even Same day deliveries are indeed happening.

They didn't need to setup Amazon Logistics and suffer poor service quality. There were near perfect couriers (for e.g. DPD they even offer tracking your courier driver on a map in near real time).

That wouldn't be surprising at all. Alibaba has long been building its own logistics for Taobao in China as well. It's quite a natural development of affairs I guess.

To clarify: is Amazon designing their own Uninterruptible Power Supply for AWS, or their own parcel shipping service?

Shipping

http://www.forbes.com/sites/walterloeb/2016/09/29/amazon-wil...

> Things scale differently in different markets and products.

This is vague.

Running on the cloud is always more expensive than running your own infrastructure past a certain size and always provides less predictable performance.

That doesn't seem obviously true. If you operate a service and I operate a service and their loads are out of phase the cloud provider can serve us both with one set of hardware, halving the price. If I own my hardware, and I'm not a cloud computing provider, I pay the full cost 24x7.

It's not arbitrary.

It's actually about whether or not it would cost more, over the long term, for a company to develop such capabilities themselves, or to outsource it.

Generally, due to economies of scale, a big, established provider will be able to provide such a service much cheaper than it would cost you to run an equivalent service.

However, it depends on how much that provider is charging you, on top of their cost of sales. It's pretty basic in that if it costs more to do it yourself, it's easier to let someone else do it. Especially when you take into account the considerable R&D costs that it would take to provide an equivalent service. Google's services would, I'm sure, be highly developed. This statement says as much, in that it claims Google has services offered by no other company.

So no, it's not arbitrary.

Of course it is. But you need to operate the service while you're building the replacement. What that says to me is that there is an order of magnitude improvement in their operational costs available to them. Depending on how long it takes to build their own infrastructure, that could imply a big boost to margins 1, 2 or maybe 3 years down the road.

The cost of moving software that relies on app engine frameworks is probably the bigger unstated one. Unless you can port those frameworks to your shiny new datacenter with automated deploys, etc, you'd be running two versions of the same code for a while.

you could probably make microservices a requirement as a governance issue for this reason alone

Why take the chance when you're about to have a lot of IPO cash? Go with Google for a few years until you (hopefully) become FB. Right now they'll focus on features, signing advertisers and users.

Absolutely.

I imagine they had a very strong bargaining position willing to commit this much, for this long. Even if Snapchat wasn't a valuable brand for Google to brag about, this amounts to ~10% of the yearly earnings [1] of Google's cloud business (SaaS and IaaS, of which I suspect SaaS like Apps for Work is the lion's share)

I'm sure they're getting a good deal, and can focus on features and getting their platform profitable, as you said.

1. "..at that pace, Google’s cloud could generate $4.1 billion in revenue in 2016" http://www.networkworld.com/article/3029164/cloud-computing/...

Maybe it is, but this is a path-dependent phenomenon. GAE allowed Snap to reach this point in their growth. If they had earlier focused on internalizing their operations they might have failed to get here. Now that they are here, it apparently makes sense for them to commit to GAE for at least several more years.

Keep in mind that Google Cloud is much larger than App Engine.

In my opinion, it's highly unlikely that you could match Google's hardware for $2 billion ever, much less in 5 years.

Keep in mind that Google's revenue in 2015 was $75 billion. Their hardware and software know-how is fundamental to their ability to make this kind of revenue, so you better believe they're investing more than a measly $400M per year (Snap's $2 billion over 5 years) back into their hardware. This Snap commitment is a drop in their revenue ocean, at 0.5%.

From a non-fiscal perspective, there's a lot of other great reasons to think so. For one, they've been building custom hardware for a decade at least to support the needs of running at the scale they do. Worth checking out this paper [1] and this video [2] if you're interested in some of what they've been doing purely on the networking side. In mid-2015 their newest data centers were pushing 1 petabit per second of cross-sectional bandwidth. [3] That's mind-blowing capacity.

So their internal data center networking is at or near top of class. In addition to this, they've also invested heavily in inter-DC and backbone capacity [4. Again, this is something that you presumably benefit from when, say, distributing your content across the world for faster access.

In addition to that, you get access to all their class-leading scheduling software and state management software that they've developed, refined, and redeveloped over the last 15 years. They know how to take a pool of compute and storage and turn it into solid distributed systems like only a few organizations do. I'm sure they get a good amount of advising from Google's experts somewhere in that $2B, not to mention services like BigTable, Kubernetes, etc. that you can find on their growing products page [5].

So though I can't back my prediction up with solid data, since I'm not a Google exec, I think it's pretty obvious that even $2B is not going to get you anywhere close to Google's existing infrastructure. Definitely not in 5 years.

[1] http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183....

[2] https://www.youtube.com/watch?v=n4gOZrUwWmc

[3] http://www.datacenterknowledge.com/archives/2015/06/18/custo...

[4] http://www.telegraph.co.uk/technology/2016/06/30/google-laun...

[5] https://cloud.google.com/products/

You seem to be assuming that all of googles revenue is going back into hardware/backend investments, and that somebody using Google cloud offerings is getting full access to everything they are investing in.

Neither have any basis in reality.

You don't need to match their hardware. You need to match the subset of features and functionality their hardware provides to you. That is a much smaller footprint and smaller/specialized engineering problem.

Key point: You dont have to match all of their hardware nor all of their software - just what you are utilizing and some of that could be done from existing open source projects - Snap certainly does not utilize every piece of hardware in existence for Google Cloud Platform

Be serious for a minute.

First, the open source world is a pile of crap that doesn't work and doesn't stick together. It's not remotely comparable to any of the offering from AWS or Google, let alone their combination of offerings put together.

Second, they definitely use EVERY piece of software and hardware. Just like anyone who has over 100m daily users.

It might be possible, but Snap could be constrained by engineering resources. Google has some super smart engineers and while Snap might be able to build something cheaper, that might mean pivoting your entire primary business to essentially compete with Google.

If there's an adequate supplier, why take the risk and time to develop second-mover competition to them rather than just buying from them and building your core business?

Two options. Companies can focus on cutting costs (roll their own infrastructure), or focus on expanding revenue. Snap clearly should and is in expanding revenue mode.

Not a businessman but who says its all about the infra. buy vs. build. It allies them with Google. It gives arguably the most powerful co. in the world a stake in Snap' growth and success.

That's a great point, along the lines of, "When you borrow $100 the bank owns you; when you borrow $1,000,000 you own the bank"

I dunno, $2 billion over 5 years is not exactly a ton of money to big G.

It is to GCP. GCP revenue is currently much less than $4 billion run rate.

They are reporting ~4B run rate for all cloud but that includes SaaS, GCP run rate may be < 1B (but it's hard to know for sure).

for infrastructure on that scale smart money would be on Lambda or some other FaaS to leverage huge price savings and create a barrier to entry for competitors

Expert on Cloud Computing here. Short answer is "No". Here is why. Google Cloud has infrastructure that scales to petabytes of data and millions of users. Google primary uses this infrastructure for storing, processing and communicating the Internet. Add the services like Pub/Sub, Dataflow, BigQuery, TensorFlow & CloudML and things like security, communication backbone, ... its near impossible to build Google Cloud or even few critical components like the ones mentioned above with 2 billion dollars. Also, if they focus on building infrastructure, it might slow them down significantly. They are better off building their chat platform rather than building the infrastructure.

Hmm. Cloud computing expert here, too. I partially disagree.

I would simply say "it depends". For companies with specific use cases, e.g. Dropbox, it makes A LOT of sense to build at least 60-70% private, and the rest on AWS or GCP. Snapchat looks like a special case to me.

If you are doing just one or two things at a massive scale (Dropbox storing files), it makes sense. But, we are talking about Snap Inc, the infrastructure, and services it needs. A single high-speed intercontinental fiber cable costs 400 Million $ (https://techcrunch.com/2016/10/12/google-and-facebook-are-bu...). To build a service like CLoudML (Hosted TensorFlow), you need to write software (TensorFlow), build the ASIC Chips by manufacturing / by partnering with a chip manufacturing company like Intel / Samsung / Qualcomm, you need custom networking software and hardware to scale massively for data workloads (Andromeda), years of research to build these tools, you need to create your own programming language (Go and Dart) to be able to scale your development to these levels, you need to build data centers in multiple regions, you need to build DeeLearning models for cooling your data centers (https://deepmind.com/blog/deepmind-ai-reduces-google-data-ce...), ...

Why would you write your own Tensorflow and Dart and Go when you can use them free of charge?

That was his/her point.

That interpretation would only make sense if TF or Go could only run on Google's "cloud".

> if they focus on building infrastructure, it might slow them down

I always found this issue of "focus" a bit strange. Two billion can buy a lot of focus. For example, say there was a separate company smaller than Google that snap outsourced their infrastructure to, allowing them to focus, then they bought that company - it comes out to the same.

The technical capabilities is a different story.

The amount of money on the table isn't 2 billion dollars. The amount at stake is the difference between 2 billion dollars over 5 years, and the theoretical best case operating costs of your own infrastructure, which very well might be higher than Google's because Google's scale is bigger. Snap might say to themselves that they don't want to be paying Google's margins, so they invest a billion dollars in development for infrastructure with identical performance costing only 350 million a year instead of 400 million. Is that worth it?

By the time the other company builds infrastructure, some other company would have built the product that Snap wants to build.

By "cloud computing expert", do you mean Amazon sales shill? It's weird enough that someone on HN would just use that vague credential of "expert" in the first place, but this thread is ridiculous ('round here, usually we recognize experts when they say something along the lines of "Hi, I'm the guy discussed in the article...", or "Hi, I actually wrote that software 15 years ago...").

You know that Snap can hire more than 3 people, right? They can have people working on the infrastructure at the same time that someone else works on "the product" they want to build. That's what happened at Google, and as a side effect, they now have a cloud platform they can rent out.

There are some benefits to using cloud services in some cases. It's rare that 100% cloud is a wise deployment move, especially for companies that operate at Snap-scale.

That's the point I'm trying to make. Snap has, say, 100 employees + $2b. The 100 employees are working on features. The $2b can now go to Google, or to purchase "focus" for in-house infrastructure, there is no slow down in developing features. Now if they can't technologically duplicate what they need using $2b is a different story which I have no opinion on.

They don't just have to build it though; they have to build it fast enough to handle their growth. $2B might not be enough (or it might not even be possible for any amount) to build what they need fast enough to not hamper growth.

Also, by buying from Google, they hedge against lower than expected growth too. If growth doesn't meet expectations, while they're still obligated to spend the $2B, they can re-sell the services for likely close to cost, given they'll be getting a discount.

Of course it's damn possible to build that with $2 billion. Will it be robust as google's infrastructure? No. Will it have (unlimited) scale as Google has? No. Will it have network connectivity and geographical distribution as Google has? No.

So, if you set scope on expectations and dedicate time and resources, it's definitely possible. Reasonable? Probably not.

It doesn't matter the amount of money you have, it's basically impossible.

Everything at Google is an internal Google secret made by Google for Google. ALL the software and ALL the hardware that's running it.

Services rely on lower-level services. If you want to copy the high level service yourself and executes it at the same level Google does, you'd need to have everything it depends on. Too bad for you, each piece is a multi billion dollar projects itself, with its own dependencies...

Let's imagine for a minute that there was a service that is somewhat standalone-ish. You may consider poaching the handful of people who can make it, for $1M a head. But that won't work well because money is not everything and you can't help it. Whenever there are ONLY 50 people in the world who can deliver what you need and you need almost all of them, you're fucked.

In short, the castle is out of reach, even the building bricks are out of reach ;)

That's really the thing with Google. They have this huge empire and they build on top of their other work. This is why Apple Maps could never come close to competing with Google Maps. This is why Google Home at launch already did a far better job at understanding queries than Alexa which has been out for 2 years.

They're so well situated, with a years worth of technology and tools, so it's really hard for a new comer to compete.

Most likely not. My understanding is that Google also has their own long-haul ocean fiber, and those things cost hundreds of millions of dollars by themselves.

So Snapchat is basically ~80% (or more) of the entire Google's cloud business (if exclude G Suite from the Google's cloud).

Has Google published cloud revenue numbers? I haven't seen those, please link if so. Thanks

http://fortune.com/2016/10/28/google-cloud-business-2/ states their "other revenue" was $2.543 billion for its third quarter. A chunk of that is from Cloud. So it is definitely the largest share but not sure if it is quite 80%.

Just to clarify, Snap's S-1 says they're committing $2B over the next five years (specifically, $400M per year) - the number you've provided for Google's "other revenue" is $2.543B just in Q3. Ergo, it's nowhere near 80%. :)

Very correct! Got my numbers all switched around when looking for more information and was somehow thinking of 2B a year. Thank you.

Wow, single cloud lock-in filed with the SEC. Isn't 400m/y a bit steep for storage and compute power?

$400 million annually represents a monthly cost per user of $0.21 at 158 million daily active users. Which seems high at scale when looking at storage + bandwidth alone, I would think, but I'm also not a big Snapchat user so I'm not sure what is going on in terms of long term storage of video on an account level.

I would think their bandwidth costs are astronomical. Since 19/3/16 Snapchat has used 14.8GB of mobile data on my iPhone alone - not including WiFi.

Bandwidth is kinda cheap these days though, if you avoid GC and AWS. Anyone have recent numbers at this scale of commitment?

They will get a bulk discount so it will be cheap probably close to zero.

Close to zero but $400M in related expense.

I literally lolled at this comment.

I assume this is sarcasm...just want to verify :)

I would also assume they have factored in some growth over the next 5 years... If they don't have many multiples of 158M DAU by 2022 they are screwed regardless of their hosting bill.

They're likely using Google's state of the art Vision/Speech API etc.

Interesting detail from the Google license exhibit:

Encryption Technologies. Google makes HTTPS encryption (also referred to as SSL or TLS connection) available. Google servers support ephemeral elliptic curve Diffie-Hellman cryptographic key exchange signed with RSA and ECDSA. These perfect forward secrecy (PFS) methods help protect traffic and minimize the impact of a compromised key, or a cryptographic breakthrough.

https://www.sec.gov/Archives/edgar/data/1564408/000119312517...

Isn't that's just a description of all TLS? ECDHE/DHE key exchange is essentially employed on any non-poorly configured modern https site, TLS 1.0-1.3draft.

How is that "interesting"? They're just saying they use ECDHE which, by definition, provides forward secrecy.

I find the "do not have an alternative in the market" comment interesting - is there really anything which AWS does not have the GCE does?

Maybe the amount of rebates received?

Snap runs on Google App Engine, not on GCE (or perhaps they use both).

There isn't an alternative inasmuch as there isn't a drop-in replacement. Thus, there are switching costs, which thus represent financial (and counterparty!) risks.

Compare this situation to Dell v. Lenovo v. HP commodity x86 servers: if one goes out of business, you can go to the others (and proactively use this as leverage when negotiating). EDIT: Or more realistically, you don't need to do much/any work to run your service on a Dell v. an IBM.

If you're already tied into GAE, Google's other offerings in cloud become really easy to tie in, from adding buckets in GCS to adding extra stuff in GCE. With the soonish coming of Google Pub/Sub, it takes a lot of effort to switch to AWS.

There's a fairly exhaustive comparison between their compute services here: https://cloud.google.com/docs/compare/aws/compute

But without anymore context into what Snapchat is saying, it's too easy to read just about anything into that comment.

> Is there really anything which AWS does not have the GCE does?

Firebase is the killer here. AWS and Azure don't have it or anything comparable for realtime data. If they do, please tell me about it because I'd love to know about it!

I'm particularly interested in this because I'm building a product supported by Firebase and will have the same lock-in described in this filing.

DynamoDB?

Was that intended to be humor? DynamoDB is to Firebase as a CSV file is to a personal computer.

Come on, HN. How can nobody be mentioning some of the much cooler stuff available on Google Cloud?

https://cloud.google.com/products/machine-learning/

It's a mistake to just compare Google with AWS, thinking in terms of basic storage and computing. That's boring and obviously there are tons of alternatives, including Snap Inc. building it themselves, for the amount of money cited.

When it comes to cutting edge AI and related, Google's offerings clearly stand out among other cloud services.

How is it different from Azure ML and Microsoft Cognitive services?

I'm guessing it's their incredible amount of user data to drive that ML and AI.

Are these sorts of things latency-sensitive enough that they couldn't use them with EC2 and similar? Or are the bandwidth needs so large that bandwidth charges cancel out any compute savings?

Are they charging per single prediction like amazon? If so then it's almost a non-starter unless ML is the product.

Sort of, but not exactly. https://cloud.google.com/ml/pricing

I'll just leave this here... https://www.google.de/webhp?sourceid=chrome-instant&ion=1&es...

Disclosure: I work on Google Cloud

Exactly. What's the closest thing to datastore from other providers? DynamoDB? Some kind of hosted Cassandra?

GCE is incapable of doing the most recent cloud setup I've built on AWS, which uses g2 instances extensively (although Azure could).

I need the actual GPU—graphics, OpenGL, etc.—part, not just CUDA, in addition to the hardware H.264 encoders that come with nVidia's GRID CPUs. GCE has no equivalent.

We're also using Amazon's Elastic File System, and I don't think Google has an alternative—though that's something I could handle differently if I had to, at higher cost.

We announced that we'll have AMD's 9300x2s and NVIDIA's P100s at Supercomputing: https://cloudplatform.googleblog.com/2016/11/announcing-GPUs...

I'd be happy to get you early access, Erich ;).

Disclosure: I work on Google Cloud.

Interesting, I'll keep any eye out. The main renderer we use (Hydra, part of Pixar's USD[0] project) would need to be ported to run on the AMD GPU, but it's designed to be backend agnostic to a degree so it should be possible.

I'm pretty sure AMD also has the ability to H.264 encode the screen at 60fps (with low latency) which is really important to us—our "workstation" is entirely in the cloud, with a Pi 3 running the display/keyboard/mouse.

[0] http://graphics.pixar.com/usd/docs/index.html

I remember seeing the CTO or VP of Eng @ Snap was the Director of Eng on Google Cloud Compute. It seems to not be a coincidence.

You're thinking of Peter Magnussen but he left for Oracle shortly thereafter:

https://techcrunch.com/2014/07/31/snapchat-vp-of-engineering...

Reply on news.ycombinator.com