Hacker News Reader

Comment by spyspy | Hacker News Reader

The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.

View on HN

> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra

What's the reasoning behind this?

Cloud Run/App Engine PM

Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs.

Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.

That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate?

Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhere to "live" in the GCP cluster?

If that is the issue, then I would have (naively) expected the solution to that to be adding a second, GCP-scale data-plane Borg cluster per zone, just for client workloads; rather than inviting these client workloads to co-mingle with Google's own workloads in the non-GCP part of the DC.

Isolation is often done in software, Google has invested a lot of effort in making sure the distinct services that they run e.g. Youtube transcoding on the same machine search is running on don't interfere with each other. Whether through cpu constrains or some other priority levels. These are features of borg.

https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=La...

I know nothing about the decisions behind where Cloud Run and GAE run, but even customer GCE VMs run on top of Borg, not just the control plane. GAE predates most or all of GCP, and there weren't separate GCP clusters when it got launched.

(Used to work for Google including the GCP team, but haven't worked for them for over 4 years and I'm not speaking for them now. I'm reasonably sure this is all already public info.)

likely just due to the age of appengine vs. all other gcp products

Used to work on 2nd gen AppEngine.

I helped ship the runtimes!

Yes, this is due to age. GAE and Run depend on pieces of infrastructure going back a long time.

Reply on news.ycombinator.com