Hacker News Reader

Comment by mehrdada | Hacker News Reader

This analogy is very misleading. Kubernetes is probably never going to run any real workload internally at Google, but gRPC powers all external APIs of Google Cloud, and increasing other Google properties (e.g. ads, assistant), used by mobile apps like Duo and Allo, and have some big internal service use cases. The reason Stubby still dominates internally is simply taking lots of time to migrate to gRPC that might be hard to justify, but I do see gRPC being used very widely internally at Google; it’s simply a matter of time. I don’t see that happening to Kubernetes; it’s a joke when compared to Borg.

Google aside, many other companies like Dropbox rely on gRPC extensively to successfully run infrastructure: https://static.sched.com/hosted_files/grpconf19/f7/Courier%2...

View on HN

I work at Google and my team has real workloads running on Kubernetes.

There's plenty of internal teams that use GCP. Increasingly this might be the direction things are heading.

GCP itself is a job on Borg. ;)

That's not true. GCE uses Borg very differently than normal Google internal systems, which you can imagine is quite natural as they are reserving different customers. GCS and other system, in turn, also differ wildly than GCE. when you talk about GCP as a whole, it becomes impossible to summarize in a few statement, and I doubt there is anyone on earth who is capable to describe it coherently even without time constraint.

What I said (GCP runs on Borg) is absolutely and technically correct, affirmed by your own comment, which highlights the power and flexibility of Borg. The point being no one[1] at Google relies on Kubernetes for raw cluster management capabilities at scale. They might use it for other things that can make deployment more friendly in some scenarios. (This doesn’t make Kubernetes a bad system by any means, just quite different and not a substitute for Borg whereas gRPC is a direct substitute for Stubby). This debate is better argued in your own eng-misc@ and not on a public forum.

[1]: no-one that we care. At Google this is obviously always incorrect. There’s always that someone who uses weird things like mongoDB and AWS.

And there's no reason why a small project should not. But nobody is going to move, say, indexing to GCP. And when it comes to power laws the big things are big and the small things are not.

This sounds like a No True Scotsman argument to me, that if something runs on GCP instead of Borg, it isn't "real". Also throw in shades of moving goalposts.

Indexing doesn't run on GCP primarily because it's legacy (as in, the first product Google ever did) and thus long predates GCP itself.

It’s neither of those fallacies. The fallacy is to suppose that if you know several people using technology X then it must be quite popular. We see this all the time on HN where people suppose that, say, Erlang is quite popular because there are dozens of companies, each with five engineers, using it. But then we ignore that there are five companies with a hundred thousand engineers each that do everything in c++. It’s the same with these other things. It’s quite like that K8s satisfies the requirements of 80% of the projects at Google and it’s also quite likely that all of them put together consume 1% of the production resources, so it leads to the question of whether it’s even capable of solving a really large problem, as mehrdada argues elsewhere in this thread.

It's not a Google product obviously, but Snapchat runs on GCP. That's quite big. Is that not a "real" product? Admittedly they're on App Engine, a much older product than Kubernetes, but I suspect they'd be able to run on Kubernetes, and perhaps that's what they would choose if they were to build from scratch right now.

Indexing has been rewritten many times over. Even if you removed all the dependencies on Bigtable and co., I think indexing would be the last to move, for quite practical reasons, due to its sheer size and design. The parent poster picked probably the worst workload to migrate to public GCP. Gmail, YouTube and search serving are easier in comparison.

I guess you’d find that on an rpc-weighted scale Stubby handles several orders of magnitude more traffic than those gRPC endpoints you mentioned.

This may be true today (although even on this metric I’d estimate Kubernetes to be off by some orders of magnitude, bordering zero). My point is there’s a path and plan forward for gRPC adoption and it’s a matter of transition to a new system (which can, admittedly, be very long). For Kubernetes, I don’t think there is a credible path for replacing Borg.

Curiously, what do you find lacking in Kubernetes compared to Borg?

Scale, for one thing.

Kubernetes is not a bad system but it’s not designed to run Google.

Can you expand on that, please?

https://kubernetes.io/docs/setup/cluster-large/

More specifically, "No more than 5000 nodes" and "No more than 150000 total pods" is fairly limiting to large (Google-large) clusters.

thanks for the answer.

I used to work on one of Borg teams at Google and now run Kubernetes platform. Nearly every Kubernetes component (node, master, networking) will melt down at fraction of Borg scale. It's not even close.

thanks for the answer.

What google system runs more than 5k tasks on a single cluster?

Heh, you have no idea. Lots and lots of systems. And 5k wouldn't even register there...

Can't edit my comment anymore but I meant 5k nodes / 150k pods / tasks.

https://github.com/google/cluster-data/blob/master/README.md...

"ClusterData2011_2 provides data from an 12.5k-machine cell over about a month-long period in May 2011."

10k replicas in a single cell is the default charge-free quota of every individual engineer at that company. It’s basically zero.

Reply on news.ycombinator.com