Four years after its release, Kubernetes has come a long way

You are correct on Google self-interest in the matter. Not so sure that Google's self-interest is so divergent from the average developer interest.

Kubernetes is hard to run by oneself. Worse, self-cobbled setups are also hard to run by oneself. Distributed systems, that is systems that require uptime and/or more than one computer, are hard.

The fact of the matter is that computing is the new electricity. While one can run oneself's own generator in the backyard, it is more efficient to buy electricity from a big provider, small-scale solar notwithstanding.

Kubernetes offers a well designed abstraction that simplifies managing distributed systems at most scales. Bonus, it has become a shared standard among providers of computing. That is a big step beyond proprietary lock-in abstractions [AWS, etc.], and likely close to the best one can hope for, technically speaking. Economically, perhaps a future solution is regulating computing providers as utilities.

All major cloud providers have/will have managed kubernetes and they are free to use the name, therefore I doubt it could be regarded as direct marketing for GCP. I would expect many to consider GCP but I just don't perceive this as a backdoor way of Google getting your apps, given the broad buy-in by the cloud provider community.

I think Google's interests are aligned with developer interests here. The main benefit to GCP that k8s success brings is that migration between clouds becomes easier (e.g. moving from EKS to GKE would presumably be easier than moving from Elastic Beanstalk to App Engine). Less lock-in benefits everyone except the current market leader.

don't believe you will get down-voted for this point of view.

running and most-important keeping a k8s cluster up-to-date is a major time investment. I think the important time is balancing the benefits you get from k8s with the time draining aspect of having to run the cluster.

also, as counter-argument to what you are saying: if I'm going to move to the Cloud, why leverage k8s at all? Most clouds have other abstractions that you can use when building if you don't mind being locked into a specific bigCo.

Upvoted. Google is like a black hole, gravity pulls you in. Imagine Google disappears tomorrow, what then? Today we have for containers ECS, for email iCloud, for search Bing. Tomorrow? ...

You seem to imply that k8s somehow means having all your hosting and deployment provided by Google, which seems pretty fallacious.

Insofar as it gets people to use GCP, it's not necessarily very useful in that regard. Other replies have pointed out that k8s provides a useful abstraction above multiple cloud hosting solution providers. I'd like to add several reasons.

1. Alternative solutions exist. For example, Docker Swarm. It seems to me that Swarm, otoh, is too inflexible for mainstream popularity (?). 2. Alternative hosting solutions exist: why GCP and not, say, AWS? 3. Some companies build on top of the main k8s features and provide their own solutions tied to their own offerings, obviating the use of GCP. For example, rkt and OpenShift.

So even if Google is the dominant interest to the extent that k8s would fail if it stopped its contributions to the project, it is not obvious that Google will be able to capture all the value that comes out of the k8s project (of course, it expects to capture enough of it to make fiscal sense)

>After some time people will realize that running kubernetes yourself is actually harder and more fragile than just running your app without it

Wow you've made some really good points backed by examples and data.

Every choice in life is a set of trade-offs. Each cloud provider (AWS, Oracle, IBM, Microsoft, Google) is likely to embrace some form of Kubernetes or Kubernetes-ish container orchestration. And each of them is likely to end up in a situation where there is friction if you want to switch away should build on top of their offerings.

But that's kind of to be expected. It reminds me of using .NET and ODBC/ADO and for so long there was this "benefit" that went like this, "But you can switch your underlying database anytime you want without code changes!" And yet that is rarely true at face value and furthermore, how many people are switching their underlying databases from one to another on any sort of regular basis. Who cares about the benefit of being free from lock-in if you never switch anyways!

In a world where everyone's on AWS or Azure or Google Cloud Engine, hasn't the ship kind of sailed?

Amazon is the 800 pound gorilla in the cloud.

Do you really want to see them completely dominate the market?

Kubernetes levels the playing field, and brings competition and choice.

Thanks for saying this man. Will keep this in my mind next time I am tempted to defend it.

You are correct, but... Would an AWS monopoly be any better?

In my company one person is the sole administrator for a 50 nodes k8 on prem cluster with only one additional person for the night shift.

So no, not hard to run, as long as you have the knowledge how to.

Of course they aren't doing it out of charity. They're doing it to prevent an AWS monopoly and to ensure that their offering is the preferred one.

And we'd have cause for concern if 1) Kubernetes becomes dominant and 2) Hasn't accumulated a critical mass of non-Google developers.

That's what happened to Android, and why Google is very successfully closing it up again.

But neither applies to Kubernetes. AWS is still dominant so Google is still incentivized to keep Kubernetes as open as possible.

But more importantly, Kubernetes is much more than a Google project now. Google could pull all of its developers off Kubernetes today and it would keep going just fine.

In your experience with these deployments, which K8s features aren't needed? Is it the orchestration as such that's not necessary, or K8s approach wrt to the workloads deployed?

You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. To me, this is the analogous case for K8s, you can probably do the same with VMs, but everything is a lot more productive with containers and K8s.

what would you recommend to use for container orchestration and service discovery instead of Kubernetes?

> I know of at least one hot startup deploying Kubernetes just to attract engineering talent.

Interesting proposition - if it helps attract better talent does that mean they needed k8s?

i am a mainly js engineer and so far have lived entirely within the firebase world as far as my backend goes. in your (and everyone else's) opinion, then: when -do- you need it? set some quantifiable bars for me please?

What if you're on a hosted service like gke/aks? The work needed to maintain the system has been minimal for us with a lot of positives

I don't think that's true. If you are using bash or chef to interact with the cloud then k8s can replace those with a much cleaner model.

My biggest fear is helm becomes chef cookbooks

I see a lot of value in Kubernetes. For one thing I don't want to wake up at 1 AM if one of my node goes down and brings down with it important containers. If I scaled the environment with excess capacity then Kubernetes should start up new pods on the remaining nodes.

"Kubernetes" is an Ancient Greek word meaning "more containers than customers"

https://twitter.com/srbaker/status/1002286820078571532

In that specific case, Kubernetes is approaching YAGNI territory. But as others have stated, it’s easy to get started with something like Amazon EKS and not have to worry about a thing.

Edit: Put another way, give me APIs instead of config file hell.

Depends on your definition of "need", it enables me to manage a ton of stuff across multiple dcs and regions easily. It would feel like the dark ages to be without it as this point.

Sadly I doubt it.

Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively.

https://www.quora.com/If-you-were-to-build-a-website-right-n...

> Google conspiracy theories - (Google is doing it to lock everyone in).

My own conspiracy theory is slightly different: Google's aim is to scorch the earth so that AWS cannot form a locked-in position.

I don't think they immediately saw this possibility, but that they moved swiftly and energetically to support the growth of Kubernetes once they did see it.

The alternatives for Google were all strategically painful. ECS might have become successful, meaning a loss for Google. Mesos or Docker Swarm might have taken off, leading to snap acquisitions of the relevant companies by Microsoft or AWS, again a downside for Google.

As the plurality contributor to the commoditising winner, Google is able to prevent the other two cloud giants from pulling ahead on container orchestration.

Disclosure: I work for Pivotal, we have a Kubernetes-based product (PKS) we co-develop with Google and VMWare.

Those are all good points, I've been surprised reading this thread at the number of people that conflate Kubernetes with Google given the CNCF stewardship of the project.

One point though on complexity is that Docker swarm is substantially less complex that Kubernetes and for many use cases could do the job.

Thank you for fighting against the FUD.

Can I ask if you have a bare metal cluster or you do this through a cloud service?

I don't think using kubernetes is useful for small projects. It's currently still a hastle, even though it has evolved a lot. Not much in terms of benefits to show for it either.

I would read up on Docker though. Having your apps in docker does provide some nice benefits like better isolation, keeps you up with newer tech and you can then easily integrate with Kubernetes. It's also rather relatively easy to achieve.

Nope, you're good.

Probably not. Container services really shine when you have a lot of different teams that all need to launch services and not worry about what happens at the hardware level or where they run. This makes sense at a lot of large companies like Facebook and Google where they have a dedicated team running the container layer. GKE makes it a bit easier but then you're locked into Google Cloud.

If you're only running a few services it really doesn't make much sense to setup Kubernetes. I setup a small cluster a few weeks ago and it certainly was a lot more involved than I expected. I ended up deciding not to use it, since it didn't really add much (for my use case) and I felt like if anything went wrong I'd be in a world of hurt trying to debug it.

Took a lot more time to setup Kubernetes than my entire current deployment system which uses https://github.com/pressly/sup.

However, dockerizing your projects might not be a bad idea depending on what you're doing. I dockerize a lot of my projects since it makes it super easy to deploy when using the automatic Docker Hub builds. Though I did have a problem where Docker upgraded and make all of my existing containers unrunnable.

I have very similar requirements and setup to you. I dockerized from the start, but mostly to ensure consistency of dependencies among different environments (test, staging, prod), the CI server, and development environments. Lets me treat the whole app system as a single binary even while I mess around with the internal structure.

If you anticipate needing more scale or control for your monolith, you will probably eventually need to learn about containers. Look at containers-as-a-service tech which includes Docker, Heroku's continer registry, ECS, and/or Kubernetes.

If you anticipate building systems not as a monolith, look at Functions-as-a-Service.

I'm in the FaaS camp for forward thinking work, and wrote up a bunch of my experiences here: https://github.com/nzoschke/gofaas

Heroku is a perfectly valid solution. If you want to keep some of the auto-build PaaS benefits whilst containerizing for (any of) lower cost at scale, running exactly the same app configuration in dev as in production, security, attaching services not available on Heroku, distribution to clients to run on their own cloud; then Docker does offer a nice solution - see gliderlabs/herokuish or redhat s2i for a familiar autobuild environment.

And despite the one-size fits all dogma of some of the kubernetes crowd, there are many solutions to docker deployment and orchestration at different scales, especially if you're looking to retain most of the benefits of Heroku and its PaaS workflow; From the low end there's AWS Elastic Beanstalk multi-docker, flynn.io, docker swarm mode, AWS ECS and logistic sugar on top of that like Remind101/Empire, convox etc., before you get to PaaS-ish kubernetes behemoths like OpenShift or flavours of Cloud Foundry.

the distributed systems knowledge that you would pick up by learning can be valuable in designing your saas in a way that will scale if your business grows.

i may be generalizing, but it comes down to balancing your software architecture in a way that is quick for you to iterate on now vs scaling out to support an expanding business. there is a comment further up about a monolithic deployment -- great for right now but if his business grows 100x it will be difficult/expensive to scale without significant changes.

starting with the seeds in place can remove a lot of pain later, so things like containers may not be a huge win to you today, you should consider what value they may bring down the road

> forward thinking engineer

Yes. Based solely on your desire to be on top of tech and given the broad adoption by every major provider, you probably want to invest at least a couple of days.

That doesn't mean there is any impetus to transition your current production, only that it is most definitely worth a cursory understanding for the future.

(potentially) interestingly Heroku uses containers as part of their PAAS, so actually your workloads are using containers, they're just handling the complexity of that for you :)

Personally I think knowing something about Docker/containers is a good plan, useful for testing on-laptop dev etc, however kubernetes is a fairly complex system and if you don't need it, and you're happy with your current process, I wouldn't rush to it.

>It'd be nice to see more P2P cluster building tools for individuals and groups of friends.

I've been wanting to build a k8s cluster for myself and a few friends. I didn't feel there was a lack of tooling to facilitate this, but I haven't really looked into it much. Could you share some of your specific concerns, and any tips if you've done this?

sometimes i wonder how openstack would have done if it was written in go

Is it though? I believe the success of Kubernetes is mostly due to Docker whose success is mostly due to features provided by the Linux kernel, to which Docker is a frontend...

Docker's code is seems not very nice to me (when I last read it), I especially don't like the error handling, which provides very little context to go on.

> Were there tools that helped keep all the services running together in a single machine?

Traditionally, services on unix system are started and supervised directly by the init process ("init system"). That has been SysV init for most of the time, but in the last decade we got two major new systems: upstart (dead now) and the infamous systemd. Using systemd can still be a very good alternative to using docker today.

> Was it a machine stack per service?

That still sounds like a good approach to me today.

If your service is not big/expensive enough to need at least one full machine, why bother cutting your app up into such small services in the first place?

For example, for web applications, a traditional setup would have been to have a couple of dedicated servers that run the database and a larger pool of webservers that serve the (stateless) application. In front of that you could have two linux boxes doing the network load balancing and failover or a commercial load balancer "box" product.

Here's Netflix from 2011 (pre-Docker): https://www.slideshare.net/adrianco/global-netflix-platform/... This was widely considered the gold standard in cloud native architecture.

Twitter from 2013: https://www.slideshare.net/caniszczyk/twitter-opensourcestac... They had a similar in-process JVM-based approach as Netflix, but they also used Mesos as a scheduler.

> But before Kubernetes AND docker, how did people manage microservices?

Cloud Foundry and other opensource PaaSes. Heroku. Chef, Puppet et al. Java app servers with WARs.

Before that was a time of heroic mythology. Shell scripts, bailing wire and raw genius.

Disclosure: I work for Pivotal, we work on Cloud Foundry.

I used to run a single EC2 box for each server in the arch. Config management was ansible/salt scripts which bootstrapped a server and it as an AMI. They were then deployed by ASG. A python script glued it all together.

Nowadays we do much the same thing, but with Docker on ECS. The glue script is gone, because Terraform.

This one here is also a pure marketing...

What an insult to one of the most impressive communities in open source. Shameful.

I get the sense that the naysayers outweight the supporters, simply because the headline alone is not news. Don't let them deter you, spend a few hours and decide for yourself.

HN tends to follow tech hype cycles, Kubernetes is entering the “trough of disillusionment” now that people are starting to use it in anger. This reminds me of prior fights like Solaris vs Linux. Kubernetes is not that bad, its actually quite elegant in many ways. It’s just that it is tackling a hard problem in the hardest way possible: by providing a new set of bottom up abstractions to replace traditional virtual machines, config management, and network and storage virtualization in the image of the Google model of massive scale workload scheduling. That means it is a whole new world to learn but you can’t quite forget about the old world yet, and need to map between the two. Is it complicated? Yeah, mainly because it’s evolving so fast and serving many audiences. It will get easier.

Simpler alternatives? Any PaaS like Heroku or Google App Engine or Cloud Foundry.

Or a serverless function platform like AWS lambda (not sure if it’s “simpler” but it is getting there)

Disclaimer I work for Pivotal who sells cloud foundry and a distro of Kubernetes

AWS ECS deployed by well written Cloudformation templates and ECR as container repository

In addition to those mentioned, for on-prem or self-controlled container clusters, it could be worth looking at Docker swarm.

It has a lot fewer features than Kubernetes, however it's also a lot simpler to get up and running with.