Kubernetes from the ground up: the API server | Hacker News Reader

Kubernetes from the ground up: the API server

kamalmarhubi.com

116 pointsluu21 comments

> When a kubelet starts up, it registers itself as a node with the API server and starts watching for pods to run. This is really great, because it means that when we get to running a multinode cluster, we can add nodes without having to reconfigure the API server

I really like the parts of this series that emphasize Kubernetes' design decisions -- it also says that all state is stored in etcd and that the API server is the only component that has access to etcd.

I'm really enjoying this series. We started using kubernetes at work a few weeks ago and detailed documentation like this is pretty hard to find.

Kamal, what resources are you using to learn the information in these posts?

My main resource is probably time. I read all the documentation when Kubernetes was first announced, and reread it when they were coming up to 1.0. I've also been following the proposals and issues a little. I've also got a clone of the codebase and read bits to make sure I'm understanding what goes on.

But yeah, it's a really fast moving project with a lot of moving parts, and I'm still not on top of everything!

Thanks for the info! I've found the end user documentation to be excellent, but the documentation for actually managing a cluster to be lack luster. This series has been perfect for filling in that gap. Thanks again!

Disclaimer: I work at Google on Kubernetes

I'd love if you could mention stuff in particular that you felt like was missing - we're trying hard to catch up!

Specifically some canonical instructions on how to harden a cluster would be helpful. Many Starting Guides have nodes use plain http to talk to the api server, thus even deployed containers can do this do.

It took me a while to find a proper kubeconfig example for kubelet and kube-proxy token auth (the one I eventually found was buried in some github issue i think).

Also, I found no information on how on what to put in the authorization jsonl file for kubelet (the given example is wrong, since the kubelet needs to write/report node status to the api) and kube-proxy. Peeking into the code helped, but I guess this information could be helpful for admins.

Thanks for the writing up, really nice series so far! I would like to know what are the competitors of Kubernetes to run docker in production and if you think Kubernetes is suitable for small / single man projects? Or what would you advice instead?

Cheers

I use kubernetes in production at work for a large website and at home for very small projects. It is awesome for both.

Kubernetes is very much still under development. Most of the work up to 1.0 has gone into refinement of the API, rather than stability. For instance, HA is only now being seriously worked on - and if your API server goes down and you need to start it on another host, if you don't have an external reconvergence mechanism you have to restart every kubelet so they can point to the new one.

Mesos is much more mature, but is more low-level, and schedulers like Aurora or Marathon do not give you the same service abstraction, but they have many features not currently present in Kubernetes. But, in return for being more low-level, you can run things that are simply impossible in a declarative system like Kubernetes. Personally I see Kubernetes-on-Mesos as being a really bright future choice, but like pretty much anything other than Marathon/Aurora on Mesos, it's pretty nascent.

> But, in return for being more low-level, you can run things that are simply impossible in a declarative system like Kubernetes.

Can you give an example? I know next to nothing about Mesos, Aurora, or Marathon.

It's like many things that are declarative - you can only do what has been explicitly enabled in the declaration handling logic. This is good and bad. It's good because it restricts what is possible (protecting you slightly more from bad users) and can make learning and using the system easier. But you can't go beyond this. With a non-declarative scheduler you can implement far more interesting (for better or worse) management systems.

One area where we've been seeing some interesting work happening is in stateful services running on top of schedulers. With something like postgres, there is interesting logic that must happen for responsible management beyond just throwing up 3 of them. You need to manage replication carefully. When a node goes down somewhere, you need to react somewhere else in a controlled way. With Mesos, you can write a scheduler that encodes your runbook. With kubernetes, you need a lot more out-of-band stuff.

Disclaimer: I work at Google on Kubernetes.

You're correct that the scheduler in the box is declarative. However, it is easy (well, as easy as writing a framework) to swap out a scheduler that might be a better fit for what you're looking for.

See this stackoverflow answer by one of the core Kubernetes team for more info: http://stackoverflow.com/questions/28857993/how-does-kuberne...

Yeah, the HA information was buried/non-existent in the docs until recently, I guess. We also ran into issues when replacing the master, b/c the kubelets where fixed to an ip and the nodes' configuration was basically immutable (provisioned by cloud-config).

What worked for us: Accessing the the master by DNS and putting etcd on a persistent volume. This way we're able to replace the master within a DNS records ttl. As the api server is not a hard requirement for the workload, this is HA enough for us.

I don't really know what the alternatives are. I'm hoping by the end of this series to convince at least myself that it's reasonably to run for personal projects.

There is Lattice, http://lattice.cf which is single-tenant Cloud Foundry. You can get it up via Vagrant or Terraform.

The underlying scheduler, Diego, has some interesting constructs, like supporting Windows, Docker, and buildpack-built containers (aka. droplets).

I had tried lattice but I don't think it is ready for production usage. When I reported a few security issues on their issue tracker, I've been told that lattice is just a stripped down version of Cloud Foundry designed for development usage.

I think it's obvious that k8s is designed for staging/production usage and I can't see where lattice fits there.

Also, lattice is not running docker containers, but garden containers. There are a lot of issues on their issue tracker about people not being able to run images from docker registry.

Hi, fair points, I would agree Lattice out of the box is not quite ready for production usage, mainly because it has stripped out a lot of the security/multi-tenant features from Cloud Foundry to simplify the experience.

That said I would say it is just a matter of time (this Fall) however. I mentioned it because it is a way to wrap one's head around Cloud Foundry ; really the production alternative to K8S is Cloud Foundry today, depending on your goal. (If you must have rootfs control and Docker, then no. If you want to stage/run your app in a scalable, reliable container scheduler with a set rootfs then yes.)

As for Garden containers, yes, the main issue has been security -related gaps between Docker images and the Garden runtime, as Garden locks things down differently. Most (not all) of these have been fixed in recent builds, as the push for Diego GA in Cloud Foundry gets close over the next few weeks.

Check out http://datacenteroperatingsystem.io/

Disclaimer: I work at Google on Kubernetes.

We would (biasedly) say that Kubernetes is ready for production today, and have lots of public folks (Samsung, eBay, Porch, and thousands more) who are doing so.

If you're in any way concerned about the management, feel free to get a trial on GKE (the hosted version of Kubernetes) on the Google Cloud ($300 trial free). We have a 99.5% SLA on it, so it's backed by Google engineers.

Really enjoying this series. Thanks, Kamal!

Great series, Kamal! Keep up the great work.

Reply on news.ycombinator.com