Hacker News Reader

Comment by tlrobinson | Hacker News Reader

Maybe this is a dumb question, but couldn't/shouldn't a KaaS (and other orchestration systems) just be a layer on top of "vanilla" CaaS?

View on HN

CaaS-es (i.e. "things that present themselves as a Docker daemon or something like it") don't allow you to provision IaaS-level resources like VMs or disks, merely connect your containers to existing resources. When CaaS-es do allow you to provision stuff (like e.g. Hyper.sh does), they do it through a direct IaaS-level API that is separate from the functioning of the CaaS itself.

The major cloud providers' deployments of Kubernetes (and other server-side persistent-orchestrator systems, like the venerable CloudFormation) are deeply integrated into the cloud platform they're running on, such that the orchestrator itself can provision resources for a container to run on as part of deploying the container. This becomes important when elastically auto-scaling a container, because each container might need e.g. its own disk, and you can't create them ahead of time if you don't know how many you'll need.

This also means that, unlike a CaaS, k8s et al can manage the very cluster that k8s is running on, scaling it out to suit the size of the current/estimated workload.

Theoretically, you can bootstrap k8s on top of a vanilla CaaS—this is how minikube installs "using" your local Docker install, and this is how deployable PaaSes like Flynn and Deis work. But this approach doesn't supply k8s with the cloud-specific integration it needs in order to provision stuff. It might work if you're deploying against something with a standardized API like OpenStack; but none of the major cloud providers are compatible with such APIs, and so they need to build their own k8s plugins that call their IaaS-level APIs, to make k8s work on their clouds.

Or, to put all that another way: if there were standard IaaS-level APIs for k8s to hook into, Docker (and the CaaSes that either use or emulate it) would just hook into those APIs itself, and there would be no need for a higher orchestration layer.

tl;dr CaaS doesn't orchestrate the underlying infrastructure whereas k8s primary purpose is to create a cloud agnostic way to orchestrate containers and the infrastructure that they run on.

Kubernetes has worked to not be coupled to Docker (http://cri-o.io / https://www.opencontainers.org/), so in a sense, it already is built on top of a "vanilla" container service.

Don't you need to setup network paths? And connect containers to each other..

Then setup/teardown in specific order.. and if you have state, keep a subset of containers alive to avoid loosing state (even if it's replicated).

Kuberneted is designed in a way where stateless pods (collection of containers sharing the same ip and identity) are decoupled from stateful ones. There are concepts designed into K8S that allows components to attempt to self-heal.

For example, a pod that requires a postgresql pod to connect to will fail and crash out. The scheduler will start a new one. If the postgresql pod is up by that point, then the rescheduled pod will no longer crash.

As far as the network paths, one of the really cool things about pods running inside a k8s cluster is that they can access any of the other pods, even if they are on a different node. However, pods typically reference services (such as postgresql) by dns name. You specify the set of pods that belong to the service by label selector. This allows the pods to come up, tear down, crash, moved to another node, while the service maintains a stable point of contact. It is quite brilliant, and other orchestrators quickly tried to copy it.

Stateful workloads are still difficult. Each distributed stateful system has its own way of setup and teardown. What we will probably see are custom Operators designed for each distributed stateful system, coming out over the years.

Reply on news.ycombinator.com