Hacker News Reader

> K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs.

So "all" we need to do is rewrite systemd as a distributed system, and make it "cloud-focused" and "modern"?

No thanks. Right tool for the right job.

It's basically a distributed system already! All you need for feature parity with K8s is a parent node sending child nodes commands via its existing API. Everything else is already there.

So, like systemd socket activation (which is already there, too)?

More like the job controller: https://github.com/kubernetes/kubernetes/blob/master/pkg/con...

It's a loop that schedules jobs. When you create a job it starts them, when one crashes it schedules another, when you delete one it stops them. Systemd already does this for a single node's service units, so all that's lacking is doing this across nodes.

Yes... kind of. A distributed system all on one machine skips many of the hard parts. How does systemd handle a network partition? How does systemd mount storage or direct network connections to the right nodes? How do you migrate physical hosts without downtime?

> How does systemd handle a network partition?

The same way K8s does... it doesn't. K8s management nodes just stop scheduling anything until the worker nodes are back. When they're back the job controller just does exactly what it was doing before, monitoring the jobs and correcting any problems.

> How does systemd mount storage

Systemd has a Mount unit configuration that manages system mounts.

> or direct network connections to the right nodes

The same way K8s does... use some other load balancer. Most people use cloud load balancers pointed at an Ingress (Nginx & IPTables) or NodePort. The latter would simply be an Nginx service (Ingress) listening on port 80/443 and load-balancing to all the other nodes that the service was on, which you can get by having the services send Nginx an update when they start via dynamic config.

> How do you migrate physical hosts without downtime?

The same way K8s does.... stop the services on the node, remove the node, add a new node, start the services. The load balancer sends connections to the other node until services are responding on the new node.

> The same way K8s does... it doesn't. K8s management nodes just stop scheduling anything until the worker nodes are back

This is not entirely accurate. K8s makes it possible to structure your cluster in such a way that it can tolerate certain kinds of network partitions. In particular, as long as:

* there is a majority of etcd nodes that can talk to each other

* there is at least one instance of each of the other important daemons (e.g. the scheduler) that can talk to the etcd quorum

then the control plane can keep running. So the cluster administrator can control the level of fault-tolerance by deciding how many instances of those services to run, and where. For instance, if you put 3 etcd's and 2 schedulers in different racks, then the cluster can continue scheduling new pods even if an entire rack goes down.

If you assign the responsibility for your cluster to a single "parent" node, you're inherently introducing a point of failure at that node. To avoid that point of failure, you have to offload the state to a replicated data store -- which is exactly what K8s does, and which leads to many of the other design decisions that people call "complicated".

Oh and what if that parent node dies, now all the children are orphaned and without a leader to tell them what to do? Kind of breaks down.

The problem is a distributed state and consensus. It's not as simple as "just have a leader send nodes commands". There are well known algorithms and systems for dealing with distributed consensus. It might annoy you to find out kubernetes uses one of them (etcd)! Use kubernetes!

You don't need it. A single synchronous redundant storage mechanism (e.g. SQL database, S3, SAN/NFS) works fine. But you could use etcd (or Consul) if you wanted. (You also didn't mention what happens when your etcd gets corrupted or loses consensus or its TLS certs expire... :)

K8s is a very "opinionated" system. People think this is the only architecture that works, but of course it isn't true.

You do need distributed consensus--your 'single synchronous redundant storage mechanism' cannot work without it. You're basically just offloading the distributed system state to this magic system you're inventing. That's fine, and that's exactly what k8s does with etcd. If you want to boil the ocean and reinvent k8s because you don't like its yaml or whatever, that's cool and I hope your employer supports you in those endeavors.

This is a very low-effort and dismissive comment, but also incredibly wrong. You do not need consensus to coordinate a distributed system. A master/slave network architecture is designed so that a single node can maintain consistency, or multiple nodes using a state storage mechanism with exclusive locking. Almost all distributed systems for the past... oh, 60 years, have not required consensus for consistency. Only modern ones that utilize Paxos etc made distributed consensus practical.

Then again, 'consistency' is a vague and ill-defined term in distributed computing theory and maybe you were only talking about a consistent distributed decentralized network with consensus.

How does your state store stay consistent, and scale to a large load of readers and writers?

Yes if you have the trivial case of a single node state store then there is no problem. You scale it by chucking more CPU and resources at it until it can't go any faster.

But in the real world that will only take you so far. How does your state store scale out to multiple machines? How does it handle potentially thousands or in extreme cases hundreds of thousands of writers a second? How do you keep multiple nodes in the state store consistent?

There's your distributed consensus problem.

Like https://github.com/coreos/fleet?

> All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it.

For example, two operators I'm currently implementing for a client, ArgoCD and RabbitMQ Topology operator. The first allow GitOps, meaning that we update config file and the software running in the clusters immediately gets updated or configuration is being set. The second let me declare exchanges and queues like any other Kubernetes object. Now I'm sure it is possible to do the same in SystemD, anything can be done with enough engineering. But is it really the right tool for the job? in any case, the function of Operators is not to simplify and complex abstractions, this is simply incorrect.

> All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it.

CNI isn’t some silly Kubernetes abstraction to scratch an itch.

CNI is a specification and a set of libraries for container networking.

Calico is built on top of CNI, Nomad requires CNI for multi-machine networking, Mesos uses CNI for network isolation, Firecracker uses CNI extensively. Docker does not use CNI directly but their libnetwork code depends on exactly the same principles as CNI. Life everywhere is simpler thanks to CNI.

Do a thought experiment. If you think it’s that easy, why don’t you try marrying systemd with Docker overlay networks and see how far you get?

I have the "If you have a SystemD everything is a nail" feeling from this comment.

Systemd is great... for one machine. Systemd has no concept of multi-machine orchestration. It has no distributed state or consensus which is the foundation of multiple machines orchestrating work. I suspect systemd by design will _never_ try to do multi-machine orchestration and prefer people use an existing solution... like kubernetes.

I love systemd but it's not designed for managing a cluster of machines.

> The main purpose of K8s is just to run microservices.

No. The main purpose of K8s is to make portable « running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ».

Systemd could do all that, but, then, it would not be Systemd anymore and would be like another K8s.

You can run containers under systemd already via systemd-nspawn (and even manage/orchestrate them via machinectl) but configuring them is just as complex as on any other containerization solution... Don't see much of a point. And containers provide a lot more than just replicating static linking, they're about namespacing of all resources and not just a different filesystem root.

> The main purpose of K8s is just to run microservices.

Agreed, except I'd add 'stateless' to the microservices part. That's all k3s is good for and trying to make k8s do more is awful.

> Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone.

Far more is needed. CRIU needs to work; live migrating containers between hosts. Network overlay, with transparent encryption, name resolution and access control are necessary. Bulletproof network storage management is crucial. Containers need to easily float among hosts, and the networking and storage they depend on must follow seamlessly.

Most of the pieces exist, but we're a long way off. I believe the future is an operating system with integrated clustering solely focused on hosting containers and the networking and storage they need.

You don't need any of that stuff to run microservices. Stateless is up to the app, not the orchestrator (you can still run a stateful containerized app in K8s). You absolutely do not need containers, network overlay, "transparent encryption" (do you mean a service mesh?), name resolution (dnsmasq works fine), and access controls are inherent in systemd. Network storage is unrelated if you mean NFS; if you mean persistent storage volume claims shuffled around nodes, yeah, that's... not a stateless microservice. And.... no, we don't need CRIU, but it works just fine without K8s.

All you need to run microservices is 1) a statically-compiled stateless app, 2) a systemd unit file to run the app and restart it when it stops, 3) load any credentials into systemd's credential manager, 4) iptables for network ACLs, 5) linux user/group permissions, 6) an Nginx load balancer to route traffic to different services, 7) journald to keep logs, 8) systemd to set per-service CPU/memory limits, 9) a dynamic DNS client to register a service name in dnsmasq when a service starts/stops. Systemd already handles intra-service network communication by handing an open filehandle to a service; alternately each service could grab its own random port and register it with Nginx along with its service name and the HTTP routes and hosts it wants to listen on.

If you shackle yourself to the design of K8s, or some implementation of K8s (K8s doesn't have all the features you described out of the box) and say "well that other system isn't K8s!", then yeah, you will find systemd wanting.

You could run Nomad and use parameterized jobs with raw_exec to get something like this. I know that systemd has some lifecycle stuff where you can create/run systemd units directly through the cli instead of using unit files which should allow you to hook the lifecycle into a nomad style service invocation but I can't recall exactly how to do this.

This doesn't quite roll into the 'only systemd,' but it allows you to build exactly what you're looking for.

I would rather systemd added a single feature to its already vast arsenal, and then Nomad wouldn't be needed either.

Figure out how to revive https://github.com/coreos/fleet as something native in systemd?

Yes, but ideally much simpler

What scale do you have in mind for your solution? How many applications, how much resources would you comfortably run like this?

“You don’t need Kubernetes to run stateless apps, you can do it simpler by…” goes on and lists 20 things one needs to deploy, maintain and learn to manage.

I don’t think your solution is simpler but you are right about one thing: one does not need Kubernetes, alternatives exist.

> CRIU needs to work; live migrating containers between hosts

Wait, k8s can use CRIU to live migrate containers?

Not commonly, I've never gotten CRIU working in real life. AFAIK it's not feasible unless you go all-in on a VMWare stack.

Ha, I had a similar thought and started creating exactly that: clusterd, a way to cluster and schedule things on systemd instances. I haven't gotten very far, but if you're interested in helping me please reach out (email in profile).

How would you run multiple instances of an app on a single machine with systemd? I.e. if they are listening in the same port.