Hacker News Reader

Comment by hacknat | Hacker News Reader

Right tool for the right job. Is K8s too complicated? For some use cases it is.

They probably should do a better job of discouraging certain use cases, but calling their elevator pitch “bullshit” is hyperbolic.

There are exceptions to every rule, but a good rule of thumb is cluster size. If you’re managing less than 25 servers than K8s is probably over kill. As you start to creep north of 40 servers K8s really starts to shine. The other place K8s really shines is dynamic load. I manage anywhere from 600-1000 16Gb VMs, I can’t imagine doing it without K8s.

If cluster size isn’t a good rule of thumb then application architectur probably is. If no one person in you company has a complete mental model of the application architecture or it is impossible because it is so complex, again container orchestration might be a good way to go.

Final point: If you’re struggling with K8s swallow your pride and buy a managed solution, then learn as you go.

View on HN

Even though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful.

Shameless plug, I‘ve been working on a project explaining how to run small scale clusters in the cloud for more than a year: https://github.com/hobby-kube/guide

You may want to update your tutorial to use cert-manager because kube-lego is in maintenance-only mode

You mean, it's stable. That sounds like a good reason to use it, not to avoid it.

It doesn’t support ACMEv2, Wildcard certs, and more. And Certmanager is alpha. The alternatives are all horrible.

Not really; I had an issue that silently broke renewals. When I found the open issue that corresponded to it the maintainers were herding people to cert-manager. There are a lot of issues where they are doing that (besides very obviously at the top of the README with warning symbols).

At the very least it's eventually going to break on newer versions of Kubernetes. Maintenance-only != LTS.

Maintenance-only usually means deprecated. There are some areas in the software world where maintenance-only can last a long time and can be de facto LTS. The Kubernetes ecosystem is not one of them.

From the Kube-lego github page, it looks like the last version of k8s is supports is 1.8, which will only be supported until 1.11 comes out.

So sounds more like it's deprecated than stable.

Depends on where you come from. In JS world, stable means there's a fork in it, and it's grayer than last weeks' meat. In server-side world, that means you're good to build a business on it for 15 years

cert-manager is marked as alpha/non-production currently.

(Off Topic) interesting that we're seeing two radically different philosophies of software development in this thread, in stark contrast.

So is kube-lego.

Thanks for the hint, I didn‘t know that. Related issue: https://github.com/hobby-kube/guide/issues/47

Ooh, thanks, this is just what I've been looking for.

> I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session

Is it grammatically possible to use more buzzwords in a single sentence? Reading that made me want to vomit.

The words have meaning. If you don't understand them you could just ask for them to be explained rather than throwing out insults. I suspect you actually do understand the meaning of what was written though so I suppose that means you're just trying to start a flamewar.

The problem is that "leveraging" can be replaced with "using" every time, with the added benefit that it won't leave a bad taste in the reader's mouth.

This is what I mean.

You: Look what I did with k8s

Top voted reply: You had to use a thing to do what you want, and that thing isn't even supported any more. [Either I was hallucinating at the time and imagined it, or there was a reply that got deleted]

I think you've just quite nicely demonstrated the original authors point.

You sound like you‘re new to this business.

Sometimes the people "new to this business" have the best ideas. It's more likely that I'm a grumpy old man worried about his lawn.

How did I not find this one month ago, as I set up a two node hobby cluster?!

>"Rook and an Nginx ingress controller with kube-lego"

Can you elaborate on this configuration, specifically Root and Kube-lego? I am not familiar with those.

Thanks for sharing about hobby-kube.

I'd say it's actually just a convention vs configuration argument.

Kubernetes isn't giving you anything you couldn't have already built with configuration management. It just happens to a standard written by a bunch of people with a background in the problem domain.

Personally, I think the abstractions are thoughtful and the system isn't really inherently more complicated than what you'll eventually build anyways, but as the article said, we all have a bias towards that crap we ourselves invented, because we already know how that works, and we find learning something like kubernetes a chore.

That said, I wouldn't say it's about how many servers you manage. If you have a single monolith, written and ran by a single team, it doesn't neccessairly get more complex as you throw more servers at it. But when you have several teams, writing and hosting several systems, it's nicer to have a convention framework instead of an undocumented snowflake platform.

I have to agre on your final point though. If you find running kubernetes complicated, go to GKE and treat it like you do your IaaS provider, which is more complicated but isn't something you typically deal with.

> Kubernetes isn't giving you anything you couldn't have already built with configuration management.

Configuration management does not give you loadbalancing. Configuration management does not give you rolling upgrades.

I mean, sure, you can do this stuff with CM as well, but with k8s, there's nothing to build. It's already there

You're trading operational complexity in other areas for arguable operational functionality that CM of haproxy (or perhaps Microsoft's load balancer if you need UDP LB services) already provides for, in software that is still under active development and has sharp edges. Your systems should be as simple as possible, but no more simple than that.

Kubernetes is not turnkey and requires a significant operational time and resource commitment. Prepare accordingly.

Ansible gives both of that.

That's the point of convention vs configuration argument but we're on the same side of the argument. :)

And there’s two sides to the equation - even if you have a world class orchestration platform if your application architecture is extremely stateful and can’t tolerate certain services being down the platform can not help. Most places I’ve observed (large and small) simply do not write software in a manner that can be containerized effectively. The abstractions and patterns developers have as a default in most languages tends toward system designs (emergent) where people as a rule write and cache files locally, datastores are expected to stay up forever and never change hostnames, and that service discovery is “too complicated” to ever use.

Configuration management absolutely can do this. Note that I'm a builder of kubernetes clusters (on-premise) but also have contributed 400+ patches to the salt configuration management tool.

Salt has an "orchestrate layer"[1] which allows running states on sets of minions. One of those layers can be to configure a service, and another can be to update the load balancers when said service is healthchecking green. Saying these things simply can't be done with configuration management is utterly false. Kubernetes just makes it easier and more approachable for those less skilled in under the covers systems and infrastructure stuff. Kubernetes is a tool that allows you to build things quickly, but it isn't for everyone.

[1] https://docs.saltstack.com/en/2017.7/topics/orchestrate/orch...

It's not about skill and approachability; it's about not having tens of thousands of systems in different companies built out of hacks and science experiments, where experience isn't portable and every system's hacks and idiosyncrasies need to be learned anew. K8s raises the abstraction level on deployment of a multi-node application architecture.

Further, as it builds out standard abstractions for external services, it should enable applications to be written agnostic as to the specific cloud provider. That's pretty valuable. It has the potential to reduce AWS's lock-in advantage.

Yes, I'm aware and you're preaching to the choir. I'm literally the lead on $employer's k8s deployment. I was just pointing out that things like k8s can absolutely be done reliably with old school bare metal and config management + some orchestration glue. Only it isn't as simple for most and generally is done poorly. I'm a huge fan of k8s.

> If you have a single monolith, written and ran by a single team, it doesn't neccessairly get more complex as you throw more servers at it.

In which world does our monolith only connect to a database that is hosted on a single node ;) Most often we have at least 3 database nodes 3 x nodes, 3 redis nodes, whatever. even when we run everything on vms instead of bare metal instances (for which k8s even works) we would still need to manage them and create an insane amount of work to manage/keep them running. in the worst case people would ssh into them

If no one person in you company has a complete mental model of the application architecture or it is impossible because it is so complex,

One caveat to this: architectures that are complex because of poor design should probably be redone rather than hidden behind yet another layer of complexity.

Or find someone to train you. The Kubernetes stack is deep. There's no shame in being taught this stuff. Nobody is born already knowing all the nuances.

As an OpenShift consultant for Red Hat, in a single day, I've run a tcpdump trace to figure out why a particular NFS vendor had trouble with traffic over a new network vlan, then, spent an hour or two with a Dev in ChromeDevTools to help them get session management cookies straightened out. Some people call that a day spent as a "Full Stack Engineer", I call that Tuesday. And I can't tell you how many obscure corner case scenarios I've found bugs in Kubernetes, OpenShift or a client's App.

I think it's actually good as soon as you're doing microservices and don't want to use any managed vendor lock in service like app engine.

The setup is easy if you use a managed kubernetes offering and it provides so many things out of the box you'd otherwise have to take care of yourself. Service discovery, service lifecycles, updates, storage management, logs, load balancing (inter-service), all those things you should automate even if you have 5 servers, and k8s makes this darn easy in my opinion.

The declarative abstraction is also very easy to use and intuitive in my opinion.

I routinely take app engine apps and run them on the FOSS AppScale platform unchanged. For the things you can't port, like using Google as your auth, there are 3rd party solutions for that already.

Choosing Google's App Engine is more than just choosing a managed solution, too. You choose it because of Google's reach: consider, for a moment, how you send a push notification to an Android device. When you do it from Google's cloud, your metal is co-located with their push infra. Also caching, logging, tracing and even a global CDN is integrated into the platform. Think of the cost of having to sort this yourself.

Docker compose can handle simple container situations and many PoCs just fine, and part of the work will be analogous to the Kubernetes solution.

If all you have is a handful of load balanced web servers and a database you can hold off on kube for a while.

The feature gap between compose and k8s is starting to add up though. Compose is nice for tutorials and POC within a containerized environment but IMO not that good for full-fledged production.

Interesting, what do you think should be used to manage a cluster of less than 25 machines?

It wasn’t a hard rule. It depend on the app and the use case, but probably just docker on all the machines with some static cloud loadbalancers in between everything. Or just straight config management.

Hosted k8s. all the benefits with almost no extra maintenance.

Okay, how about for a tiny budget?

Nomad (www.nomadproject.io) is what I would recommend.

Nomad is ONLY a task scheduler across a cluster of machines. Which is why it’s not rocket science, nor is it operationally complex.

You say I need X cpu and X memory and I need these files out on disk(or this docker image) and run this command.

It will enforce your task gets exactly X memory, X cpu and X disk, so you can’t over-provision at the task level, Docker and friends don't do this.

It handles batch(i.e. cron) and spark workloads, system jobs(run on every node) and services (any long-running task). For instance with nomad batch jobs you can almost entirely replace Celery and other distributed task queues, in a platform and language agnostic way!

You can do load balancing and all the other things k8s does, but you use specialized tools for these things:

   * For HTTPS traffic you can use Fabio, Traefik, HAProxy, Nginx, etc.
   * For TCP traffic you can use Fabio, Relayd, etc.

These are outside of Nomad’s scope, except that you can run those jobs inside of Nomad just fine.

and it’s all declarative, a total win.

> You can do load balancing and all the other things k8s does

Nomad can do many of the things that k8s does, and even some things that k8s can't, but it does not do all the things that k8s does. For example, nomad doesn't have network policies.

Also, just to approach the base level of functionality and security in k8s you need to setup consul as well and bootstrap all the tls stuff, and at that point it's not much more work to get k8s going.

Nomad is pretty cool, and is rapidly catching up to k8s, it will be interesting to see what happens in the long run.

Yes, True enough. Arguably, network policies are outside of Nomad's scope. It's a resource scheduler. Nomad doesn't turn up interfaces for you, or do routing of IP traffic.

Nomad is not a one-stop shop, like k8s tries to be, it does resource scheduling in a nice declarative manner, that's about it.

It's much more in the unix toolset philosophy, let 1 tool do 1 thing well, and make integration as painless as possible.

k8s is much more in-line with the systemd way of thinking, it owns all the things and can be the only shiny in the room.

I'm not sure I'd agree that turning up k8s is as easy as turning up consul, nomad and vault. K8s in my experience tends to require lots of babysitting. Consul can require some babysitting, but vault and nomad require basically no babysitting, except the occasional, mostly painless upgrade.

For me it's really about operational simplicity. I have to keep this stuff running tomorrow and next year. Shiny, super-fast moving be all and do all stuff like k8s, Openstack, etc tend to be a giant pain that requires loads of babysitting(most notably at upgrade time), as the edge cases tend to hurt a lot, when you hit them.

I don't really see how nomad is operationally simpler than k8s. To run a service behind something like traefik on k8s I would:

  bootstrap a CA for cluster tls
  run etcd
  run k8s (apiserver,etc,kubelets)
  run your application
  run the traefik ingress controller

To run a service behind something like traefik on nomad you would:

  bootstrap a CA for cluster tls
  run consul
  run nomad (servers, clients)
  run your application
  run traefik somehow

I think the only thing that is really more complicated in k8s is the networking stuff, but that's only because it has more features, like having cluster dns automatically configured and being able to give every pod its own ip address which means every service can bind to port 80 without conflicts, and policy enforcement.

We are talking about different things. I'm talking about keeping Kubernetes/Nomad alive and breathing and happy. The ops part of devops. You are talking about running stuff under them. I agree they are similar in running applications under them.

Operationally simple:

* 1 binary, for both servers and agents. * 1 config file. For consul & nomad 2 config files. * Upgrades are simple, bring down a node, replace binary, start it back up.

Docs are straightforward, it's easy to understand how Nomad works it's not complicated, you can get your head around the server/agent split and nodes and everything in an hour probably (for both nomad AND consul). k8s is a very complex beast, with many, many binaries and configs, there are helpers that get you setup, but they all have their own pros and cons, and sharp edges. Chances are you would not want to use a helper in a production setup, which means you have to understand all those moving parts.

Keeping a k8s cluster running nicely and upgraded consistently requires many full-time admins. Keeping a nomad cluster running requires very little work(I do it part time, maybe an hour a month on a busy month).

Arguably for dev/testing under consul & nomad, you would do this:

  consul agent -dev &
  nomad agent -dev &
  nomad run myapp.nomad
  nomad run traefik.nomad

Adding vault to the mix for secrets: vault server -dev &

For production use it's obviously more involved, but most of that is just around understanding your operational requirements and getting them working with nomad, not really about nomad itself.

No, I'm talking about operationally too. If your consul and nomad deployments are only one binary and one config file then you're not using TLS . Half the effort of setting k8s up is bootstrapping the CA and certs for etcd and the k8s components

> k8s is a very complex beast, with many, many binaries and configs

Because it's much more in the unix toolset philosophy, let 1 tool do 1 thing well. Is that a bad thing now?

hyperkube does put all the server side components in a single binary, there's still a bit of configuration though. A lot of the options are repetitive, I bet one could wrap hyperkube with a single config file and some defaults and the end result would look a like like nomad.

You keep going on about setting k8s up, and not about maintenance. How much time in a week do you take to babysit your k8s cluster? Do you have an HA setup?

OK TLS takes 3 files, 2 for the key and crt and 1 for the config. If you get your TLS certs out of the vault PKI backend, it's very, very simple (https://www.vaultproject.io/docs/secrets/pki/index.html) the linked page covers the complete steps.

Again, I keep talking about maintaining Nomad/k8s for years. I've been running nomad in production for a few years now, I've had no downtime from nomad, and I spend about an hour doing upgrades every once in a while. I don't worry about nomad, it's just there and works. I run 3 nomad servers per data center for an HA setup. k8s doesn't even test their HA setup in development (source: https://kubernetes.io/docs/admin/high-availability/building/) . There is no way it works out well in real life, if they don't even test it yet.

Nobody I know that runs k8s pretends it's easy to keep running for years. Most places that run k8s have dedicated engineers to babysit k8s. I babysit our nomad, and lots of other infrastructure, and I do development of applications as well.

> How much time in a week do you take to babysit your k8s cluster? Do you have an HA setup?

I don't have a k8s cluster... so zero :-)

I don't have a nomad cluster either, because every time I look at it and start planning out what I would need to do to bootstrap consul+nomad and secure it, it starts to look more like a k8s install.

> There is no way it works out well in real life,

except that every cluster on GKE or created using kops, kubespray, or even kubernetes the hard way is HA, so it's not like no one is running an HA cluster. I think from k8s point of view, there isn't much to test as etcd is doing all the work.

Setup and install is the least of your issues when running something like nomad/k8s in production. The part that matters more, is what's it like to babysit it, and keep it running.

I agree people are running k8s HA in production, but there is a reason those people are dedicated k8s engineers. It's because it's a giant pain the ass to keep it running. Hence what I mean when I say it's "operationally complex".

Most people using GKE don't actually operate the k8s cluster, they let GKE run it for them. They just use it.

Using k8s and using nomad are similar from a developer perspective. Operationally they are night and day different.

Anyways, I suggest you go play with both systems, and try them out, put some non-important stuff in production under both of them.

Hosted k8s on google can be as little as $50/month. If your project has really little load, I think serverless with docker containers is the way to go, which might be under $1/month.

However that's not supported yet by Google or AWS, so you have to find another provider.

what if you can't use hosted k8s. what if you are on vmware 3 node (no vsphere). what if you at least need consul/etcd, postgresql, haproxy + two nodes (should be as highly available as possible). how do you keep everything up to date, how do you deploy (no downtime) without a complex ansible config.

for myself I use k8s either bootkubed or kubeadm, I create ignition configs via cloud-config (coreos) and create coreos nodes via the vmware gui. after that I ssh once into either one node (bootkube) or in all nodes and call kubeadm or bootkube after that I have a working cluster. Maintance is done via coreos update operator. and I can upgrade k8s via either kubeadm or via kubectl on bootkube. (so basically the most complex thing on k8s is updating k8s, everything else isn't as hard as people thing. p.s. I do not have a bach/major in tech).

What did you find complex about updating k8s? Serious question since I work closely with k8s and they have a pretty fast release cycle.

running into bugs and that I need to ssh into the nodes ;) (serious answer) I can basically automate nearly everything, os updates (coreos), deployment updates (ci/cd), however updating etcd or k8s is a manual operation on bare metal, either I'm on bootkube than I need to login and update kubectl, than I need to update all daemonsets/deployments in kube-system (it's not always easy and bootkube has strange bugs, https://github.com/kubernetes-incubator/bootkube/issues/977) So one should actually use kubeadm, once again I still need to update kubelet, so I need to ssh into the machine (anyway) I also need to update kubeadm and run it, again a manual operation.

*ssh means I need to do it on my own, for small clusters this can be done manually for bigger clusters it can be scripted via ansible or even better create an own software that does it. My point is, that's the only thing which is not "just works" on k8s (yet).

Edit: I also run into a bug that I could not upgrade k8s with kubeadm (but thats gonna be fixed soon: https://github.com/kubernetes/kubeadm/issues/727) Edit2: the good thing is that reloading kubelet won't kill anything.

I'd look at using a regular configuration management system for maintaining k8s itself, e.g. Puppet.

Similar to a couple of the other replies, updating kubelet or the like tends to touch enough dependencies that it starts looking sensible to do updates as rolling VM re-provisioning and rescheduling tasks, so you just sidestep the lifecycle bit of a kubelet upgrade gone pear shaped.

I haven't had to touch any of that since February though, but I will be again soon. I hope kubeadm and its ilk have gotten better at handling the lifecycle of a long lived cluster. I've done all that in terraform (from scratch), and I'd really rather have a more standardized option that's not buy $HOSTED or $VENDOR solution.

If you're on AWS, I highly recommend Convox [1]. They use AWS ECS instead of k8s, but EKS support will be coming soon.

Their rack project is free and open-source [2]. They also have a hosted console that can manage your racks, deploy from GitHub webhooks, and send slack notifications. But you can use convox without that.

[1] https://convox.com

[2] https://github.com/convox/rack

At less than 25 machines throw it on Heroku. Yes, it’ll be more expensive hardware wise, but you’ll probably save that much in not operating you’re own servers.

What good alternative can you suggest for less than 25 servers?

Docker Swarm

Mesosphere

K8s is not the only game in town. The question is not k8s vs. old-school ssh and apt-get, but rather k8s vs. Hashicorp Nomad or AWS Fargate.

For a few microservices, Nomad is really nice and simple.

Or mesos/marathon, though I'm in no way qualified to make comparisons vs k8s.

Or mesos/marathon, though I'm in no way qualified to make comparisons vs k8s.

I am qualified to make that comparison, and Mesosphere all the way. Rapid, no bullshit deploy, not having to deal with key stack components being alpha, simple management, wide support (including running k8s on mesos if you are so inclined). There is a very long list of why mesosphere can be considered to be a stable and mature product.

Every time I work with k8s I think "this is cool and all, but absolutely not ready for production use" - and in my book production use means not needing a small army of sysadmins (sorry, SME's) to keep it running, not changing core components every change of the moonphase, the majority of core components being stable and battle-tested, and a learning curve that doesn't look like a altitude-vs-time plot of the latest SpaceX launch.

> Final point: If you’re struggling with K8s swallow your pride and buy a managed solution, then learn as you go.

Can you make vendor recommendations, please?

Less than 25 servers is not overall when you take into account people generally choose a larger instance type.

Reply on news.ycombinator.com