Hacker News Reader

Comment by ggregoire | Hacker News Reader

I'm wondering if Kubernetes would be the right choice in our use case. Or if something like Ansible would be better suited and easier to setup and use?

I'll soon have to manage 50 remote bare-metal servers, all of them in different cities. Each one of them will run the same dockerized application. It's not 1 application distributed in 50 servers; it's 50 instances of the same application running in 50 servers, independently.

A frequent use case is to deploy a new version of this application on every server (nowadays I do it manually, it's OK since I manage only like 10 servers).

A nice-to-have would be to avoid downtimes also, when I update the application (nowadays I have a downtime of 2-5 minutes (when it goes well), which matters for us).

View on HN

No, as @dragonsh said, Kubernetes will be a bad choice for you. Kubernetes is complex, specially its network stack, you'll need to setup an VPN from each site to your control plane site to keep some basic functionality working (kubectl logs, for example).

If you don't care about a centralized API to probe status and manage each instance, Ansible should be enough to orchestrate these installations and with little effort (that also depends on the application at hand) getting zero-downtime rollouts with Docker can be easily done with it.

However, if you want a single control place to probe status and want to avoid writing your own rollout scripts, Hashicorp's Nomad [1] might be a good solution for this. It is a lot simpler than Kubernetes while still giving you nice primitives to describe jobs/services, health checks, rollouts strategies and etc. Treat every site as a datacenter of its own, setup a job of type "system" (akin to Kubernetes DaemonSets) and all you need on these sites is internet access to your HTTPS endpoint of the Nomad control plane.

If you want to talk more about this, hit me up on Twitter or Telegram, I use @rochacon as my handle virtually everywhere.

[1] https://www.nomadproject.io

Edit: grammar and typos

I would also investigate Terraform if the Ansible path appeals to you.

How can zero downtime deploys be done with Docker?

I had to write my own custom blue/green deploy script to hot reload traffic to proxy_pass definitions in nginx upstream configs since I don’t use Docker.

IIRC, if you use docker swarm it wil handle the routing for you. So if an app is running on port 8080 on two swarm nodes, if you access port 8080 on each node, you might actually be accessing another node.

Since docker swarm knows if an instance is down, it will know to not use that instance.

See https://docs.docker.com/engine/swarm/ingress/

I run on Docker Swarm and there are hiccups as one container goes up and another goes down during deployment.

I think you can try a tool like Envoy for doing routing between versions plus some other custom stuff.

At my job, the devops team had to write some kind of special program that maps whether a Marathon/Mesos instance is up when dealing with Envoy. Not sure if the same is required for just plain Docker?

Agreed, Nomad seems perfect for most use cases where cluster management, deployments and rollouts is required.

It's a bit overlooked now because every DevOps person nowadays seem to think Kubernetes is the only rational thing as it will look good on a CV.

I predict Nomad will be on the upswing the next few years as people realize Kubernetes is extremely complicated to self host.

No kubernetes will be a bad choice for your use case. But if you learn all the intricacies and manage a behemoth besides your own application it will help in your resume to brag that you are a modern devops and a fashionable one using kubernetes.

If you need a practical suggestion use simple solution like LXD [1]. But as you are already using docker stay with it and ansible.

As you are using ansible you are already doing infrastructure as a code, you are way ahead of curve already.

Probably when you grow a bit bigger use packer [2] and terraform [3], but I think ansible will do just fine.

Kubernetes was designed for Google kind of problem the burden to maintain is quite high with lots of moving parts, unless you use Google GKE or Amazon or managed kubernetes service. So don't fall for it unless you need it for your resume.

[1] https://linuxcontainers.org/

[2] https://www.packer.io/intro/

[3] https://www.terraform.io/

GKE and EKS are very expensive which in my opinion makes using them unattractive for smaller scale projects. But Digital Ocean now offers managed Kubernetes for free, I've played around with it and it actually seems really easy to set up and manage. Would be great if the competition would follow suit.

> GKE and EKS are very expensive

> Digital Ocean now offers managed Kubernetes for free

GKE is just as "free" as DO, in that you only pay for the nodes you actually use (those nodes are probably a bit more expensive, admittedly). EKS is the outlier in this trio in that you also have to pay (a lot) for the master. Could you clarify what GKE could do to "follow suit"?

It seems, nothing! I was just wrong about GKE being expensive, not sure why I thought that (though it seems that until late 2017 they did have costs per cluster).

This sounds like Chick-fil-A's use case where they decided to put a k8s cluster in each restaurant. https://medium.com/@cfatechblog/edge-computing-at-chick-fil-... https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at...

If you have a single node not a cluster in each location then k8s buys you a lot less. Some people still advocate single-node k8s "clusters" for benefits like rolling deployment.

I’m still stuck on the Chick-fil-A part. Haha.

It's handy to have all those k8s instances scale to zero on Sundays.

I am fairly certain the second the old dude dies they are going to be like "Motion to open on Sundays..." in the board room.

S. Truett Cathy (who I assume is the old dude you mentioned) passed away in September of 2014.

Ah I did not know that. I figured he was still hanging out and getting a kick out of refusing to open on Sundays, given how much of meme it has become.

actually rolling deployments are painful. it's like THE feature of k8s. and since k3s is so simple, it's alo not to painful to run it.

I've seen rolling deployment using plain distribution upgrade happily working on larger fleets than that, but that's borderline uncivilized.

Echoing others, you really don't want Kubernetes here.

I only have limited experience, but Kubernetes works very hard to abstract away the actual machines, and it works best that way: you just say "Deploy 20 instances of job X", and k8s will somehow find the machines to deploy them. You don't care where they are running - k8s handles that.

Once you start to care about actual machines and which jobs are running where, k8s starts to make less sense. You're paying for huge complexity (required to abstract away the machines), but you're getting none of the benefits - it just becomes a glorified wrapper around Docker daemon.

I struggle to understand when Docker is supposed to be introduced to a stack.

If you have a 100 user SaaS platform, do you run Linux containers on top of Linux just for isolation?

If you need strong guarantees of isolation and if every user has their own dedicated installation. If you are a general service where isolation is enforced at the application level then just deploy straight away.

Docker is useful when you don't want to care about the underlying operating system or you might be deploying on different operating systems etc. Also if you want to quickly switch providers (like going from Ubuntu 18.04 on digital ocean to Windows Server on Azure) since it's a self contained image.

This self contained image is useful during development too. I can cleanly separate dependencies between applications and dev/test/prod. For example I develop a node app based on node 12 + pg 11 but I tinker with ghost too which requires node 10 + mysql. Having this in Docker keeps cruft and complexity off my macOS installation at the cost of space. Having multiple versions of a runtime/database etc can quickly become a nightmare to manage on local operating systems.

Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.

> Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.

On that same note... when do Terraform / Ansible start to make sense?

I have found that Ansible is necessary if you have more than one server and useful even for less than one (eg managing a service account on an old legacy shared machine owned by another team).

I use terraform mainly for provisioning infrastructure (how many VMs on which platform, how many databases etc) while Ansible for configuring the software side of the deployment - like the dependencies, which version of code to pull, configuring nginx, cache etc

Containers are fantastic at isolating one software application from another. All their dependencies - runtime, library, packages - live in separate chrooted filesystems.

User-to-user isolation is hard, and extremely hard if any of the users could be malicious. Docker is broadly neutral here - adds a few tools, might add a few vulnerabilities.

It can be introduced at many points.

in dev: You can setup dependencies without worrying about underlying os. So it is easier for building a new dev machine, adding a new team member, etc.

in integration test: you can setup multiple integration test environment as long as you have enough resource. Running multiple integration tests without interfering other team members is the major benefit here.

in prod: as you may know, for isolation and horizontal scaling. Easier roll back and update?

Docker in production is a potential disaster, it runs everything as root from the daemon, we are heavy k8s users, and we are transitioning to rootless cri-o containers specificaly because of this. Investigate podman which is a docker cli clone that builds cri-o containers from docker files and allows you to run them under a designated user id, the only restriction is that in rootless mode you cant map ports that are less than 1024, for that you need to run them under sudo which defeats tbe purpose. Howver we use an ingress controller to map inbound requests to containers so it can do the port mapping. We used to use traefik, but now we use nginx as ibgress, becuase we can run the excellent lua-waf module.

We run on multiple clouds, so vendor apis and k8s implementations are not usefull to us.

Ansible all the way.

I'm really upset with the way Google let their marketing team run roughshod all over the place with that software. Kubernetes is almost never the tool to use. It's entirely insecure, overly complicated and almost never fits the intended supposed benefit. Worse it feels that the entire CNCF ecosystem is ran entirely by marketing people with "developer evangelists" that have never coded a single line of code in their day - it's a real shame and quite honestly an insult to professional engineers.

I don’t understand when to use Ansible, Puppet, Chef, or Terraform...

Terraform: First you use this to create VMs.

Ansible/Puppet/Chef/Salt: Then you use this to install your stuff in the VMs. Just pick one of these and stick with it.

Install k8s with a/p/c/s. :)

K8s is not about configuration management, it’s about dynamic application management. Some parts infringe on areas where CM tools work as well, but k8s is all about managing containerized applications.

Trying to set up flows for ”works on my laptop”-dev, ci/cd, loadtesting, a/b tests, canary releases, autoscaling and rollbacks for multiple teams of devs? K8s really simplifies these things.

The idea is that you have one api spec to rule the whole stack (from a dev perspective). If you go down a more light-weight stack, a lot can still be achieved. More duct tape required though. That being said- I love duct tape!

Ansible can be used to create VMs too. I used to use it to provision AWS instances and to configure them after.

https://docs.ansible.com/ansible/latest/modules/list_of_clou...

Never use Ansible, Puppet or Chef. Those are old dead tools.

Those are tools for configuration management. If you instead use packer, docker, you can build your vm/image at build time, and use Terraform to setup vm with that image. Use etcd (in the image set to pull config) or similar key-value for distributing configuration.

Not "setup base vm with terraform" and then "ansible to install and configure it". Just build your vm/container image with the software you want already installed, and a etcd or other pull-configuration from a pre-set source. Done.

Now you dont need configuration-management, and dynamically changing infrastructure, since you moved it to the build-step.

I don't think configuration management tools are dead. I use puppet to build my image with packer. It's way easier than build with a bunch of shell scripts. I agree that you shouldn't use it on live servers.

Very curious about what kind of service has these requirements! Servers in 50 different cities suggest you likely care A LOT about latency. Going bare metal likely means you’re dealing with huge amounts of computation. But tolerating 2-5 mins of downtime on every deploy (until now) means availability isn’t a big deal.

I’ve never seen this combo of requirements on any service before, what is it? Or are my assumptions just wrong about what these things suggest about your business?

We detect and monitor commercials airing on terrestrial TV networks. :)

Pretty cool stuff. I'm assuming you have to set up Antenna's where you have your servers colocated and monitoring the transport stream's going through?

Obviously you don't have to spill the company sauce but I'm trying to imagine the benefits of having all of this data.

Cool, so, looks like the product generates daily reports for advertisers about performance of ad spots over a trailing 1 day, lining up info about (a) what ad ran in what markets at what times plus (b) what increase in mobile/web traffic/conversion events happened from those markets coincidental with advertising, if I understand https://www.decidata.tv/advertisers.html right. Looks like a complementary product (maybe running as a second process on the same nodes, or maybe running as part of the logic of the thing that parses the video) also produces quality reports about aired content ( https://www.decidata.tv/operators.html ). I don't really know video very well, probably some very clever stuff you can do here to parallelize the processing.

So hmmmm, a few things to think about for the tooling:

() How quickly can you notice a single-node service outage and react / restore service?

() How much downtime is ok? Is it ever ok to fail to process data for an entire day for one of the ~50 markets (can you just say "this report is incomplete" and reprocess it later)? How much wiggle room do you have (how long does it take 1 instance of the application to do its local processing job on 24 hours of data, and how long does the downline data pipeline have?)

() Are you responsible for the part that watches TV + saves it to disk, and ALSO the part that batch-processes the saved video and produces the ad metadata / quality metadata? Are those different services deployed separately?

() Yikes, is there, like, a person on call in each of 50 places who can unplug the thing that's watching TV and plug it back in, if a wire burns out or if the machine that's watching TV fails? That sounds like it could be an operational nightmare but maybe this is a solved problem / maybe you can buy watch-TV-and-save-it-to-disk as a service?

() How do you make sure that every node is running the same version of the software with the same version of all the configuration?

() How do you release new versions of the software? Do you (a) have a non-production environment which runs side-by-side at every node, processing the same data, and measure that the output is equally correct or better, the performance is the same or better, and there are no new errors or failures -- then promote the non-production code to production automatically after a while? Or do you (b) deploy a "canary" to one of the 50 nodes and watch how it performs for a while, then deploy to the rest if they all behave okay? Or do you (c) just ship the latest code to all 50 nodes on Friday nights when no one's looking, and check some health metrics over the weekend, and roll it back on Monday if a metric looks bad?

() How do you keep track of what software versions you released when? If someone asks "What changed on 'X date'" does your tooling let you tell them pretty quickly and pretty accurately?

() If you discover a bad bug (the parser is corrupting data; the new parsing job has a slow memory leak and fails catastrophically after 21 days of uptime) and you need to make a change in a hurry, do your runbooks let you make that kind of change quickly?

Doesn't seem like you really need k8s or ansible or chef or whatever for this, as much as you need to write down your operational requirements and decide what technology will meet them.

(ouch, I ended up formatting this in a way I didn't quite expect...)

Please tell us you’re planning to monitor extra-terrestrial TV networks. It’ll make this year seem so much cooler tech wise, you know using spacex’s new satellites.

Ok, joking aside, intruiging as there seems to be a few companies providing monitoring services for various things. How did you all find the market for this?

Hah very cool! I was indeed way off on my assumptions :) I guess the 50 different cities is for picking up different regional broadcasts? And bare metal over cloud compute is because you have to connect special hardware?

This is a pretty typical retail backoffice recipe as well (changes only during maintenance windows).

I almost want to suggest Windows Group Policy for something like this...

I do this on ~30 servers with GuixSD, so there is a single file to configure all of them. To update the application I wrote a Guix pkg for it that pulls from our repository (simple to set up with Guix) everything is controlled in emacs by me to change/start/stop all the servers. https://news.ycombinator.com/item?id=17084561 is similar to what I'm doing. Caveat: you have to go on google scholar to read guix and nix papers to understand the system architecture, then their documentation also downtimes aren't an issue for us since these are internal servers, and this is done during non critical time but it's usually just restarting the new app so a few seconds. Other caveats, security fixes we just edit in changes not run guix pull but I also have that automated now after mailing list help. You may have totally different req otherwise look into nix/guix, Guix the pkg manager runs on any distro.

Kubernetes is complicated, it's a whole new world of terminology and abstractions that you need to learn, and it changes quickly.

That said, maintaining a large cluster is complex too. Often, there is much more involved than making sure each image's config files are up to date. Kubernetes shines when you have a number of different docker images, where each needs its own unique network, scaling, cpu, and disk settings, i.e., hardware requirements that can't really be controlled with just config files.

In your case, the fact that all of your images are identical probably cuts against using Kubernetes. However, the fact that these images are running on different networks where each may require their own tweaking leans towards it. If you expect to start adding many different docker images to your cluster, with different hardware requirements, it's probably worth the learning curve to get on to Kubernetes.

Disclaimer: I've never run Kubernetes on bare metal, just hosted environments. Bare metal may make Kubernetes much less attractive.

At a high level, your use-case maps well to a Kubernetes DaemonSet, which is a resource that represents the desire to run a copy of something containerized on every node within your cluster. Kubernetes would give you some things that might be useful:

* A unified control plane (monitoring, statusing)

* An automated rolling-upgrade system

* "self-healing" in the form of automatically restarting your containers if they go awry

However, for your use-case, it is likely more effort than it's worth:

* Settin up and maintaining Kubernetes on bare metal servers is a huge amount of work

* A lot of Kubernetes features are not needed for your use-case

* The Kubernetes networking model essentially requires you to set up private network spanning all of your sites in order to work but provides almost no benefit for your use-case

Technically you could use a DaemonSet for that, but the primary use-case is not that for DaemonSets. It should be used for "daemon" containers, like logging daemons (e.g. fluentd), network plugin daemons like calico, flannel or monitoring like prometheus. If you want to run multiple instances of your application, you should define a replicaset or deployment with setting Pod affinity instead.

That's super fiddly though: you will be leaning on (anti) affinity and counts to retain the 1:1 relationship between hosts and pods. I would argue that daemonsets aren't for any specific workload, they're more about how you want scheduling to happen (1 per node, on some subset of your nodes).

This is where K8S becomes messy because of its rapid development history.

Applications should just be a `Deployment` manifest with the type of distribution being a child setting (X replicas with affinity or 1-per-host). There's no reason to have competing ReplicaSets and DaemonSets just for this detail.

I disagree on this one, DaemonSets are for really a different use-case. There is no replicas field you can set, and strictly 1 Pod will run on 1 node. A Deployment is more flexible, because you can run even more Pods in different cities if necessary. What if there are more/less nodes are in different cities or different specs, which could and should run more Pods? You can't fine-tune DaemonSets for those cases, it really is for daemons.

It's not about use-case, it's that "X number of replicas with affinity" OR "1-per-node" should just be a setting within a Deployment. No need to use a completely separate DaemonSet definition just to run 1-per-node.

Kubernetes is supposed to installed nodes close to each other, typically in the same server room, or datacenter, otherwise the latency would kill the etcd cluster. Ansible is perfectly fine and simple for your use case, why would you bother even thinking about overcomplicating it? Simplicity is king!

You could rather easily manage this flow with ansible. K8s abstracts a lot of the pieces in your ansible plays away, but comes back at you in form of a rather complex stack. Say you want to host instances of this app in a DC/cloud, k8s would be more appropriate, and perhaps then it would make sense to go single node “micro-k8s” style (for single server locations), to give you a common api for managing deployments, network configuration, security etc.

Win with k8s is that once you’re tuned, a lot of “config crap” is abstracted away, basically. But - there's a reason ansible is supported specifically through the “operators” SDK, something we use to manage seamless upgrades of a k8s deployed app:

https://github.com/operator-framework/operator-sdk/blob/mast...

Edit - forgot that if meteics based autoscaling of the app is a requirement my experience is that k8s makes this a breeze. Probably not a req if using single node hosts though...

I haven't used ansible in a few years, but I sorta loathed the transport after a time with even dozens of boxes -- constantly creating ssh connections was slowing things down significantly.

I switched to saltstack some time back and it's much better while having a similar sorta interface. The docs are worse than ansible imo tho. ymmv

You shouldn't be creating ssh connections all the time, but instead using persistent connection to each server that ansible opens once at the beginning, and reuses for all subsequent tasks.

See ControlPersist/ControlPath OpenSSH client configuration.

Never found this incredibly reliable - and if you change networks, sleep, or power off your machine, those connections die. Salt is just better at all of this imho.

I mean, I guess if you use a dedicated host, it's more reliable - but I've always just pushed from the client using Ansible whereas with Salt you have the master running the cloud running somewhere most usually.

Do you often do these things in the middle of running a playbook? I would hope not. :)

It's not meant to have the connections persist forever, just long enough for repeated short ssh connections to have almost no overhead, and therefore be much faster.

Yeah, good point.

And for even more speedup, use Mitogen https://mitogen.networkgenomics.com/ansible_detailed.html

You could also go full docker and experiment with docker swarm. I have little experience with swarm itself, but AWS ECS has deployment strategy. You could say you want 50 container instances on a fleet of 50 machines with deployment strategy of spreading across all machines and you would guarantee that every single machine is populated with a docker container. Maybe try to look for a similar functionality thinking about you bare metal machines as a single fleet of instances. Do you manage your containers configs with environment variables? I think docker also let's you share the hosts environment with the docker container, so you can pre provision the server with the necessary information and once the swarm is deployed, the container figures out what to do with the hosts env. Applying that to the spread across machines deployment strategy and you solve your problem.

I would personally go with ansible and consul if your application is something like a rest api, should be trivial to setup.

Summary: 50 production instances of a single application.

No, you don't need Kubernetes (though fwiw if you used it what you'd have would be 50 nodes with a single daemonset, 1 pod of the app per node, if that helps understand) - but I'd suggest not using Ansible.

Instead, use Packer (or similar) to create a machine image/snapshot/whatever your server provider calls it, and then deploy that same image on all 50.

You might have some small amount of host-specific things left to configure, and by all means use ansible for that if you want, but there's no need for error-prone running of the same large playbook 50 times.

Personally, I find that final setup is little enough to do it with Terraform (which is provisioning the servers anyway).

I've used SaltStack in a similar setup with some success, it's easy to keep all the servers in sync and prevent configuration drift.

Check out BalenaCloud. It's a container orchestration service for edge devices.

Reply on news.ycombinator.com