Hacker News Reader

You aren't "missing" anything, you are just doing things differently. Containers are a way of storing code and configurations in a single deployable entity.

Let's say I want to change a services version from 1.2 to 1.3.

If I did this in Ansible, I might do it by commanding all of my services to pull the code of version 1.3 from a git repo and restarting to make the change.

If I did this with containers, I'd tell them to pull the image version 1.3 from docker registry and after it's pulled to shut down v.1.2 and start v.1.3.

Those things just have some smaller differences that are good in terms of A and worse in terms of B.

Ansible helps you by keeping it closer to the traditional way of just pushing code changes to a machine and that's that, and Docker opens up this world where the infrastructure can scale up or down (elasticity) quicker horizontally.

Kubernetes is just an extrapolation of that elasticity.

Ansible is "good enough" if you don't need elasticity or you have programmed another custom solution to handle that.

Docker/Kube and Ansible both have Git at their hearts, so if your app components have all they need as code, and you don't worry about elasticity, you won't gain a lot from wrapping it up in a Dockerfile (as opposed to just git cloning).

I use Ansible as well and it does not allow me to scale my app containers easily, especially because of dependencies. I think I either have to manage _everything_ by Ansible including firewalls, IRB/SVI on switches, Openstack security zones, users, roles etc. or I am asking for problems and they will definitely come. NB I do not like to manage my switches, routers and firewalls with Ansible.

But my infrastructure is fairly complicated, some bare metal services, many VMs, several docker hosts with dozens of various app containers, private Openstack and some public cloud services (VPNs), Ceph cluster, nagios, ELK etc.

Ansible does not watch and maintain the infrastructure and services state - it is a passive tool (I am not sure if Tower is different). You definitely could configure your monitoring to invoke Ansible script if your app server goes down and reschedule services that ran on that host to a different one and reconfigure other parts of infrastructure as needed (dependencies, VXLANs, security rules, service discovery db update, load balancing update etc.) But you would essentially copy the Kubernetes controller functionality.

Kubernetes can take care of the service discovery, load balancing, deployment, configuration and networking and other parts of your infrastructure and it does it pretty well in my experience. It maintains the declared state and reacts to its changes.

Somebody posted here that it takes just several hours to learn it. YMMV but deploying a resilient cluster in my environment took me much more time. And I have to agree that documentation is pretty weak. E.g. provisioning Cinder volumes and connecting them to the VMs running Kubernetes nodes was a real horror.

Btw Ansible generates many of my k8s YAMLs and deploys them.

> Ansible does not watch and maintain the infrastructure and services state

Yeah, but what I've done is that i've deployed a tiny go binary on each cloud instance that is run every few minutes. Instances take turns checking on each other using a round robin sort of approach (no complicated leader election algorithms etc...)

The script knows how to check the health of the other instances/services and to restart them or alarm if they get stuck.

For fifty hosts, it works fairly well.

I didn't say in the original post, but we are not a product group, so our stuff is 'semi production'. We don't have customers to worry about.

>Somebody posted here that it takes just several hours to learn it.

Sure I can probably learn it quickly, but what I don't want is to now have complicated and mysterious kubernetes problems to solve on a deadline.

I understand linux pretty well, been using it for 20+years, so I'm not intimidated by OS level troubleshooting. Sure without containers, you have to be more careful to keep your dependencies from overlapping, but it hasn't been a problem we couldn't handle till now.

I don't particularly want to trade problems I'm familiar with solving for a whole new set of unfamiliar problems unless there is a clear benefit.

you get the ability to hire people with a keyword ... "kubernetes SRE"

but i think thats pretty much it if you're a smallish team with an already well implemented IaC stack.

Though i'd definitely encourage anyone to try the GCP Kubernetes before trying to self host it...

The former gets you a taste for why its getting such good publicity. the later explains why its still controversial.

as someone in almost the same situation (about 50 instances), but chose to go with docker, I just want to say you made the right choice.

Of course a few years ago, being young and a one man ops team, I wanted to use the cool new thing and so did everyone else. I went with Ansible after being a Chef guy for a while due to ease of getting started. That was the easy part. Love Ansible, but it was slow to get new Docker functionality for a while, and not having the built in functionality of having Chef runs happening periodically without paying for Tower means I have to be more mindful and deliberate with keeping my infra in sync and up to date.

Enter Docker and months and months of headaches just to get something usable on the local workstation level. So much wasted time dealing with breaking changes and figuring out exactly which version to use and very rarely upgrading. Debugging and troubleshooting becomes an unintuitive nightmare. Even after all that, our site still ended up running really slow locally in Docker. This is because with apps (such as CMS) that handle many, many files, I/O slows wayyy down. Ended up discovering Dinghy (shout out to codekitchen) and got it to a decent state (but make sure developers don't accidentally install Docker for Mac as it is and always has been a CPU gobbling mess).

Then on top of that there is the container orchestration (consul-template), monitoring (Prometheus is cool, but takes a bit to understand what is needed to get the metrics you want), logging (fluentd is again, cool, but oh man is parsing logs a PITA to understand), debugging tools etc.

I can't even imagine what adding k8 on top of the many quirks of Docker would mean. I luckily took one look at needing a whole zookeeper cluster just to get started, and immediately gave up on that.

What do you use for log aggregation and monitoring/metrics? And how do you deploy new versions with zero downtime? And how do you manage kernel/OS upgrades?

> log aggregation

Graylog, with applications using various GELF libraries to send logs to it.

> metrics

graphite/grafana

> And how do you deploy new versions with zero downtime?

I don't deploy with zero downtime, but within say ~5min. This is acceptable to us. We use jenkins with a lot of tests to ensure components are in good shape, then we mostly manually deploy them, but with a script. Sometimes we deploy directly from jenkins.

> Kernel os upgrades

All our stuff is internally hosted, so we handle those infrequently. This stuff isn't exposed the the open internet, so we upgrade when we get around to it, or when we encounter a bug.

Thanks for following up! I'm not surprised because I noticed on HN and in other online discussions that almost all the people that advocate against Kubernetes and PaaS (Heroku, App Engine, Clever Cloud, Scalingo, etc.) are willing to tolerate downtime during deploys and during OS upgrades.

> are willing to tolerate downtime during deploys and during OS upgrades.

Yeah, that's not a big deal for us. We don't have customers, and this environment is for internal use at the company.

You could say it's a 'semi production' environment.

I like the "semi-production" concept :-)

Deploying with zero down time with Ansible is kinda the same as it's always been:

  - Deploy to one host  
  - Wait for health checks to pass  
  - Deploy to second host.

Load balancer takes care of the rest.

Because it's cool, new and good for your blog/CV. I don't see the need too, if stuff is in cloud.