Comment by brendandburns | Hacker News Reader

Comment by brendandburns | Hacker News Reader

Parent Full thread

Disclaimer: lead engineer on kubernetes here...

It really depends on how you do deployment. Containers provide deployment (and more important rollback) that is better than other deployment tools like Puppet/Chef/... because they are atomic (they either work, or they fail, they don't get stuck in the middle) and they package up all of their dependencies within them, so that they don't have the "well it worked on my machine" problems.

Systems like Mesos and Kuberenetes, decouple applications from the individual machines (and the operating system on those machines), and are online systems with self-healing properties (so that they will fix themselves rather than waking you up in the middle of the night)

k8s and mesos turn containers into an API that spans an entire fleet of machines, and enables you to dynamically use (and re-use) the fleet of machines for multiple different applications. No more dedicated boxes for mysql, mongo, etc. This in turn enables you to have an easier ops experience, because every single machine in your fleet is homogenous (same OS, same patches, etc) OS management is abstracted away from Application management, so that they don't interfere with each other. Since things in the API are expressed in terms of applications, it's easy to add health checks and automatic restart to the system, and provide self-healing properties as well. Both kubernetes and Mesos also make replication a first-order primitive so that it is easy to scale in response to load.

Ansible is sort of orthogonal to systems like Kubernetes and Mesos. Kubernetes and Mesos are designed to be online, self-repairing systems. Ansible is a way to easily execute commands on a bunch of machines. I can see collaborative use cases, where you generally use Kubernetes for deployments, but use Ansible for querying some data while debugging, or somesuch.

Anyway, sorry for the extended response. There actually is way more that I could say about the topic ;)

Thanks for the detailed response. The point about atomic deploys is a good one. However, if we all agree that immutable deployments are a good thing, I've been wondering how is that fundamentally different from launching new instances via AWS/Rackspace/Google API? Is there really a fundamental difference between shipping a Docker container to a set of servers, or just relaunching/rebuilding servers?

You also mention the "no dedicated Mysql, Mongo boxes" etc. I am 100% for that. However, how can you really make that work with databases/systems like MySQL which were fundamentally designed to work on one machine, and scaling them across machines is usually very painful, or at least, let's say, not very "idiomatic" (if I can use that word here lol). I can see the auto scaling part working with distributed databases like Riak/Cassandra, but even there the solution is not clear-cut and "out of the box". It still feels like some manual orchestration work is needed - correct me if I'm wrong.

I can totally see the "online, self-repairing point" but only for application servers that were designed from the scratch to be easily scalable by just adding servers. Which is the case for most scripting languages PHP/Ruby/Python et al and for well designed JVM/CLR/native/Go applications as well. However, I would argue scaling the app boxes/containers is the EASY part. Again, you can always scale up by "just" copying machines (I know, it's always harder that that). The hard part comes with managing the database servers or your cluster of messaging queues, or some other stateful thing that has to persist data SAFELY. Does Kubernetes/Docker REALLY make my life easier with those kinds of things? Is the answer to use DynamoDB/BigTable/RDS/managed queues and forget about that hassle of managing a database or a queueing cluster? Looking for answers :) . Thanks!

> I've been wondering how is that fundamentally different from launching new instances via AWS/Rackspace/Google API? Is there really a fundamental difference between shipping a Docker container to a set of servers, or just relaunching/rebuilding servers?

There's two perspectives to this:

* From a user's perspective, ignoring performance implications, containers should be the same as VMs.

* From a hardware perspective however, containers are much more lightweight in terms of CPU, Ram and Disk space, since they all share at least the kernel.

Why should you care? Well if something is less resource consuming, it means that (a) you could run it for less than what you pay for VMs or (b) it can be sold to you for less. There's some additional benefits like fast "boots", strict decoupling of persistant and non-persistant storage (which I find an advantage, restarting a container cleans up whatever you don't care for), but at the end I think it comes down to money.

To add to this. Sure you can spin up ec2 instances and you don't have to worry about atomicity. However, what happens if you want to run more than one service per VM? That's where containers allow to take full advantage of the system.

> No more dedicated boxes for mysql, mongo, etc. This in turn enables you to have an easier ops experience, because every single machine in your fleet is homogenous (same OS, same patches, etc)

Is that a good thing? Granted, the AWS account I used to managed was 8-10 machines at most, which is probably well below the point where Kuberenetes makes sense, but I remember that it was useful to configure different kind of machines for MySQL than Apache (disk needs were vastly different, for instance) and that the MySQL instance for the web server had different needs than the MySQL instance for offline data processing.

Especially if you're working with Postgres, which doesn't do multi-master...

oh since postgresql 9.3 upwards + repmgr(d) it's now way easier to scale and running postgresql clusters, even in docker.

You have to design you applications bearing in mind where do they run (AWS, baremetal...) and now, you have to design your applications bearing in mind the "Datacenter OS", which is fine, but adapting solutions to new ways of doing takes time.

To me, unless you have a big park of machines, these systems are a total overkill... but I guess that time will say.

To me, unless you have a big park of machines, these systems are a total overkill

I think that's an important point, and one which container vendors are not going to labour, as they want as many people as possible on their platform, even before they really need it. A lot of people are trying to use docker or coreos who really don't need to, and as they're not the focus of containerisation efforts, they'll suffer as they find out they're not really tailored to what they want to do, which is just get their small web service running reliably with the minimum of fuss, and be sure they can rebuild it or move it between providers easily.

If you have 1-10 machines which don't change much, use Ansible or similar to get predictable (re)deployments and don't worry about using containers.

If you have > say 10 machines, this sort of stuff becomes more useful, because you are herding cattle, and need the infrastructure necessary to keep that herd going, even if a few die off from time to time - then you can scale to hundreds easily as your business grows, you can manage lots of workers reliably on one VM in containers etc, etc.

For probably 90% of websites out there, with a sane setup that's never even going to become an issue and they could run easily on just a few servers.

that's not true. what happens if one of your server dies?

you either heal it back or you shoot it. shooting is way faster and docker could help you by that.

Also 1-10 servers could be much, it really depends how much stuff you need.

Also docker adds "some" security. Docker isn't the perfect match, but on our site we run a match between ansible and docker (without coreos) and are very happy.

In our aws cloud we have another system which only uses fleet and coreos, the cluster upgrades itself which is a big plus, but doesn't work that good in our internal infrastructure with proxies, firewalls, etc..

We run 5 machines in aws.

And have like 6 vsphere esxi nodes.

If one of your server dies you deploy another in 5 minutes with ansible, with a small number of servers that rarely happens though.

Reply on news.ycombinator.com