Hacker News Reader

Comment by raspasov | Hacker News Reader

Maybe it's just me, but I fail to see how all those project make the developer/devops person's life less complex. I don't have the experience to have managed a cluster of 1000s of machines but I HAVE managed a cluster of ~50 nodes with MySQL, ElasticSearch, RabbitMQ, ZeroMQ, PHP, Clojure and the whole gang at various points in time. I am yet to see a single project (Mesos, Kubernetes, Docker, whatever) that would DRAMATICALLY make my life so much better. Not trying to be overly negative here, just looking for answers and better solutions.

EDIT I also continuously wonder how do things like Ansible fit into the picture. Are they competitors? Are they supplemental to the likes of Mesos/Kubernetes? Are they orthogonal?

View on HN

Disclaimer: lead engineer on kubernetes here...

It really depends on how you do deployment. Containers provide deployment (and more important rollback) that is better than other deployment tools like Puppet/Chef/... because they are atomic (they either work, or they fail, they don't get stuck in the middle) and they package up all of their dependencies within them, so that they don't have the "well it worked on my machine" problems.

Systems like Mesos and Kuberenetes, decouple applications from the individual machines (and the operating system on those machines), and are online systems with self-healing properties (so that they will fix themselves rather than waking you up in the middle of the night)

k8s and mesos turn containers into an API that spans an entire fleet of machines, and enables you to dynamically use (and re-use) the fleet of machines for multiple different applications. No more dedicated boxes for mysql, mongo, etc. This in turn enables you to have an easier ops experience, because every single machine in your fleet is homogenous (same OS, same patches, etc) OS management is abstracted away from Application management, so that they don't interfere with each other. Since things in the API are expressed in terms of applications, it's easy to add health checks and automatic restart to the system, and provide self-healing properties as well. Both kubernetes and Mesos also make replication a first-order primitive so that it is easy to scale in response to load.

Ansible is sort of orthogonal to systems like Kubernetes and Mesos. Kubernetes and Mesos are designed to be online, self-repairing systems. Ansible is a way to easily execute commands on a bunch of machines. I can see collaborative use cases, where you generally use Kubernetes for deployments, but use Ansible for querying some data while debugging, or somesuch.

Anyway, sorry for the extended response. There actually is way more that I could say about the topic ;)

Thanks for the detailed response. The point about atomic deploys is a good one. However, if we all agree that immutable deployments are a good thing, I've been wondering how is that fundamentally different from launching new instances via AWS/Rackspace/Google API? Is there really a fundamental difference between shipping a Docker container to a set of servers, or just relaunching/rebuilding servers?

You also mention the "no dedicated Mysql, Mongo boxes" etc. I am 100% for that. However, how can you really make that work with databases/systems like MySQL which were fundamentally designed to work on one machine, and scaling them across machines is usually very painful, or at least, let's say, not very "idiomatic" (if I can use that word here lol). I can see the auto scaling part working with distributed databases like Riak/Cassandra, but even there the solution is not clear-cut and "out of the box". It still feels like some manual orchestration work is needed - correct me if I'm wrong.

I can totally see the "online, self-repairing point" but only for application servers that were designed from the scratch to be easily scalable by just adding servers. Which is the case for most scripting languages PHP/Ruby/Python et al and for well designed JVM/CLR/native/Go applications as well. However, I would argue scaling the app boxes/containers is the EASY part. Again, you can always scale up by "just" copying machines (I know, it's always harder that that). The hard part comes with managing the database servers or your cluster of messaging queues, or some other stateful thing that has to persist data SAFELY. Does Kubernetes/Docker REALLY make my life easier with those kinds of things? Is the answer to use DynamoDB/BigTable/RDS/managed queues and forget about that hassle of managing a database or a queueing cluster? Looking for answers :) . Thanks!

> I've been wondering how is that fundamentally different from launching new instances via AWS/Rackspace/Google API? Is there really a fundamental difference between shipping a Docker container to a set of servers, or just relaunching/rebuilding servers?

There's two perspectives to this:

* From a user's perspective, ignoring performance implications, containers should be the same as VMs.

* From a hardware perspective however, containers are much more lightweight in terms of CPU, Ram and Disk space, since they all share at least the kernel.

Why should you care? Well if something is less resource consuming, it means that (a) you could run it for less than what you pay for VMs or (b) it can be sold to you for less. There's some additional benefits like fast "boots", strict decoupling of persistant and non-persistant storage (which I find an advantage, restarting a container cleans up whatever you don't care for), but at the end I think it comes down to money.

To add to this. Sure you can spin up ec2 instances and you don't have to worry about atomicity. However, what happens if you want to run more than one service per VM? That's where containers allow to take full advantage of the system.

> No more dedicated boxes for mysql, mongo, etc. This in turn enables you to have an easier ops experience, because every single machine in your fleet is homogenous (same OS, same patches, etc)

Is that a good thing? Granted, the AWS account I used to managed was 8-10 machines at most, which is probably well below the point where Kuberenetes makes sense, but I remember that it was useful to configure different kind of machines for MySQL than Apache (disk needs were vastly different, for instance) and that the MySQL instance for the web server had different needs than the MySQL instance for offline data processing.

Especially if you're working with Postgres, which doesn't do multi-master...

oh since postgresql 9.3 upwards + repmgr(d) it's now way easier to scale and running postgresql clusters, even in docker.

You have to design you applications bearing in mind where do they run (AWS, baremetal...) and now, you have to design your applications bearing in mind the "Datacenter OS", which is fine, but adapting solutions to new ways of doing takes time.

To me, unless you have a big park of machines, these systems are a total overkill... but I guess that time will say.

To me, unless you have a big park of machines, these systems are a total overkill

I think that's an important point, and one which container vendors are not going to labour, as they want as many people as possible on their platform, even before they really need it. A lot of people are trying to use docker or coreos who really don't need to, and as they're not the focus of containerisation efforts, they'll suffer as they find out they're not really tailored to what they want to do, which is just get their small web service running reliably with the minimum of fuss, and be sure they can rebuild it or move it between providers easily.

If you have 1-10 machines which don't change much, use Ansible or similar to get predictable (re)deployments and don't worry about using containers.

If you have > say 10 machines, this sort of stuff becomes more useful, because you are herding cattle, and need the infrastructure necessary to keep that herd going, even if a few die off from time to time - then you can scale to hundreds easily as your business grows, you can manage lots of workers reliably on one VM in containers etc, etc.

For probably 90% of websites out there, with a sane setup that's never even going to become an issue and they could run easily on just a few servers.

that's not true. what happens if one of your server dies?

you either heal it back or you shoot it. shooting is way faster and docker could help you by that.

Also 1-10 servers could be much, it really depends how much stuff you need.

Also docker adds "some" security. Docker isn't the perfect match, but on our site we run a match between ansible and docker (without coreos) and are very happy.

In our aws cloud we have another system which only uses fleet and coreos, the cluster upgrades itself which is a big plus, but doesn't work that good in our internal infrastructure with proxies, firewalls, etc..

We run 5 machines in aws.

And have like 6 vsphere esxi nodes.

If one of your server dies you deploy another in 5 minutes with ansible, with a small number of servers that rarely happens though.

A few ways in which this stack can simplify your life:

- Abstract away elements of failover, disaster recovery, clustering, load balancing

- Abstract your application services cleanly from underlying servers

- Give you more consistency across and within development, test and production environments

- Allow for immutable deployments

- Allow for canary releasing, A/B releasing, rollback etc

- Better isolation of processes and services

- Move away from general purpose operating systems to lighter weight single purpose OS such as CoreOS

Configuration management still has a role to play in this stack, but somewhat smaller than the old world

It doesn't. As an average developer it's my public/private cloud providers job to take my images/dockers and schedule them correctly on whatever underlying fleet of hardware they run.

I can only assume Tectonic and related tech are aimed at those developers who run hybrid or private clouds for the rest of their company.

This is also very true. The purpose of systems like Kubernetes/Mesos is to be set up once by a company, and for most developers to just interact with the CLI tool to run containers. Tools like Google Container Engine, make this into a software as a service product, where the cloud provider provides the API. In that world, as a developer, you never really think about the OS, just your app. But unlike in traditional PaaS, there is no framework that restricts what you can code (language, libraries, etc).

> But unlike in traditional PaaS, there is no framework that restricts what you can code (language, libraries, etc).

I'm not sure how traditional PaaSes are limited in that sense. Heroku accepts a wide variety of software through the buildpacks mechanism. Cloud Foundry supports both buildpacks and docker images, with clean extension points to add further mechanisms -- for example, .NET-on-Windows deployments, which is under development.

Disclaimer: I've worked on Cloud Foundry and I'm employed by Pivotal.

The main difference between an PaaS like Heroku and a general purpose cluster manager service like GKE is that the former can simplify my making certain assumptions about your workload.

For example if a service knows your container is serving a web application then it can sensibly provision load balancers, DNS, HTTPS, auto-scaling, static content caching, automatic QPS monitoring etc. to support your app with little explicit configuration. And with a commercial service you get an SLA for those things.

You can get much of this with a general purpose cluster as well, but of course you need to configure it yourself, and more importantly - debug it when it goes awry.

Disclaimer: I work for Google

Sadly in the Cedar stack, Heroku did away with Varnish or static asset caching from their "Dynos". Instead all assets bundled with your app are served using Ruby and dyno time. While a simple and clean architecture, this either raises the cost or forces you to place static assets on other servers (or use CDNs). And you'll still need to debug someone else's stack just as much -- when you get a service-specific error for exceeding memory limits or having requests take too long because of instance-level queues, etc.

At this point, I never pick PaaS because it's less work, I pick it simply because I've used it before and it's easy to get started with. Production apps are never hands-off if you're the one developing them ;-)

There are already a couple pretty good responses, but I would add that all of the technologies you mentioned (along with others in this space) are still evolving and maturing.

Docker popularized containers, and now we're seeing the supporting technologies that will help fulfill some of the promises of container-based deployment. Big companies have had this technology internally for years, and now it's starting to get productized.

I'm referring to things like schedulers, networking, monitoring, etcd/consul, and more. I personally don't think all of the pieces are there yet to make it as seamless and easy as it could be, but it's coming.

Reply on news.ycombinator.com