Instead of containers, give me strong config and deploy primitives | Hacker News Reader

Instead of containers, give me strong config and deploy primitives

abe-winter.github.io

334 pointsawinter-py179 comments

So... systemd + Ansible?

I really disliked SystemD before I got my hands dirty with it. Assuming you're developing with any modern language, the language itself probably wraps OS differences anyways (Node / Golang / Rust / Ruby / Python / anything-libUV-based to name a few), not like you can convince developers to change their habits anyways.

People act like Docker makes builds reproducible by magic, then go on to not pin any Golang deps or curl internet resources or not generate lockfiles for NPM, ad infinitum.

There is and never will be a shield operations people can put around their developers. You just need good developers. If you want SoA scaling or automatic cloud configuration, why not just use the most established tools available? Linux's init system and kernel _already do_ resource management, even outside of LXC. People act like a "cloud scheduler" is the only thing that lets multiple processes work together.

This whole idea of isolation is also really inane. Each app will have different disk, io, network, cpu requirements. I also dream of the day Kubernetes can do this "sorting" better than an experienced operations team can, but that day is _far off_ (and I will die before I put redis/mysql/postgres/cassandra/any-other-mature-datastore behind docker and iptables)

You've done nothing to justify your "systemd + sensible" in this comment. You've just ranted against other technologies. I don't see the point.

I suppose systemd could be swapped for "any modern linux init system" and Ansible could be swapped for "any modern cloud configuration tool". There are probably 3 or 4 items that come to mind for both categories.

The idea is that Linux already solves that vast majority of these problems, as do cloud providers. I then ranted on a couple of the "problems" that containers and container orchestration "solve".

You're right tho - a bit of a ramble. Been a long day / on-call week :P I will think about how I could have been more eloquent for next time.

> [...] Ansible could be swapped for "any modern cloud configuration tool".

Deployment tool, not configuration tool. Ansible is awful for keeping environment configured, it was from ground up designed as somebody's deployment script.

Amen.

Ansible is great for fire and forget machines you don't need or care about. You need real config management with an audit trail when you go enterprise.

What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well?

Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management.

I could easily imagine that the 'modules' in Ansible aren't as robust as those in, e.g. Chef, at least for some platforms. But my experience so far having used both Chef and Ansible with Windows VMs (on VMWare) is that Ansible is much better.

> What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well?

Hosts that were temporarily down when Ansible tried to ship configuration. CFEngine or Puppet just apply the changes some time later without operator's intervention.

> Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers.

So does Ant or Gradle. Something is different about the operation, though, because nobody uses Gradle to manage server configuration.

> I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management.

The architecture and the protocol.

With Ansible you get synchronous sequence of operations that breaks on first problem (e.g. host is down for maintenance). This is OK for deployment, but not for ordering a configuration change, for which you don't care when it will be applied, as long as it's soon enough. Note that deployment and managing configuration are similar, but different things, and it's not uncommon to need to do both (although not at the same time).

Then the protocol. Ansible uses brain-dead idea of using the debugging/interactive channel (SSH) to configure things, including possibly SSH itself. One small mistake can cut off your all remote access at once. It's negligible risk if you only configure SSH daemon when deploying the server, but it increases quickly in the log-term.

Not to mention that managing SSH access properly (e.g. not ignoring warnings about hosts' public keys, but not just that) is much more tedious than keeping a CA, so SSH is a bad choice for unattended operations.

Yeah. Would be really interesting to see a container technology that people were eager to use for DB management. Persistent storage is the thing that hasn't clicked yet.

My guess is this technology will be some kind of cluster management. If there were a 'platform' for horizontally scaling data that managed sharding, rebalancing & backups plus gave me user-space hooks to override policies, databases might slowly & cautiously build on top of it.

Most DBs have their own of these (pgpool, mongos, redis sentinel, all of cassandra & hadoop) but would be cool to centralize the common reqs and collaborate.

> would be cool to centralize the common reqs and collaborate

Oh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!

The problem is that the constraints of wanting your data and CPU on the same node, needing data synced to more than one node, and avoiding being too much of a performance bottleneck, all combine to make it very hard to do better than something which has privileged knowledge of how the database works under the hood. Performant consistency is hard.

smartOS gives you persistent storage btw. And its lovely ZFS storage..

saying smartos gives you persistent storage is like saying docker gives you persistent storage because you can use

  docker run .... -v /storage:/storage myapp

That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?

technically it still persists, just in the form of hawking radiation instead of bits & bytes. Some people think this is what makes amazon glacier so cheap.

I'm not too knowledgeable on the subject but I know there are a lot of groups working on persistent storage for Docker. I think most of the time this means a docker volume that lives on some network attached storage (this could be redundant across the compute cluster or on dedicated storage servers), which is then linked to a container via the network. If you remove that container or destroy the host, the volume persists and you can recreate the container on a new host and re-attach it to the associated volume (automatically).

As far as databases and horizontal scaling, this could probably be done currently within each Docker container at the database layer as you mentioned (CockroachDB also comes to mind). So let's say you add a new docker container with its own persistent+redundant storage volumes, and that container is running a cockroachDB node. You can add more containers as cockroachDB nodes, each with their own persistent+redundant storage, and you get all the DB sharding etc across nodes from cockroachDB while the redundancy+persistence are handled by Docker (plus storage solution) as far as restarting, migrating, reconnecting, etc. This probably has all sorts of problems I'm blind to but that's my rudimentary understanding.

Handling horizontal scaling, sharding etc at the storage layer directly through Docker sounds much much harder, and I don't even know what I've been talking about so far so I'll leave it at that

Here's a list of some people doing persistent Docker storage :

  Flocker
  Portworx https://portworx.com/
  StorageOS https://storageos.com/
  Rancher Longhorn https://github.com/rancher/longhorn
  Quobyte https://www.quobyte.com/containers
  Nexenta https://nexenta.com/solutions/containers
  Infinit https://infinit.sh/docker
  EMC REX-ray https://github.com/codedellemc/rexray
  Ceph
  Gluster
  Mesos https://mesosphere.github.io/marathon/docs/external-volumes.html
  Kubernetes https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  vSAN
  Nutanix http://next.nutanix.com/t5/Nutanix-Connect-Blog/Containers-Enter-the-Acropolis-Data-Fabric/ba-p/11122
  Netapp (Docker volume plugins for SolidFire+ONTAP+E-series)
  AWS EBS/EFS

I'm biased but checkout Portworx. It is designed to be that and if you don't think it is, I'd be curious why is missing. https://portworx.com

https://github.com/CrunchyData/postgres-operator is the most promising thing I've seen lately.

As an Ops guy, I preach Ansible + systemd all day everyday, but so many of our Devs (and Ops) have drunk the containerization Kool-aid.

Unfortunately, the same culture which fucked modern web design (a new JS framework every 6 months) is starting to creep into Ops; some REALLY smart people make a tool which promises to be X but with X's main problem solved. They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).

The JavaScript situation is grossly exaggerated. It's doing very well. It's just that the people who aren't productive are also the ones shouting the loudest on HN. Meanwhile, a lot of people are being very quiet getting tons of stuff done really efficiently. (Source: Me, very happy React developer who is so glad we're finally out of the terrible "wilderness years" of server templates and jQuery.)

> terrible "wilderness years" of server templates

wat.

That's fine, it's:

> terrible "wilderness years" of server templates and jQuery

where things go off the rails.

I reckon

> terrible "wilderness years" of ... jQuery

is the part where things go off the rails (and literally off the Rails).

Didn't Rails originally ship with Prototype? I liked Prototype.

I will rather take any server templates and jQuery over flavour of the month JS framework, and will keep doing that until my boss or customer tells me otherwise.

No need for JS for what can be done in plain HTML/CSS.

Offloading view changes to clients can both give a better experience (changes are reflected ~instantly vs a refresh and a round-trip to the server) and can scale much better (e.g. Server only used to process CRUD operations, CDN or nginx / Apache used to distribute static assets like the JS app, index html, CSS + images etc).

It can also potentially help with things like caching, but more traditional server-side templated designs can also use things like Varnish with the ESI feature, utilizing caching mechanisms in the framework / app (e.g. Memcache or whatever), etc.

I used to think SPAs were dumb, but now I love Rails 5 API-only mode + EmberJS + EmberData. It also makes server-to-server a breeze, because you already have your API. Mix in JSONAPI, swagger, and oauth and it's a pretty happy environment to build software in.

> changes are reflected ~instantly vs a refresh and a round-trip to the server

You can achieve that with a little sprinkle from jQuery.

For UI/UX rich experiences, taking advantage of OS animation frameworks and widgets, I would anyway advise native applications.

> and can scale much better

Offloading everything like that to client is antisocial behaviour. Seriously. This goes beyond "bad engineering"; for any popular website, this literally means saving a bit on your servers while causing lots of electricity being wasted by your millions of visitors. This is the computing equivalent of selling extremely fuel-inefficient cars to regular people.

It can give benefits but all too often I'm left looking at a spinner wondering what the hell is going on before eventually being presented with a simple page that would have been trivial to render server side.

c2.com being a tragic example of this.

At first I said to myself, "C2, wtf is this guy talking about I could load that on a calculator".

I was wrong. Load time was an abysmal 14 second(!) wait. I normally don't make jokes on HN but today I'll be pouring out a sip of dark roast for my metaphorical 'dead homie'.

Changes on the client still usually roundtrip to the server to get the data (via a Rest call these days, then parse the JSON response, then render and apply the diffs to the DOM).

And I don't buy the argument that javascript-based client apps scale better than servers rendered partials where there's been over a decade worth of solutions for scaling out the server layer. Also, you'd be applying a lot of the same techniques for scaling a pure API as something spitting out partial web views.

I've been exploring Docker lately, but I've just launched a small project using Ansible and systemd, much much simpler to setup and deploy updates.

> They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).

That's a nice summary. The 2nd level ignorance of those people is astounding.

Oh, sure, write the 618th package manager, I'm sure you'll be the one who finally gets it right.

I suspect that systemd will end up in the same state...

Funny, i consider systemd very much a part of the containerization cool-aid...

Ansible + systemd + containers is a match made in heaven.

https://developers.redhat.com/blog/2017/05/04/breaking-up-th...

The idea with Kubernetes isn't necessarily that it can do resource allocation more carefully and more intelligently than a human ops person, but it can do so at a level of automation and speed that a human can't.

Pods die, Kubernetes spawns new ones. Server goes full, Kubernetes spawns new node. Server runs out of disk space, Kubernetes evicts nodes until everything is good again. Server goes down, Kubernetes shuffles all the pods over to a different node. And so on. It does this at a scale of hundreds or thousands of machines. The utility it brings also applies to small clusters. You don't remove ops, you just flatten it into Kubernetes' operational model.

Equally importantly, Kubernetes also brings a declarative deployment model. You have the same with Systemd, but at the node level. Kubernetes is a bit like Systemd at the cluster level; its "units" are pods declared in YAML/JSON files, but unlike Systemd they run on any node the cluster, and are subject to a resource allocation model that includes things like persistent storage (e.g. AWS EBS volumes). You don't get anything close to that with Systemd, which is much more analogous to Docker.

I haven't heard anyone ever claim that Docker brings reproducible builds -- why would anyone assume that? What Docker give you is reproducible deploys; build once, deploy many times from the exact same image. Maybe that's what you're thinking of?

I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production.

But, for the sake of the argument:

> Pods die, Kubernetes spawns new ones.

AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes...

> Server goes full, Kubernetes spawns new node.

I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go).

> Server runs out of disk space, Kubernetes evicts nodes until everything is good again

Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone.

> Server goes down, Kubernetes shuffles all the pods over to a different node.

This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way.

As for the reproducible builds, it was in response to the article which says:

> The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere).

You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"...

I'm not an AWS employee, just figured I'd use a common cloud as an example.

Your responses assume that I'm running in the cloud and AWS specifically. What if I need to run on-premise?

Kube is actually fantastic for on-prem/dedicated/colo. I still wind up needing to manage my _non_ Kube stuff - and fall back to Ansible + systemd for the data layer for example.

Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point.

After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again.

I should probably also admit somewhere in this thread I actually use and prefer Chef :)

tar does reproduce a deploy but onto c clean slate machine. docker is just a clean space where you can take an issue from there and easier reproduce elsewhere because there is no state left over from installations of previous packages and/or other software.

What makes containers in general, and Kubernetes specifically, compelling to me are the following:

- the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be

- since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application)

- if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes

There are other things as well, but I wanted to stay roughly within the parameters of your examples.

If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product.

If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.

What if you're in the ground-zero scenario: there is an application but practically no infrastructure?

I'm in that situation now (with a SaaS product), and my approach so far has been:

* set up tests

* set up Docker images

* set up docker-compose for easy local deployments

* do lots of work to make the app behave inside Docker

* take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions

* bootstrap KOPS/K8s with CloudFormation

Future work:

* set up Jenkins CI on top of K8s

* K8s deployments of the app for CI

* staging K8s deployments

* K8s in production

This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance. I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough".

Has anyone here gone through a similar scenario, bootstrapping ops infrastructure in a small business? Would you have any remarks regarding architecture, or the politics required to make the importance of solid ops and the improvement of software reliability and security clear?

Generally it's fun resume builder but not actually time well spent on your business. One super server can go far enough and hopefully you can design it to be able to scale as needed later.

This is a risky assertion to make, in some cases for us.

All engineering advice comes with the implicit caveat "subject to your local concerns".

Yes, of course you may be working in a startup that requires 25GPUs to serve even one customer. No sarcasm. I can imagine some startups that might meet that requirement.

But there are an awful lot of startups that start massively overengineering their footprint early when all they need is a web server with a cold or hot spare (use a load balancer with automatic failover if you've got one in your cloud or something, if not I wouldn't stress, and automatic failover on prototype-level application code can cause its own issues) and a database server with a good backup story and some mechanism for quickly bringing up a new one if necessary. (This generally leads you to some sort of clustering thing or a hot replicated spare because it doesn't take long before your database requires hours to rebuild from a backup or something.)

You're often better served just giving occasional thought to how you might split things up in the future and using that to at least hint your design than actually splitting things up immediately.

> you can design it to be able to scale as needed later

What's different about what you're referring to above and what the commenter mentioned? It seems to me like they were designing the app to be able to scale.

Except they are also doing the dev-ops yucky work to do the scaling. It is a lot of extra work with no value until later.

Say they are making a crud app. Keep no state in session. Deploy as easily as possible. Boom, done. Later you can add a load balancer to scale to 100 servers. Right now you can run it on 1. It is 'designed to scale'. But actually running your single app in a 1 node k8s cluster? That is days of extra work for nothing right now.

It's tough to give a definitive answer because every company is different, but I work at a very small strange shop, 5 devs + 1 manager, maintaining 10-15 custom websites (5 of which are on a single VM) and I have been deploying our new apps in Docker (no K8s). I use each container as a "miniature VM" which runs the entire web app (except the database), blasphemous I know. Compared to putting multiple apps on one box, the Docker method adds some minor complexity, but keeps apps isolated. That was my biggest requirement, to prevent devs from cross-pollinating applications, which happened constantly when everything was on a single server. It was much simpler than setting up Puppet on a bunch of legacy machines. I also considered putting each new app on its own VM, but went with Docker because a lot of our apps hardly get any traffic and would have wasted quite a bit of resources to spin up a VM for each (all our servers are in house).

The pros to Docker so far: Dependencies: Dockerfile gives a list of explicit system dependencies for each app. This can be done in other ways with package files or config management but this was not being done before and this is an easy catch all to force it for any different type of environment. Logical Grouping: App environment (Dockerfile + docker-compose.yml) lives alongside codebase in a single git repo Deployment: Deploy to any box with `git clone myapp && docker-compose up` for testing/dev instances or migrations Development: We mount the codebase from a host directory into each container, with git hooks to update the codebase, which works well for us (we have no CI) Plus it's fun!

Cons: Operational Complexity: Devs/Ops teams probably won't want to learn a new tool. I setup a Rancher instance to provide a GUI which makes things a bit easier to swallow. It has things like a drop in shell, log viewer, performance metrics, etc. Network complexity: we never needed reverse proxies before, now we do. Clustering/Orchestration: We don't cluster our containers, but the more we add the more I think we might want to, which would add a whole new layer of complexity to the mix and seems unnecessary for such a small shop. Security?: lots of unknowns, lack of persistence can be bad for forensics, etc. Newness: Documentation isn't great, versions change fast, online resources may be outdated.

Like you, I'm sometimes unsure if this is the right choice. Maybe a monolithic server or traditional VMs + Puppet would be easier, simpler, better? In the end, I think Docker just fit with the way I conceptualized my problem so I went for it. You may never get that "definitely good enough" feeling, but if it fits your workflow and keeps your pipeline organized and manageable, then I say go for it.

That's interestig to me, the Rancher bit: I went the route of writing down all my routine docker-compose invocations in a Makefile and I gave that to the devs with builtin documentation (list of targets + examples of workflows), but I see how Rancher could standardize that.

Very interesting ! I am a solo guy but I sort of followed the same way you did. And when I had to go with the Kubernetes road because managing multiple Docker over multiple boxes became too complicated, I just went back to one website = one VM... Giving me time to learn all the k8s stuff, which will be probably be useful soon but just not right now.

Docker-compose is horribly broken, it manages deps in a much too coarse manner, effect lay becoming a broken package manager on top of whatever the container has.

I agree the code quality and "feature parity"/consistency with Docker is absolutely crap. I'm now hoping K8s will provide a better experience while also being more amenable to quick and scalable AWS deployments.

> What if you're in the ground-zero scenario: there is an application but practically no infrastructure? > I'm in that situation now (with a SaaS product), and my approach so far has been: > * set up tests > * set up Docker images > * set up docker-compose for easy local deployments

In my personal opinion, this is all good work to do. One minor nit: You probably want to get out of the habit of thinking about `docker-compose up` as a deployment, and just think of it as a much cleaner, much more "prod-like" local development environment.

> * do lots of work to make the app behave inside Docker

I'm curious about this. What was the overall nature of the work? Things you would have had to figure out in a distributed platform anyway, or container-specific issues?

> * take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions

This is amazing, I applaud you for your foresight. I wish we had known about this pattern much earlier.

> * bootstrap KOPS/K8s with CloudFormation

This is where there's going to be another giant time sink, unless you're already pretty familiar with Kubernetes' resources—Deployments, Services, PersistentVolumeClaims, Secrets, etc—how they work together, what they enable, what their limitations are, and so on. Not to mention picking a network overlay plugin, deciding on a service discovery model, and understanding how Kubernetes works. BTW I am currently working on this very knowledge, and at least for me it's been pretty heady stuff at times.

>Future work: >* set up Jenkins CI on top of K8s >* K8s deployments of the app for CI >* staging K8s deployments >* K8s in production

Everything here seems relatively sane, in broad terms. I mainly caution against moving too quickly and also against spending engineering efforts on this if you are also an IC for the applications your teams are writing.

>This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance.

Honestly, you seem to have the right mindset for this and so far you haven't mentioned anything that I see as a huge red flag.

>I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough".

Welcome to ops! There will never not be an operational issue that either needs to be fixed or is silently waiting to strike at 3AM on Monday morning.

Tar definitely doesn't solve the issue of reproducible deploys, a tar deploy has a high chance of working differently or not working at all unless the target environment magically happens to be totally identical to yours.

We use docker to solve the distribution/deployment headaches where we don't know and definitely cannot influence the (different) environments where the package will be deployed. Docker made a headache go away for us - we can make a deployment available to others, they can run it, and it generally will just work properly the first time.

The earlier alternative for that style of "reproducible deploys" was full virtual machine images, not tar. The alternative in the years before shipping VMs was easy was configuring it manually on site or (if it made financial sense) shipping pre-configured hardware. Something like tar was never a reasonable alternative for deploys - even decades ago when big unix vendors controlled/owned/managed your whole system, them 'simply deploying' a tar package worked usually, but didn't work reliably.

Dunno if this has been mentioned, but on the reproducible builds front, you can use bazel (Google's build system) to produce docker images: https://bazel.build/blog/2015/07/28/docker_build.html . It's a little more difficult and explicit to introduce non-determinism into a bazel build.

Doesn't auto rolling evacuation like that give the ability for one malicious user that found a flaw in an app to DoS an entire kubernetes cluster by filling disk space?

Systemd + ansible is very close to what we have. We are python shop so we use: - salstack to deploy and update virtualenvs and applications. - salstack to install and configure upstart jobs - logging goes to /var/log/upstart but this gets shipped automagically by salstack configured per app awslogs config. - All config is read from zookeeper. - The only thing that is initialised on the machine is a ZK host environment variable (by salt-cloud). Everything is ummutable. (Using Postgres RDS for the moment). A system dies we can just make another one with salt. We can run multiple services on one machine or one per machine or install them in lxc containers trivially.

Minor nitpick... I believe you mean "saltstack". Yes?

Sorry, yes saltstack. Typing on my phone while my kids jump on my head. saltstack is like a toolkit for all this sort of stuff.

Off topic, I just looked up the definition of nitpick. Apparently it refers to removing nits, the eggs of lice :O

https://en.wikipedia.org/wiki/Nitpicking

I did not mean to imply your comment has lice :)

I have kids in primary school, there are probably some lice in there somewhere.

I really like the CoreOS approach of using systemd for container supervision.

We use ansible and systemd with around 100 services in 3 continents and works well.

Cool. If any of it is opensource, please post the links in the reply, I'm curious.

+1 for systemd + ansible. That is exactly my setup on ubuntu 16.04 deployments.

I see a lot of "me too" responses to this, and for the life of me, I can't figure out what systemd brings to the table. I understand the Ansible part, but what does systemd do that's notable? My understanding is that it's an alternative to init, and I can't imagine calling out init as part of my deployment/config strategy. What am I missing?

I'm wondering the same thing, as someone who has only started using systemd. From what I can tell it handles:

- starting the various pieces of the app in the correct order - permissions (uid, cgroup, maybe even virtual file system) - environment variables, working directory - passing around sockets - restarting things if needed - monitoring resource usage - centralized logging

  If you’re deploying C programs that rely on system
  libraries, things may get tricky if you cross flavors or
  versions of linux. But you can probably deploy static-linked
  executables more easily than setting up docker.

For various reasons static linking isn't a good idea for C applications. glibc doesn't work well when statically linked. musl works well. But glibc has strong backward compatibility using symbol versioning. It's not usually worth the trouble to build and link aginst musl if your distro uses glibc. What you should be worried about are all the other third-party libraries, which aren't often written to be statically linked--because of namespace pollution, because of slightly different semantics between dynamic and static code, especially at application startup.

Fortunately, Linux supports packaging apps much like macOS bundles, where all shared objects are kept under a relocatable subtree. When building a binary that will be installed in bin/, for example, just use the following GCC flags:

  -Wl,-rpath,'$ORIGIN/../lib' -Wl,--enable-new-dtags

The first flag tells the linker to find libraries in the lib/ directory adjacent to the binary file itself. The second flag tells the linker to try LD_LIBRARY_PATH first rather than the embedded rpath. Debugging and regression testing can be very difficult without the ability to use LD_LIBRARY_PATH, and unfortunately LD_LIBRARY_PATH has lower precedence than embedded rpaths, thus the need for --enable-new-dtags to change the default behavior.

Note that $ORIGIN is a special string that is expanded by the runtime linker, not by the shell at compile time. It's an unfortunate choice of syntax. Getting $ORIGIN to pass through to the compiler without being evaluated by a shell can be tricky when dealing with recursive make and shell invocations, such as from an RPM spec file.

Another example: when building a Lua module I'll use

  -Wl,-rpath,'$ORIGIN/../..' -Wl,--enable-new-dtags

because Lua modules are usually installed under $(prefix)/lib/lua/5.?/, two directories below lib/.

Allowing LD_LIBRARY_PATH in production is a potential security risk.

Isn't that a loader configuration issue? You can use different loaders in production vs when debugging.

And then you might just end up using setcap on it anyhow, perhaps to load privileged ports, which is treated like suid, making it so that the linker won't look at LD_LIBRARY_PATH, in which case you have to create a conf file with your paths and feed it to ldconfig after using ldd on the program because you're confused about why it suddenly doesn't load its libraries.

And there's another fun security restriction in there where chown erases any setcap permissions you had applied, just in case you had to adjust something and didn't understand why you suddenly lost your setcap permissions.

Why? If you control that, you likely also control PATH and others. There are lots of variables which mean game over if you have write access to them. And you also need a local fs write access to use it, so that's another step. I know it could be something to restrict, but it's really far down the list... (configuring selinux/apparmor would solve the same issue but in a more generic way)

Using -Wl,--enable-new-dtags merely restores the default behavior. More specifically, it only restores the usual precedence behavior.

Most people don't set the DT_RPATH ELF tag. It's usually considered best practice to NOT set DT_RPATH. And that's generally good advice, except that advice assumes absolute paths. Most people aren't familiar with $ORIGIN, which allows using relative rpaths. In any event, my point is that most environments already allow LD_LIBRARY_PATH. If it's been restricted then --enable-new-dtags won't change that.

What --enable-new-dtags actually does is set DT_RUNPATH instead of DT_RPATH. DT_RPATH has higher precedence than LD_LIBRARY_PATH; DT_RUNPATH has lower precedence.

To reiterate, DT_RPATH and DT_RUNPATH are independent of LD_LIBRARY_PATH, and don't effect whether it's permitted or not. I recommend --enable-new-dtags to avoid confusion and unintentional side-effects.

I've known about $ORIGIN for awhile, but only recently hit upon --enable-new-dtags. I was hacking on a project recently ported to and packaged for CentOS 7. The relevant components were a Lua C module (acme.foo) binding a shared library (libacme-foo). The in-tree unit and regression tests used LUA_PATH, LUA_CPATH, and LD_LIBRARY_PATH so the in-tree objects were loaded instead of any that might happen to be installed on the system. I was adding some debugging code to the shared library and running `make check`, but my changes didn't seem to be coming through in the tests. I was totally confused.

After far too much wasted time, I eventually realized that CentOS 7's base Lua 5.1 package was built with an embedded rpath of /usr/lib64. A version of the package I was hacking on was already installed on the system. So /usr/lib64/libacme-foo.so was linked when dlopen'ing the Lua C module instead of the in-tree libacme-foo.so that should have been found with LD_LIBRARY_PATH. That's how I became familiar with the precedence of rpath and LD_LIBRARY_PATH, and the availability of the --enable-new-dtags option.

I'm not sure why Red Hat built their core Lua 5.1 binary with an rpath, but they almost certainly should have used --enable-new-dtags when doing so. Notably, /usr/bin/perl on CentOS 7 is built with --enable-new-dtags; it has DT_RUNPATH=/usr/lib64/perl5/CORE. Just as notably, AFAICT Python isn't built with any rpath.

> "If you develop on linux and run on linux and you’re in a deploy-as-source language, you may not care that much about repeatable builds."

If you develop in Elixir or Erlang, there's an additional issue that containers (such as docker) tend to interfere with hot code reload.

There are work-in-progress solutions (such as Elixir Distillery) that emphasize turning your app into a single package which can be deployed anywhere.

Even if you don't use Elixir, you may enjoy seeing how the Distillery packaging system works, and how hot code reload works.

A good intro is http://www.east5th.co/blog/2016/12/26/deploying-elixir-appli...

Likewise when doing a reload of a Rails service.

In the long term I predict that base OS everywhere will improve support for deployment, workload scheduling, resource allocation, endpoint discovery, and dependency management. These will match and eventually surpass the additional capabilities that containers offer, and then we can all go back to putting files on a server and restarting a process, which is all that 99% of us actually need.

The alternative is that container platforms grow their own hypervisor, effectively replace operating systems, and then build out their capabilities in storage, filesystems, networking etc until they are functionally basically indistinguishable from operating systems c.f. VMware. But I think the former option is more likely, it's just a gut feel.

Docker containers can still support hot reloading of Erlang/Elixir applications. It's just not a particularly-pleasant experience OOTB. It's almost always easier to just restart the container or to not use containers in the first place.

Hot code reload interferes with immutable deployments. I'd prefer the immutability over hot code reloading.

Nope, hot code reload is the same thing. They've just moved the immutability line even closer to the code.

> What will it take to build this? 6 months x 3 smart devs who understand the linux kernel. So like $600k.

It seems like PaaSes are the new JavaScript frameworks[0]. Kubernetes now has most of the "critical pieces" mentioned and Helm addresses the "rigid" config criticism. It's not newbie proof yet but I doubt your 1.5 man-year project would get anywhere close to K8s is in terms of maturity and stability. Please do your homework and consider contributing to an existing project before re-inventing the wheel.

[0] https://github.com/search?q=paas

Do Linux kernel devs really make 6 figures on average??

The median salary for a software developer in the United States is six figures: https://www.bls.gov/ooh/Computer-and-Information-Technology/...

In the Land of Unlimited VC.

Salary is not the only cost.

Nomad [0] does some of this already:

  * config file for resources.
  * It does log capture/delivery mostly.
  * it does chroot exec.
  * it handles network ports, file paths, etc
  * handles batch workloads.

It can't handle the load balancer piece directly, but if you use the consul tie-in, you can make it work, since it does service discovery, and then something like fabio [1] /traefik can handle load-balancing.

[0] https://www.nomadproject.io/ [1] https://github.com/fabiolb/fabio [2] https://docs.traefik.io/

Using fabio with nomad and loving it.

Have you seen Nix? It can be used in this way, and its configuration language is relatively good.

I love the idea of nix but haven't had a chance to try.

Can you post a link to how nix handles resource permissioning?

A note about nix: I've found the language to be very obtuse and the documentation out of date and fragmented. However the worst sin has been the attitude of several community members: insulting new developers, ignoring valid questions, and self-promotion at the expense of civility. I love the idea behind the system and like most of the community, but the very vocal jerks really put me off.

I've never learned the nix language, but that hasn't been an issue in my usage for the last couple years. It's really easy to add a new expression for a package, which is all I've ever used it for.

Sucks to hear about the negativity. For what it's worth, I've gotten a handful of pull requests accepted with positive experience each time.

I just wish to report that I had only good experience in communicating with Nix/Nixos/Nixpkgs community.

When someone could solve something that didn't work for me, they did. When no one could or didn't have time, they kept silent or politely tried to help me find the solution myself. And everything absolutely for free!

My mileage is different than yours. When I started using NixOS on my laptop I've asked for help a bunch of times on the #nixos IRC channel and people were really helpful.

This was somewhere in the second half of last year.

What do you mean by "resource permissioning"? There doesn't seem to be anything in the article about that.

Both nix and guix seem to have potential. Not sure if they are the "silver-bullet" (in the good sense of actually nailing a hard problem well) but perhaps. IMO they are a bit too (local) package oriented than necessary.

People want absolute determinism at all levels (which Nix can promise), but don't want to go through the hassle that requires.

Nix + Nixops gives him everything he wants. He'd have to learn to write nix expressions, which does put people off.

Containers are great if you've got racks full of actual, physical servers and want to use them efficiently. Most people don't have that problem, though.

People like the idea of isolation that containers (LXC) provide. We don't have to worry about stepping on toes, my container works with any number of other containers running on the same host, not sometimes, all times.

The env is moving fast, Kubernetes and Helm are on a tear. It's also worth taking another look at cluster standup CoreOS+Vagrant/Tectonic/Minikube/kubeadm are filling in gaps.

minikube was not quite primetime when I tried it, but I saw release notes a few weeks later that fixed everything I ran into. (IIRC it was ingress or DNS problems).

Your impressions about Kubernetes sounded quite out of date when I read your post. I strongly suggest you give things another try. Things move fairly fast in this area.

Yep with today's minikube, you can do ingress. For example, I'll point myproject.minikube.local to 192.168.99.100 and use the stable/nginx-ingress helm chart.

When I'm running my application stack on minikube, I can set a helm template variable to myproject.minikube.local. The nginx ingress controller will route the myproject traffic to the proper backing service.

On prod, I'll set the template variable to super-social-money-maker.com and turn up the stack with a helm install. Here, ELB does the ingress for me, but as an app developer, I don't have to be concerned with the particulars of the cluster's ingress implementation.

So do VMs and that was a "solved" problem years before containerization took off.

The VM overhead is ludicrous compared to LXC containers though, even with all that pass-through that modern hypervisors do. I don't get all the hate for containers from the VM folk, those technologies are by no means exclusive, it's just that spinning up an entire virtual machine to run some microservice strikes me as wasteful.

Because your environments are too small and you have too many services.

When you run a bigger site, a lot services are using entire boxes/VMs so it doesn't make sense to stack them.

When you you have tons of systems, you optimize for easier management and debugging. You do not care about "wasting" some resources.

I think your confusion is based on the assumption of microservices. If you're deploying something which shuffles packets, yes, containers are great for quickly spawning instances and getting more utilization on a given host. Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant. (There's a similar point in overhead: unless you're doing privileged calls in a loop, hypervisor overhead hasn't been significant for at least a decade unless you're talking about something like VirtualBox)

Note that I'm not saying containers are bad, just that the people who disagree are almost certainly speaking from experience in different environments where other trade offs make sense compared to where you're working.

> Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant

If you spin up the VM once and it runs for days/weeks that is true. In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy. That's where VM startup time becomes an issue.

> In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy.

That's kind of my point: that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments. If you're doing anything which copies much data or does non-trivial initialization work, saving a minute by launching a container vs. a new VM is still a healthy improvement but it's not the expensive part of a new deployment and unless you're churning many jobs per hour it's a rounding error in the total execution time.

(I'd still prefer a container for that kind of workload but it'd be for the consistency & simplicity aspects)

> that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments.

That might be true for enterprise (I simply don't know) but in the startup world these are these are actually the most common workloads: Stateless apps that scale vertically.

I agree that startup time doesn't matter much if you need to do lot of initializations. But in my experience in most infrastructures these are a few databases of which there are orders of magnitude less than other workloads.

This reminds me of the awesome article by the founder of skyliner: https://blog.skyliner.io/the-happy-genius-of-my-household-2f...

Google built Borg/Kubernetes because they own all the host machines which are super powerful, and they need multiple apps on one machine to cut cost. But for the rest of us building apps, different VMs provided by cloud provider, such as EC2, already provides a nice way of resource isolation that is also cheap, so why do we need to run multiple containers per machine? Suppose c4.4xlarge is too big for your app, for cost effective reason, you want to leverage Kubernetes to run multiple apps on the same VM, why not just spawn multiple c4.xlarge or c4.2xlarge VMs, and run only one app per VM for you?

I admit this model won't fit us all, but for a significant number of us, this is already a good enough solution like containers, and it works today without needing to setup all kinds of fancy software defined networks, which we never know how long it's gonna last without signaling an issue.

"It also gave me a duct tape feeling or batteries not included – like the critical pieces (docker support, DNS, load balancing) weren’t integrated into the original design."

I'm sorry, this article^H^H^H^H^H rant isn't worth reading. I don't understand why people write these posts where they so poignantly display their complete ignorance alongside their strongly held opinions. It's tacky and doesn't reflect well.

Load balancing is literally built-in via kube-proxy, NodePort and/or CloudProvider integration. Kube-DNS has existed for years (and is intentionally removable, but not at all duct-taped on). And the claim that Kubernetes didn't have Docker support... I mean, I literally don't even know what to say to that.

my understanding is k8s was originally designed for lmctfy and docker was added shortly before launch

Yeah, lots of this article didn't make any sense. Especially with regards to Kubernetes.

Most of this seems like reluctance of new Google or "Beta" technology.

I tried my hand at Docker Compose, then Docker Swarm as "native" Docker solutions for quickly deploying scalable WordPress LEMP stacks. I would completely agree that neither of those are mature for this type of implementation, especially when it comes to persistent storage.

Kubernetes, on the other hand, is another story. As long as you stick to using Docker strictly for building your container images, then Kube literally takes care of everything else. And this is all coming from a strictly ops perspective.

EDIT: I would also note that Kubernetes docs are probably the best I've seen. They may be complicated, yes, so up your reading/comprehension skills and quit wining :)

These pieces were bolted on later, as an after thought. They are critical and should have been there for the start.

Many people have been forced into docker for a few years already, facing a ton of critical features that are completely missing. kube is a brand new project that only started to be usable recently, you seem to not take that into account.

It doesn't help that marketing around containers is "over promise and under deliver".

Yeah I would agree with that. I really dove into and started fully deploying Kube around v1.5.

I would agree that Docker oversells and underperforms as well. I think it should be relegated strictly as a containerisation primitive and let the big boys handle orchestration.

Docker documentation seems to lag behind a whole minor version of the actual software.

It still baffles me how they are pushing Swarm as a production-ready orchestration tool but don't have persistent volumes completely figured out. But I'm just happy Kube is doing it right.

>These pieces were bolted on later, as an after thought.

First, I don't know why you think this? Moreover, what is the practical implications of the claim? Have you ever had a problem with DNS service discovery in a Kubernetes cluster? Or else what is the point of bringing this up at all? (Nor does this change the inaccuracy of the other claims)

>Many people have been forced into docker for a few years already, facing a ton of critical features that are completely missing. kube is a brand new project that only started to be usable recently, you seem to not take that into account.

It's been 1.0 for well over a year and I still don't know what you're getting at in regards to docker. Kubernetes is (more or less) powered by Docker (at least historically, it's getting to a point where the container runtime is pluggable via CRI/CRI-O).

>It doesn't help that marketing around containers is "over promise and under deliver".

Also no idea what this means. Kubernetes and containers have done everything I've expected them to do. I usually see this comment from people who haven't actually ever used them for more than a day and gleefully point to articles talking about stale docker images with vulnerabilities, ignoring that nearly the exact same problem happens with naive VMs + CM deployment methodologies. Or otherwise think they're magic, when they're not.

Not seeing anything to change my initial conclusion here.

Judging by the comment. You're the one who used Docker for a day or on nothing critical. Maybe you're also lucky to have started with Kubernetes, maybe even fully managed on Google Cloud.

I used it for 3 years at different companies with more or less hassle. The route to container has been full of dangers.

>You're the one who used Docker for a day or on nothing critical.

Not really, but thanks for coming for me instead of offering any substance beyond hand waving.

>Maybe you're also lucky to have started with Kubernetes, maybe even fully managed on Google Cloud.

I mean, sure, I think it's crazy to try to use Docker in Prod without an orchestrator and don't think Swarm Mode is mature enough for such a use case. And no, I don't use GKE.

Now that we've dropped anything that was really being discussed, let me agree with you, Docker itself is not my favorite, but that has nothing to do with containers or orchestrators in general and instead has to do with the QA of Docker and the frequency of breaks and regressions.

We both agree that orchestration is critical. Yet orchestrators are fairly new, the docker hype was strong long before they were viable, or existed at all.

You'll forgive me but implying that one one should just use an orchestrator points to one ignoring their history -or lack thereof-.

I have seen entire clusters taken out by a regression. Sadly, even the best orchestration cannot do much better than the core it is running on. You can google "docker in production" to get more substance.

>I have seen entire clusters taken out by a regression.

I think we agree a lot actually :). Not sure Docker would call this one a regression, but... https://github.com/Azure/ACS/issues/3#issue-201986759 (that's me)

I've started using K8s and so far so good. Minikube makes local dev easy. However, I use K8s with ansible. So now I can:

* Build a fully baked container for local use

* Build a container that mounts my local filesystem for the code directory so I can have a sane dev-test cycle with hot-code reloading

* Mount a directory when running locally containing credentials for accessing Google Cloud services

* Spin up a K8s cluster in arbitrary GKE accounts, template and deploy my deployment and service accounts, pin some with external IPs, etc

* Push containers to GCR

* Deploy my containers wherever - shared directory ones can only run locally.

This is for an architecture with about 4 microservices and that will probably grow.

Now I'm working on an ansible playbook to zip the code directory of one of my microservices, upload it to GCS, then run a build container on my cluster to build the docker image for my microservice and push to GCR from there so I don't have to waste time pushing large containers up to GCR. Once this is done I'll look at promoting containers through dev/test/prod environments since all config will be done with env vars.

I've never read about using ansible with K8s, but to me it's a no-brainer. Most people seem to cobble together bash scripts, but using the best of both has really led to a good experience.

The benefit we hope to get is isolation and cost efficiency.

That's interesting. I'm using docker-compose for local development at the moment. Do you have any experience with it and would you choose Minikube over docker-compose?

You didn't ask me, but here you are:

I'm using docker-compose as well but I'm looking at moving to Minikube, so I can more realistically test my K8s clusters before sending them down to CI/staging.

I also don't want to maintain both docker-compose and K8s as I basically am ops by myself and I have NaN tasks in my backlog.

I haven't used docker-compose. I'd been holding back with docker until all the stuff like service discovery, scaling, secrets, sane local dev and deployment experiences were available. So far K8s seems to tick all those boxes.

Since you're already on GCP/GKE, have you considered using Cloud Container Builder? https://cloud.google.com/container-builder/docs/

It provides an API for what you're already doing with uploading zips to GCS, doing your build, and having the resulting container end up in GCR.

Disclaimer: I work at Google on GCP.

Thanks. I'd missed that. It looks pretty close to what I want.

Just one suggestion - I'm uploading python files. It'd be nice to be able to exclude my `venv` virtual env directory from being zipped and uploaded since that's strictly for local use. From the docs I'm not sure if that's possible (I've not gone into detail with them yet).

If I can also run tests as part of the build process then I'd be sold.

So many docker containers I use at work just have a ton of apt-get or yum statements (depending on a debian or centos base) and people rarely pin dependency versions. I really feel for the "repeatable build" issue, not to mention dependency rot.

If you want to keep things up to date, there are many examples scripts out there that either 1) checking to see if your base container is out of date (if someone updated jessie:8, you should rebuild) and 2) checking if any packages are out of date (sometimes limiting to just security updates) meaning you run apk, yum, apt, etc. within each container (or extract their package lists and examine them outside the container).

You gain a lot in the isolation, but debugging can become more difficult and you've introduced new areas of dependency rot.

I generally like containers, but hate how there are so many different orchestration formats (k8s, marathon, nomad, swarm, etc.) many different network layers (weave, flannel, etc.) and many different ways to package/fit components together (CoreOS, DC/OS, etc.)

The ecosystem is messy.

I'd like to build something that attaches cryptocurrency payments to API calls, basically a 402 payment required implemented with the Lightning Network.

With such a system in place, pulling updates could be pinned to a particular API call, with a particular payment address, which itself was attached to a particular data store. In theory, as long as you had access to the wallet that did the build before, you'd be able to do the same build at a later date.

I'm greatly simplifying this here, but my conclusion after getting a crypto payment OpenStack instance launcher PoC working was that applying crypto payments to deployment methods basically provided a lightweight federated layer that allowed logically linking processes, similar to what you would get with a Zapier/Apigee/IFTTT knockoff. Applied to microservice architectures it would allow deployments to be both trustworthy and repeatable at later dates.

Json web token (jwt) is probably what you're looking for

"6 months x 3 smart devs who understand the linux kernel. So like $600k."

Show me that $400k/year linux kernel job.

He's talking about contract work, for which the quoted sum is reasonable.

Depends on job level. For a senior SDE maybe. For principal definately. Actual kernel/embedded systems/driver develoment in Seattle Dresden and Cape Town off the too of my head. https://www.amazon.jobs/en-gb/search?base_query=linux+Kernel

An employee costs much more than just the salary he receives at the end of the month.

Show me the $200k job with $200k of overhead instead then.

Looks like any senior job at google/facebook/microsoft/apple.

150-200k in salary + 50-100k in stock + 50k-100k for health/401k/parentalleave/food//office/hardware/concierge/gym + employer taxes + ...

rkt seems to meet his "define resource access in a config file" requirement in principle.

The separation in rkt between setting up the isolation and handling the images or running of the code means that you can opt to launch a rkt image in anything from an actual chroot to a full container or even a full VM, or you can write your own if you have weird needs (e.g. if you want to do a seccomp-bpf type sandbox, you can do that).

E.g. I have deployments with etcd in a chroot (using rkt), and the rest deployed as containers, which with rkt defaults to use systemd-nspawn, and which means they have systemd running as the init in the container too and logging goes straight to the journal just like everything else.

Cron really depends on scale. For small deployments it's simple enough to simply use a cron or systemd and use a simple wrapper that uses e.g. Etcd or Consul or even a database table to arbitrate who gets to run for jobs that shouldn't run more than once. In practice depending on exactly what you do it's often simpler/better to write the jobs so it doesn't matter if they're started more than once anyway. Seen too many nasty surprises in systems where people just assumed nobody would ever end up with the scripts overlapping in time only for something to slow down processing until they do just that anyway. If you're first doing that, all you need to do to handle a smaller distributed system is to put your locking mechanism somewhere shared.

At scale it becomes more complex, but the overall principle still holds, though you may want to separate scheduling aka. the various distributed cron's.

As for load balancer, I agree it needs standardising, but it really is a small enough issue for smaller setups that it's fairly easy to solve. I've come to rely on setups that uses Nginx and rewrites incoming hostnames to a local hostname (e.g. foo.com => foo.com.service) and use resolvers in Nginx to point them to a SkyDNS instance. That means all the container config needs to do is set/delete Etcd keys to receive traffic for a given hostname.

But there's several load balancers out there that now can pull config directly from a suitable shared datastore, be it Redis or Etcd etc.

> When I used kube in 2016 the cluster turnup support was bad; it seemed like you could use it managed in the G cloud but not anywhere else.

You might like to look at Kubo[1], which was jointly built by Googlers and Pivots to make this more manageable. We built it based on requests from joint customers who want BOSH as their infrastructure-level platform and the choice of Cloud Foundry or Kubernetes for application-level platform.

[1] https://github.com/pivotal-cf-experimental/kubo-deployment

edit: forgot my usual disclosure. I work on Cloud Foundry for Pivotal, though not on Kubo.

Are employees of Pivotal really called "Pivots"? Am I shallow for finding this repulsive? Same with googlers for that matter.

I've never heard Pivot before. Googler has been around for ages.

A lot of things about Pivotal inspire strong reactions, pro and contra. I don't see why a demonym should be any different.

(Also, it saves syllables vs the alternatives)

Twitter employees call themselves tweeps, which might be the worst of the corporate self-nicknames.

I applaud jacques_chester's persistence. On every Docker, Mesos and k8s thread he manages to inject Cloud Foundry and / or Bosh. Impressive.

Because on every such thread, HNers raise alternatives.

I'm an HNer, I know of mature solutions that I trust, therefore Socrates is mortal.

No real opinion about anything in the article except the use of config files.

After using Consul, I never want to go back to config files again.

https://www.consul.io/intro/

Config files as copied around by scp and edited by hand are a pita -- but as checked into source control as a template and staged to machines by your deploy; the very simplest and most robust solution.

Compared to going on your Consul's website to change a value and that value automatically being updated in all of the systems?

Actually yes, I want to version control that change and deploy it like I deploy code. Not blindly fire changes that I can't track into the ether.

You could always disallow consul web interface updates and have `consul kv put` scripts in SCM and validated/deployed automatically.

Yeah man, I could setup a quorum of consul boxes, integrate consul's api into my application, figure out how to manage changes in that system, and then do as you say -- make a change management system to apply deltas on deploy.

Or, I could put one more file on each box in addition to the other 100,000 files I put on there...

Obviously you wouldn't choose consul for this one feature, but if you're already relying on consul as service discovery in your network, why not?

You don't need to invent a change management system, you just need to utilize from a myriad of existing consul OSS.

I'm not saying consul makes all the sense in the world for you, but there's definitely sound architectural thinking behind what the parent is talking about. It's not just a bunch of crazy talk :-D

Can consul create change log trails to keep the auditors happy? Some of us are working in environments that require more rigor than a pet project.

Not throwing stones, a serious question. Made a few decisions to favor log files and change control systems vs a web app that had ACL's but no real auditing or change control mechanisms.

Just because you can store configuration files in consul doesn't mean you should. Consul's KV is strongly consistent, you don't need this kind of guarantees for configuration, and you are inheriting drawbacks caused by the overhead.

As anything it will work fine for small setups, but it will start having issues once you will add more nodes.

Depending on your goal using VCS (for example svn, yes not as hip, but it has a smaller footprint of not cloning the repo) provides versioning, or a database (which provides easy way of changing values) or a hybrid approach will be better.

Consul has some nice features around change notification that make features like pushing a change in conf without needing a code deploy trivial to implement. Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.

There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.

That feature is hacked around http, where the client is constantly connected over http (through long standing connections) and reconnects every 5 minutes (by default) if there was a change. You would be much better to use Zookeeper for this (through watches) even though Zookeeper is also not the best tool for this for similar reasons.

Using a distributed locking system for notification is really missing why it is for, why not use Postgres' notifications, AMQP is it because it's easier to tell it is not the right tool for the job?

I currently have colleagues implementing an event bus (although the use cases they mentioned is that they will most of the time send messages to individual machines or small group of them) for over 10k machines. Their first choice was to use Consul's KV, but after persuading them that we potentially could have scaling issues, they decided to drop it, unfortunately they are still misguided and plan to use Serf which will have its own issues as well, this time it might be network capacity.

The issue though is much simpler. We are all communicating through IP protocol, which is inherently point to point, why building abstractions to the problem, when network already matches perfectly the requirements. I think we are so used to using a tool for everything that it feels weird to not have to use one. I was able to have a working POC in 10 lines of Python code (of course a final solution would be more). Argument against using what I proposed was to not implement custom solutions because of maintenance issues (what will happen if I decide to quit), the thing is that one will write far more code to implement the solution with Serf or Consul.

> Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.

This is not true, KV is strongly consistent including agents (otherwise there's no point in being strongly consistent since all communication supposed to happen through agents), you could relax requirements on individual requests by adding ?consistency=stale, but if your application makes all requests this way, then it is a sure sign that you chose a wrong tool for the job.

> There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.

Similarly there's nothing stopping you from using Solr as NoSQL storage, encode data using thrift and then store it in 2 column table (id and blob) in Postgres (then have a tool to edit the data directly which encodes/decodes the thrift data), using Coherence (Oracle's P2P caching solution) as primary source of data that is ETLed from Postgres data[1] etc. All those things I have seen done and it is really painful to watch.

[1] Perhaps this doesn't look as bad, but ETL to coherence took 5 hours, and Coherence since it was a caching solution if it detected a network partition just to be safe it decided to invalidate all data it had. The solution was to throw money at a problem, so why not have 2 or 3 environments with initial setup so while something happens to one you can switch to another while reloading the data. This kind of worked until a contractor installing a security system in a data center decided to plug in a dumb switch to two main distribution switches configured to work in HA, creating a loop and shutting entire network down. The site was down for hours until all of the data was loaded back.

>kind of like a fancy chroot

You might argue that containers are fancy chroots. They are just built from scratch and packed each time

This exactly hits the problem with containers (and especially Docker and Kubernetes, but to a lesser extent CoreOS's model as well). The crows that seems to like/use it seem to be the people that aren't responsible for maintenance, security etc. and may not want to become the responsible party either. It's often developers that are either simply a lone wolf or part of a team that is big enough that release management falls on one or two people and 'the rest' just wants to churn out code in 'their way' and have that 'way' magically work in production too. Sadly, that is not how the world works.

You will always need some form of configuration management, release management, infrastructure and operations, no matter how compartmentalised and 'serverless' the software coders think they are. Right now, the idea seems to be that if you stick your code and practically a complete user land in a container, you don't have to do those things, as if old style deployments were the reason for those practises to be in place. Obviously, it is the other way around, and by trying to bypass them, you will run into the exact problems they are meant to prevent. (i.e. not knowing what is available/what is released, not knowing the configuration within the infrastructure in a global scale, not knowing the side-effects of two opposing configurations because you have no management for that, not knowing what versions are used because it is spread out over different/many unmanaged files inside containers somewhere, not knowing what is secure/patched, not knowing what else is influencing your environment, no control over parameters that should be globally overridden...)

We take a different approach: you get do write your software, and may only configure requirements within its context. Those have to be defined in a SaltStack Formula with sane defaults, that has to run in a local environment, but also DTAP-style chains of environments with no changes to the app. You don't get a say in what it will run on, but your required facilities will be available. If it doesn't pass automated testing (i.e. sticking it in a VM, running a highstate and expecting a working app) it doesn't get passed to DevOps and you're stuck until you fixed your stuff. With regards to scaling, we make sure that a few calls are always available to the app and/or it's formula, so that we can register where it is and connect it to the correct balancing pools (i.e. incoming requests, but also FS, DB, queue pools). When it's up, it registers, and as long as the automatic health checks pass it gets requests sent to it. If the load is too high, we scale up, if it's been idle too long it scales down. But the app/software/code isn't aware of that nor does it need to be, and using the reactor system, we don't need 'extra' management software to make that happen. You set your KPI's and health checks where you need them, just like you would on any other setup, and you're golden.

I don't think that's a fair characterisation.

Operational platforms have been built on containers and more are arriving all the time. Red Hat rebuilt OpenShift on Docker and Kubernetes. I work on Cloud Foundry for Pivotal. There are many others, including self-developed platforms like the one you've built.

The general platforms are written to make it easy for operators and developers to remain friends, by sharply drawing the boundaries of who does what. Operations gets manageable black boxes and development gets free platform features (routing, logging, service injection, balancing, healthchecks etc).

If a developer needs to care about how these are implemented, or do work above baseline to get these features, then my peers and I have more work to do.

Your last paragraph could clearly be implemented in k8s / etc. but the defining characteristic is that you've built organizational controls to protect & optimize a distributed system. And the main problems you bemoan in the first paragraph are issues of, again, organizational control & maturity.

It's almost, like, technologies are just tools that can be either applied or misapapplied to a given domain, either for better or for worse. Huh.

It's obviously all about implementation and application, but instead of always reading 'Docker is the solution' it would be nice if it were listed as 'just another tool in your toolbox'. The problem I personally have with this and some other 'emerging' technologies is that they are touted as a 'replacement' for Ops, management, or infrastructure, while in practise, it's obviously not.

I think the higher level goals are pretty much aligned with those of kubernetes. That it uses 'containers' to achieve this, is just a side effect.

And it's surprising flexible! You don't need any overlay networking, you can just define a IP range and it will 'just work' as long as your nodes are in the same ethernet segment. If you grow to more than, let say, 500-1000 pods, you need some routing but still can do without overlay networks.

I'm currently working on a project where I use the kubelet standalone. I can give it a pod manifest and it will make sure the specified containers are running. This doesn't need any cluster components or data stores and is already useful for tiny use cases. Also using this now for my tests instead of docker-compose.

Isn't resource access like that found in CloudFoundry or Chef Habitat?

Isn't load balancer standardization found with Kubernetes Ingress? Though it doesn't let you select the lb algorithm though.

Isn't cron found in Kubernetes Cron Jobs?

That's sort of what systemd could have been, before it tried to do everything.

Kiss is not something the systemd designers don't seem to understand, nor modularity

> my iterative dev setup often isn’t that similar to my ‘under docker’ setup

This seems like a strange complaint to me, because it's one of the best things about iterating with docker. Make a change, do a build (super quick because of fs layers) then run it. If it works, ship it!

Not only that, but now I can do the same thing in Windows. It's a wonderful time to develop linux applications.

I don't know about the OP, but a lot of people rely on hot code reloading to see live changes, be it a Node.js app or a React UI. The Docker build workflow is way too slow for that. When I work on React apps, I expect the UI to seamlessly reload every time I save a file.

You can accomplish this perfectly well with Docker, of course, though you have to use volumes and so on. Docker for Mac's mounting support has inotify/fsevents on both sides, though the volumes are not fast. (Fast enough for a React app, but enough for a Postgres database.)

Exactly. First thing I do when setting up a new project now is to set up a Dockerfile for the container I will build things in. For scripting languages etc. it's also easy enough to just mount your working directory anyway, so the Docker container provides the environment but you don't need to rebuild the container for every change.

E.g. for my Ruby web projects I'll often have things like this in a Makefile:

    rundev:
        docker run -t -i --rm -v ${PWD}:/usr/src/app -p 8080:80 ${IMAGE} bundle exec shotgun ${SHOTGUN_OPTS}

With a Dockerfile that might be as simple as this:

    FROM ruby:2.3.1-onbuild
    EXPOSE 80
    CMD bundle exec puma --port 80

Gives me a predictable environment where I "only" rebuild the Docker container when I add/remove dependencies, and when I run it in development, shotgun will hot-reload the Ruby code from the mounted host directory.

Saves me from polluting my hosts with whatever version of code I need for different projects.

At this point there's very little code I'm willing to run outside of containers, not for security reasons but because they prevent my laptop from getting full of cruft.

I couldn't imagine going back to developing outside of containers.

"What will it take to build this?

6 months x 3 smart devs who understand the linux kernel. So like $600k."

Really? $600k for three smart developers for 6 months?

So a smart Linux kernel developer costs $400k per year all-in?

How about various web and node.js developers LOL. Where are the other fields with comparable costs?

kubernetes + vault is a powerful combo.

Everything the author describes sounds like docker-compose to me. Combine that with Convox and you've got a powerful, easy-to-use cloud platform.

Can convox do docker compose?

Yes, the point of Convox is that it deploys your docker containers to AWS using your docker-compose config. It sets up a VPC for you, ELBs, autoscaling group, private docker repository, the whole deal :)

I like this concept, but maybe add sandboxing too.

Something like Ubuntu MAAS? https://maas.io/

wisdom from (not-so-distant) past

https://news.ycombinator.com/item?id=13262045

https://news.ycombinator.com/item?id=12804185

I'd be interested in a project like this if it was targetting FreeBSD.

What's wrong with strong config and deploy primitives + containers?

So you are asking for cloudron?

Wouldn't all of this be provided by the Plan9 paradigm?

No, I think it's unikernels solve all problems.

I just want bare metal servers. I really don't get the craze over containers.

EDIT: spelling, sorry bears, no metal for you.

Anyone who serves metal to bears should be put in a container.

bare :(

You know, the entire time I keep telling my self "don't type bear, don't type bear" and my wife distracted me and I typed bear.

Am I the only one that sees the container-stuff as an intermediate stepping stone until AWS lambda (or similar offerings) can be applied on every coding Problem?

> ls $home

No network connection.

> sudo ifup eth0 && ls $home

AWS coreutils: unable to connect to target filesystem host. Please open port 22 and ensure that the aws-guest user has full access to the filesystem.

> find -name aws -exec rm -rf {} +

> sudo apt-get install coreutils -y

Reply on news.ycombinator.com