Comment by sargun | Hacker News Reader

Comment by sargun | Hacker News Reader

The article seems to be tying two ideas together - immutability and infrastructure as code. I’m unsure why the author decided to link the two ideas.

Immutability, IMHO, is a terrible idea for infrastructure as a general purpose rule. Making it more complicated for developers to deploy systems, and requiring deployments to go through many steps (as immutability typically requires a full blue / green deploy), means MTTR will be longer.

Also, trying to fix a problem in situ takes longer (because you don’t have the tools for it).

Lastly, immutability is a lie. If you’re using something like v8, over time, the JIT and GC may change behavior. The cloud vendor may deploy new software.

If your software can be deployed as a fully self contained, stateless application, perhaps you can ignore these problems, especially by constantly recycling infrastructure. Now, the real problems begin to occur when you stop recycling - that’s when really interesting longevity problems appear.

/rant

IMO immutability is a byproduct of adopting Terraform--it doesn't manage day 2 stuff like patching, so the solution in some people's mind is "immutability" instead of bringing in another tool to manage the day 2 stuff.

I personally see immutability as a sign of organizational immaturity around system administration, if you're concerned about drift on production systems you don't have the right config management and RBAC in place. It's never more efficient to replace a fleet vs patching it in place, especially from a time standpoint.

Patching is relatively easy to solve with immutable infrastructure. Build a new fully patched AMI every night and when you release you pick (version of code, version of AMI) to deploy.

I don't think the benefits of this outweigh the complexity and time costs to create and operate it. Having a scheduled task patch the servers regularly with extremely limited "break the glass" RBAC on your servers is better for 99% of environments.

There's really no complexity. Our deploys are blue-green (i.e two ASGs and a load balancer, no need to make it sound fancy). You need to do this anyway because instances in the cloud are inherently unreliable.

Deploying new code: publish a new container, scale up green, switch, whack blue at your leisure.

Patching: publish a new AMI, scale up green, switch, whack blue at your leisure.

If you can patch it, doesn't that mean it's mutable?

Nope, because once it's deployed to production it will never be changed without another deploy.

Then you're not patching it?

How our system works:

1. Every night we build a new freshly patched Ubuntu image.

2. Three times a week we run a deploy that rebuilds our entire infrastructure in place with the latest Ubuntu image as the base [1].

3. Once the deploy runs we never run apt install or apt update again.

So rather than

    server.update()

we do

    server = server.update()

[1] This sounds fancy but I promise it's stupid simple, all the complexity is in the auto scaling groups and the load balancers which we don't manage. And it's the exact same procedure to do code deploys which we do multiple times a day.

How do you handle stateful resources? Say for instance you have multiple elasticsearch clusters with terabytes of data on each cluster.

Well the NixOS idea is the opposite of what you describe, minus the implementation which does have mutable data.

Reply on news.ycombinator.com