Hacker News Reader

Comment by tazjin | Hacker News Reader

Slightly offtopic from what the article focuses on, but the thing that bothers me about all these built-in CI/CD things in software like Gitlab is that people seem to be perfectly content with building an artifact and then just ... pushing it out.

In traditional deployments that may be some kind of "copy this thing over SSH and make it go!", in Kubernetes-land it's more like "lets just modify this API-state to point at the new image tag!".

Neither of those produce a consistent record of changes applied and state mutation like that is very error-prone.

We actually use Gitlab's CI at work, but we have our own deployment solutions built on it that end up making git commits into the NixOS & Kontemplate repositories which then run their own pipelines to deploy.

This way we can always answer the question "what set of applications at which versions was deployed at $time?" and also roll back consistently.

View on HN

Sorry but it's not clear what you are saying.

What is the fundamental difference between modifying your k8s deployment to point to a new tag and your solution?

You can easily roll back (except with database migrations) and easily see what version is deployed when.

The idea of keeping all state around your deployments in git is being called GitOps: https://www.weave.works/blog/gitops-operations-by-pull-reque...

Infrastructure as Code is a useful thing and this is essentially building on top of that idea. For example, you can spin up an exact copy of any environment automatically including all software running on it. If something utterly takes out an environment you can spin up a new copy of it and be certain it's identical to what was there before. You can rollback and manage multiple pieces of software at once if there's dependencies between them. You have an automatic and immutable audit log of who did what and when. Using git you can keep track of all infrastructure related changes in one place instead of trying to piece together information from multiple sources when debugging.

When you update a Kubernetes deployment in the API directly you're essentially modifying a global variable. If you do this as the result of some imperative process (e.g. a CI pipeline), you don't have any record telling you what the value should be - you only know what it currently is.

You're also only modifying a single piece of your whole state at any given time, meaning that if you have the services A and B and their deployment pipelines independently modify state - you have nothing that declares any relationship between which versions of these should be deployed together.

There's a little piece of infrastructure wisdom I've learned over the years, I refer to it sometimes as "tazjin's law":

Any infrastructure component not controlled by a reconciliation process will eventually fail.

Versions of dependent components will get out of sync, configuration is being deployed independently of the application, and so on.

In order to reconcile your current state with your desired state you must know what your desired state is.

Does that explain it?

That doesn't explain it. You always need some global variable saying what the current state is. In your case, maybe it's the branch pointer in your git tree. In the k8s deployment, it's a tag value.

No, those don't contain the same information.

Our Kontemplate repository contains the state of an entire cluster, including all versions and all configuration.

A tag value is a single piece of mutable data that is in no relation to the other relevant data.

The tag points to the current version of your docker image, which contains the current configuration, except secrets which you don't want in a repo. Older deployments point to the older docker images which contain the older versions.

Still don't see any advantage to justify the additional complexity in your system.

We have more than one service, more than one environment, and auditability requirements (due to being a financial institution).

I find it more complex to try and keep track of remote state modifications than to have a single source of truth, but whatever floats your boat ;-)

You can export k8s state as JSON/YAML and put that into a git repo if you want.

I had this exact same problem with the way i & my team approached our kubernetes clusters. I come from a network management background and in that space we all know of a tool called RANCID. A simple tool that logged into your switches/routers, made a dump & removed counters/oscillating data and stored it into CVS.

This predates infra-as-code.. it took years before someone wrote a clone that used Git instead of CVS

Made something stupidly similar for kubernetes: https://github.com/pieterlange/kube-backup

What about Helm?

I aim for as few moving components at runtime as possible. Helm has tiller, which I'm not a big fan of.

In addition its design (with warts like the separate versioning layer of charts) adds complexity that I don't think is necessary for most use-cases.

Reply on news.ycombinator.com