The Configuration Complexity Curse
blog.cedriccharly.com
blog.cedriccharly.com
Turns out the complexity didn’t go anywhere, it was just biding it’s time, waiting for the right moment to strike back.
Damn you, entropy!
S3 is an example of a correctly-favoured service. Object storage is a non-leaky abstraction. Object storage doesn’t need to know what an object “is” or why something is storing one. There’s no use-case-specific policy that can be applied to only specific “types” of objects. There’s just objects and buckets, and policies you can apply arbitrarily to any object or bucket because they’re policies about objects and buckets, rather than policies about some higher-level thing.
If your component’s API doesn’t create a clear, obvious, “self-contained” abstraction like object storage’s “objects and buckets” abstraction, then you don’t really have an extractable component; you just have a monolith that talks to itself using an extra layer of indirection.
Nobody said making the leap to distributed systems architecture would be easy. In fact, quite the opposite has been alerted, over and over.
Do not complain about complexity and simultaneously reach for hard problems.
I’m not trying to be snarky, I’m just pointing out that even a on-the-surface-ideal abstraction is leaky as hell. If S3, storing objects, can’t keep it’s abstraction clean, how can we be reasonably expected to keep our own abstractions clean?
The fact that all of those services that seem to be "a part of" S3 can be implemented on top of S3, without touching the code of S3 at all, and without the higher-abstraction-layer code having to reimplement any of the S3-layer logic to do its job, is precisely what makes S3 a well-factored service.
Unless you're writing in a compiled language, use the same language your application is written in for your configuration. If you have a python application, have a python file for the configuration. Same for ruby or whatnot.
You don't need to use the entire language, but at least use the language's lexer/parser (cf. json/javascript). That way, all existing tooling for the language will work for the config files (ask me about how saltstack happily breaks their API because you're not "supposed" to use it, despite the fact that they have public docs for it). Additionally, people won't need to figure out all the stupid corner cases in your weird language that has no uses outside of a few projects.
Additionally, by making your configuration language an actual language, you also simplify a lot of the system design, because the configuration can act directly against your API. This means using your tool from other tools becomes much more straightforward, because the only interface you actually need is the API.
The existence of "configuration language" is, itself, a mistake.
I don't understand why people hate on new languages so much. Learning a language is pretty easy, at most a week or two of concentrated effort. On the other hand, we put years of effort into software projects. If learning a new language would make us 10% more productive with a sizeable chunk of the work, that's a HUGE WIN.
Configuration at scale is something new and we don't have good ways of doing it yet. Inventing new languages and tools to make it easier will be hugely rewarding; WAY more than a 10% improvement. Some of the stuff we invent won't be perfect. So what? Let a thousand projects bloom, we'll see what works and what doesn't. That's how our profession advances.
I believe these are the reasons why everyone ends up using ad hoc solutions and doing it their own way that works in a case-by-case basis despite not enjoying as much generalisation as says the new language/toolkit/library/framework which is basically an implementation following specifications designed to solve a more general/abstract problem in a particular way such as described in this article.
Nonetheless I'm glad this article has received a good amount of upvotes to appear on the front page of HN. I'm just wondering what the conversion rate is like i.e. how many % of people who clicked in would go through the entire article and learn about CUE lang in its details and how many % of these people would end up using CUE lang. And then there is the question of how many % of these people would stick to CUE lang over the long run (says over the course of 1 year).
Question one for anything that you aim at production should be: "Do I really need this?" and only if the answer is a very clear yes and you're not just trying to implement $COOLTECH because you are distracted by its shininess or because 'Google does it too' then you should go ahead and implement.
My #1 technique for improving installations is to rip out unnecessary superstructure which is obscuring why things are going wrong, and more often than not is actually part of the problem. Works every time.
The same goes by the way for modeling your development process on whatever Spotify does with your 6 people development team, and in fact for any other piece of tech that you bring on board. Each of those pieces has a cost of implementation, a cost of maintenance and a cost of cognitive overhead associated with it. The best shops out there use the least number of technologies they can get away with.
I think it is an inherent error in basically all orchestration tools (Kubernetes, cloud build/formation, etc), that they don’t support scripting.
Pulumi has a high-priority issue for deciding how they're going to support arbitrary programming languages: https://github.com/pulumi/pulumi/issues/2430
I'm watching this with great interest.
Edit: looks like it's not high priority anymore :/
- Helm v2 support didn't use Tiller, instead it rendered the YAML client side and pushed that to the server. Sort of how Helm v3 will be, but it didn't work very well in practice with the charts I wanted to deploy. It felt like Helm suppor t was an afterthought and the best way to deploy was to port all your Helm chart dependencies to Pulumi.
- It was really intuitive to create functions that deployed the same thing repeatedly with slightly different settings. I used it for Office365 email server settings. I managed a couple domain names that use Office365 email so I had a single function that deployed the correct settings to the specified domain name, and I just called it twice with slightly different parameters. Way more intuitive than Terraform.
The ability to template Yaml in Typescript and create infrastructure is mind blowing, all the time checked by the compiler. Well it's not so much templating as using Typescripts built in JSON syntax.
Using VSCode we can refactor our infrastructure code, i.e. create functions for sub-levels of our Yaml.
So we effectively combine our Kuberentes Yaml and infrastructure. It's great. Try it.
(I maintain the k8s provider at Pulumi)
(I work on the k8s provider at Pulumi)
Is `-k` very recent?
Unless your scripting language is both deterministic, essentially syntactic sugar for a dependency digraph that the orchestration system knows how to calculate reduced patches of, it’ll break the “convergence” abstraction of the orchestration system: in est, it won’t know for sure what state the system is in after the script runs, so it won’t know what needs to be done to put the system back into any other state later on.
Not that such scripting languages don’t exist. Apache BEAM, for example, is a set of SDKs for various languages that let you write code that compiles to dependency+data-flow digraphs. But it’s probably not the type of scripting you’re thinking of.
I should also mention, though, that systems like k8s do allow arbitrary programming (not necessarily “scripting”) through Operators (https://coreos.com/blog/introducing-operators.html). Rather than scripting the orchestration system itself, you write a new type of component for it to orchestrate, where that component is both created/managed by the orchestration system, and, through delegation, is responsible for implementing the method by which the orchestration creates/manages other specified types of resources.
I think a lot of these tools are based on the assumption that declarative configuration should be sufficient. The tools eschew scripting because introducing it would call their basic assumptions into question.
Do this instead:
1. Use flat yaml files. No loops nor conditionals, no complexity.
2. One (single) yaml file templated by ansible just for secret/sensitive stuff.
3. Done.
Boring is better and is easy to diff.
If you're throwing it all into a docker container as a single artifact you don't need a separate language to patch into a complicated config parser. Just use hardcoded objects/maps in whatever language you use.
I do prefer to have lots of dumb files than having to deal with the cognitive load of a tool that in the end will generate lots of files anyways.
I'm going to chew on this. For me, Don't Repeat Yourself has been one of my highest values. My former colleagues would copy and paste code everywhere. It was a pain to make changes to, and I relished refactoring it.
But I also regret some of the libraries I wrote in my early years. They aren't designed how I would today, but now several applications depend on them.
One thing I will say is that it's okay for your first draft to be ugly. It helps to see all the duplication before you design the abstraction.
Obviously if there is a manageable way to reduce repetition, take it, but I would not add a lot of complexity for the sake of brevity. That's turning your dev team into a data compression algorithm made of meat.
I've been jokingly pushing for over-application of syntax-focused DRY to be called "Huffman coding"
I was just arguing that adding a lot of complexity to reduce duplication is an exercise in diminishing returns pretty fast.
I used to say "less complexity" but I've found that even for me, sometimes when I do something that seems simple and less complex, it ends up requiring more cognitive load (it's not descriptive enough, or it tries to do too much "magic", etc).
When you treat configuration as flat files your configuration becomes a 1:1 mapping of the thing you have running.
I won't speak to which is "really" DRY, but I think the original formulation is more useful. The more typical focus on surface syntax misses out in two respects.
First, a single piece of knowledge might be represented in multiple places even when they look different. For instance, if I'm saying "there's a button here" in HTML and in JavaScript and in CSS, there won't be any syntactic repetition but that's not DRY per TPP.
Second, just because there is syntactic repetition doesn't mean you're encoding the same piece of knowledge. Sometimes two pieces of code happen to be the same, but each looks that way for different reasons and are more likely to change independently than together. In that case, unifying them isn't DRY (per TPP). I joke that you're not improving that code, simply compressing it.
I think that original formulation is pretty spot on, although even then would note an exception for deliberately saying important things in different ways for error detection and clarity, but only if they are actually checked against each other.
I mention all this to illustrate that we are nowhere near NetAppAmaGooSoft scale. Nevertheless our deployment is complex. We can't just have a few hundred identical machines. There are many heterogenous parts, and they have to be hooked up to each other. We currently do this with ~20K lines of HCL applied with Terraform. It's mostly written by some hella-smart infra engineers, and is very well factored.
Still, it's a BEAST. The Terraform Enterprise workflow is tedious, and writing configuration is a lot of work. We would love to replace it, with ...something better. There are alternatives out there, but nothing that's obviously much better. It wouldn't be worth the migration effort. As far as I can tell, this is the state of the art, and it sucks.
My coworkers are sick of hearing me say "We are not Google", so I'm sympathetic to the YAGNI argument, but there really is a problem here. Flat YAML files would be a nightmare for us. I bet there are a lot of companies out there that have worse solutions than we do. The default of "each team rolls their own ad hoc deployment tools" masks it somewhat because it's not obvious that the company has 17 solutions to the same problem, with varying levels of effectiveness and reliability, all of which are expensive to write and maintain, and none of which will be reused when a new team gets organized.
Terraform is a valiant attempt to solve the problem with middling results. We can do better, and we need more attempts! I'm excited that CUE exists. It's not ready for my use case yet, but it's very promising. The best thing about it is that it scales well, up and down. If you have simple needs, you can just write flat CUE to start. It's just JSON with some syntax sugar. The fancy stuff can come later, but it's there when you need it.
I'd argue that we might once again be on the cusp of true serverless but in a way that might become ubiquitous. If we could unlock a shared platform like GitHub but for running software we'd be in a much better place.
But I have to ask: isn’t there something simpler which can handle taking a declarative specification and adding imperative behaviour to it? I’m writing — as anyone who knows me might suspect — of S-expressions & Lisp.
They have an advantage over CUE in that one might well choose to write one’s entire program in Lisp & S-expressions. It doesn’t look like CUE is intended to be the whole-program language.
I remember that Tcl used to be commonly used for config-files-which-need-a-bit-of-scripting, but while it is an awesome language (really!) one probably doesn’t want to write an entire program with it, but rather use it to stitch functions written in C together.
Some people say a Turing complete config language is a step too far. With great power....
I find it's syntax more suited to constructing data structures than I do LISP if only because "{}" and "[]" are more concise ways to indicate "object" and "array" than using an S-expression.
We’re already programming in code, just by shoehorning one domain-specific language into another domain.
It works mostly pretty well. It's a lot more organized and powerful than having a bunch of YAML. It does feel a bit "heavy" though. There's also parts of the kubernetes api that are hard to deal with (log streaming for instance..)
I don't put everything in the tool though. Anything that's a one-off I just check in YAML for. But anything that you might do repeatedly I add to the CLI, and the website is for more general non-ops use
Then we can use whatever tools we want to generate and customize as needed.
YAML is ugly and doesn’t really solve a problem.
Jsonnet on the other hand is far more elegant templating language that solves a real need of generating json/yaml files.
Please please don’t use a text templating language. You’re in a world of hurt.
I’d rather have something like that for infrastructure management.
Tooling: I looked at them all. yaml, kustomize, helm, deployment manager, etc and in the end I went with Terraform. The reasoning was simple. We deploy Infrastructure, not kubernetes services and terraform lets you do the entire thing with one tool. It has the ability to do loops and clunky if-like statements but if you want real programming power then you can create your own terraform provider which is what we did. The pattern of how to create a provider already exists so you are not re-inventing the wheel. We even add our own services to our provider so that we have one tool for setup of GCP resources -> Kubernetes resources -> our own service(s) resources. 1 tool, 1 flow, 1 statefile and the same pattern when working with it all.
Establish patterns: "My service is a multi-tenant service and should do X". Don't allow people to get away with special cases when it comes to deploying infrastructure. Create patterns and stick to them. As an example, one pattern I have is that I have 4 variables that all of my terraform modules use: project, region, cluster and tenant. A combination of any of those variables is enough to create unique resources for everything you deploy... dns names, storage buckets, databases, service accounts, namespaces, clusters, etc. Those variables allow me to know where your git repos are, what GCP project your deployment lives in, what kubernetes cluster you are on and what namespace you are in. Patterns.
Keep it simple: There will be pressure to have custom settings and provide as much flexibility as possible. Try and avoid this. Set defaults for values and try and reduce the number of configuration options available to others. In our case, there were around hundreds of environment variables being set on a deploy and most of them were the same. I took the list, standardized the environment variable names, DATABASE_USER, MYSQL_USER, MYSQL_USERNAME, DB_USERNAME... come on guys!, and deployed them as a secret in kubernetes so that all of our running services can access them. Reduce complexity.
>"kustomize would be the most well known tool now it that is it integrated into kubectl. This seems to work well enough and could be feasible for simple use cases. In comparison to data configuration languages though, this strategy breaks down when configurations grow in complexity and scale. Template writers lack abstraction and type validation that matters when things get complicated, as the semantics are locked into an opaque tool and not exposed as language features."
How exactly does this strategy break down? This sounds a bit hand-wavey to me. Isn't Kustomize essentially patching? And isn't the type validation done by the underlying API objects in the yaml that Kustomize is patching? Am I missing something obvious or a more subtle point?
* [0] https://xkcd.com/927/
The other, less good piece is people not being up to the task of going down to the fundamentals and building back up from there. And not just the technical fundamentals, but also those of system purpose, user needs, and how value gets delivered.
Which honestly, I get as well. Having tried using Kubernetes, I'm unimpressed. But it has such enormous momentum that even if I were sure I had a better solution, I doubt I'd bother going my own way. Instead I'd just try to mitigate the pain. Ideally I'd find a way that might lead people out of the technological cul de sac they ended up in.
The ecosystem is fantastic, but it's hard to make the case to migrate an existing configuration to it for the sake of using nix.
Dhall is another option in this space, BTW. It has other faults that score CUE existence points, thought.
I love the idea of Terraform, but find that the language only goes part way, and is itself pretty idiosyncratic. It's pretty nice, though, if you move up by another step of abstraction and write code to generate your HCL. If you get lispy about it, then you can have infrastructure defined as data, generated by code, that is also data...