The golden age of configuration languages
cosminilie.ro
cosminilie.ro
A huge proportion of people "doing devops" (whatever that is supposed to mean) are in fact really just lobbying for the likes of terraform and kubernetes to be adopted, preferably via some expensive Amazon/Google/Microsoft service. This article is no different. Now the answer is for the sysadmins to code Javascript and use something called Pulumi. I don't remember this being what DevOps was when I was interested in it in 2015.
To be sure, the scale is larger than ever before. But history is replete with "IT in the large". Just look at mainframes, quietly ticking away for decades on loads that people are re-discovering how to handle.
Partly I blame this on the hubris of the young - "this ain't your granddad's IT." Heck, I recognize it in myself from years ago, when I was all in on Linux, but the BSDs had been around a while.
But I do believe there is a wanton lack of communication in the industry and academia. Part of this is on purpose - companies don't like "giving away" their hard won lessons, nor is it (short term) profitable to pay people to write documentation. But there's a bigger culture around "if you don't understand, you're just not smart enough." It's toxic.
We could sorely use something like the group in "Anathem" that does nothing but study history and remind others of it.
I wish we could learn the lessons of the past without having to repeat them and adding yet another standard to the fourteen already out there.
I'm not convinced it's a hubris thing. Many of these "two steps forward, one step back" advancements happen because a commercial entity has intentionally blocked the tech tree from advancing in a certain direction that would undercut their position. The tree still advances, but it has to grow around the blockade, and that involves a certain amount of seemingly redundant work. It's only redundant in the sense of technological novelty, though. It's not redundant in the social context.
People become attached to initial ideas, resist any change, regardless of relative merits.
Everett Roger's The Diffusion of Innovation [1962], marketing's seminal textbook, explains all. Everything since is rehash (at best) or worse than wrong.
But there still needs to be documentation. The why must always be explained, even with the most readable of code, that can only layout the what and the how.
This kind of fanatically DevOps-y approach has a few problems:
1. It's slow and expensive to develop all these scripts.
2. It encourages "disposable infrastructure" in which people think, gee, I need a test version of {very big complicated thing}, so I'll just run these scripts and create an absurdly expensive duplicate that they then forget to shut down. You end up with a cloud filled with VMs that were used once, by one guy, for twenty minutes for a demo, and then never properly deleted.
3. There's no real discoverability or way to iteratively figure out what you want with these things. If you aren't entirely sure how to set something up, you can't really do it by coding it because you'd end up being limited by the tools. Whereas shells and vim and such are designed for interactive work. But then once you've got it set up, for many tasks you hopefully don't need to set it up again anytime soon, so at that point automation is just pointless overhead. It'd be quicker to just drop some notes in a wiki.
With respect to the article, it did very much remind me of the great wheel turning. I used to be an SRE at Google and I saw there two relevant trends.
One was that when I joined they still had some parts of the infrastructure managed by a giant Python script called the Babysitter. The Babysitter was in theory split into configuration and implementation aspects, but somehow not really. It's dead for perhaps 12 or 13 years now so many Googlers would never have encountered it. The replacement system (Borg) used an intentionally limited configuration language, partly due to the experience of the Babysitter in which people had full programmability, and used it! But the borgcfg language rapidly started sprouting various kinds of programmability features too, and in fact one day someone demonstrated you could build Conway's Game of Life in it. Seems like this new call to use full programming languages to create configuration is another turn of this wheel.
The other was hitting the limits of what was automatable. One of the tasks I had to do in my job was to set up some services in new clusters, and shut them down in old ones. This was a multi-step process that was very time consuming and annoying, so there were many attempts to automate it. Unfortunately most were unconvincing, partly because the speed at which new clusters needed to be set up was somehow sort of aligned with the speed at which the general set of tasks was itself changing. So you could attempt to automate the task, but by the time you'd finished the steps involved were already changing, so the next time you wanted to run your script it was wrong. Also: setting up clusters of hundreds of machines isn't something that's easy to test virtually (there were no "mock datacenter" libraries back then).
In the end, what I noticed was the big automation wins came from the infrastructure teams when they improved the underlying systems to be more robust. Us glorified sysadmins wrote a lot of scripts, but it was often hard to prove they really saved a lot of time when the cost of implementing and changing them was taken into account.
Addressing the first point, well, that's almost getting into business level ROI calculations, which at least on an individual developer level might be reasonably approximated by things like "is it worth the time?"[0], but that's never been a perfectly answerable easy question; for one thing, you might need to consider that the time spent is learning that will pay off in the future. I like to think of it as "technical investing" as opposed to technical debt.
As for three, that's a big problem, no matter the scale or project. I do not have the benefit of your experience with software in the large (thank you for your stories!), but even on a smaller project I'm working on recently I run into this problem: we have git repos. We have documentation in the repos. But to get to that documentation, you have to know how to get to the repos, and where do you document that? I tried imagining automating things completely, and the answer I came to was ultimately I'd have to build an AI to on-board new developers to the project. Not ideal.
And like you said, the scaffolding surrounding projects is frustrating too. Again, same project, but this time I'm digging into the build system. Turns out it was converted to make from the previous build system that appeared to be mostly shell scripts, yet the shell scripts are still around, unused, but like little land mines, waiting for someone to tinker with them and waste a bunch of time.
That also sounds like an opportunity for the old guard to write down what they know before it's (they're) gone. I wonder if any of them have, and if so where.
I mean, even if we do assume each new tech fad is mostly driven by hubris, if the results benefit the ecosystem, which cloud undoubtedly has through democratizing scalability, so what?
As far as the secret keeping of industry, I don't by that at all. I've worked for a diverse set of organizations, and every single one of them wanted to either trumpet their own practices to build clout with developers, or utilize knowledge in the public domain so that talent was easier to find. If anything companies avoid having secret sauce in their infra at all costs.
But the activity of keeping secrets is still with us, even in this age of widespread FLOSS. Just look up how many software patents are still held. It's no good trumpeting your organization's advantages if everyone has the same advantages.
I also dislike like the patterns it encourages or a the "run loads of infra!" culture that comes with it.
I think most businesses need to really evaluate this issue before blindly putting everything on AWS since it can really get pricey once you need good performance.
So my point is, all these AWS specific config languages really create an even more tight lock in with the platform, maybe it would be best to do it in a cloud agnostic way, which of course also comes with its own issues.
Even worse, organisations rarely do it even once they're up and running and spending large sums - and even then I have never seen the impetus for cost control ever come from within the technology department. It's finance that reins in the spending later, and usually by sticking with the same cloud supplier but limiting activity - so now you need to get someone to sign off on your infrastructure before you provision any of it - which is largely the problem the cloud was supposed to solve.
I've seen the pendulum swing the other way, too. At former large employers, I've had to cobble together (shadow IT) servers from discarded equipment because the process of getting cloud boxes allocated was so onerous.
Like every large software vendor, yes, it's moderately profitable to produce good software and sell it but it is much, much more profitable to abuse bad procurement practice.
Shadow IT is a sad state of affairs but at least it builds the skills and culture to economise on computing cost.
"Businesses need to evaluate before blindly using AWS" - Why is that seemingly logical sentence proven false so many times every day in so many companies?
In my opinion, the success of AWS is due to the way it interacts with corporate hierarchy. It allows every level to externalize some annoying part of their job in exchange for a little more money. Even Finance is happy because capital expenditures (that require more complex planning and tax handling) turn into operating expenses.
Eventually the board will wonder what happened to earnings per share, but by that time the project to migrate off amazon will be impossibly large and most people with the skills to run their own infrastructure will have retired.
I think eventually we will see a European native cloud provider spring up with prices that make sense locally the same way EC2 prices make sense in America, and the big three cloud providers' pricing for EU datacenters will be driven towards Hetzner's pricing.
Deployments I've written in Salt just use the Python API (https://docs.saltstack.com/en/latest/ref/clients/) to make changes to machines or infrastructure. That's the power of it, and unfortunately its terribly underutilized.
There are legitimate criticisms of Salt of course. There's a lot of moving parts, there's a lot of half-baked stuff, the crypto has been questioned, but the core idea is honestly pretty great and its always been a pleasure to work with.
Haven't bothered with Ansible (yet) as it didn't officially support states when I was evaluating options, and statefulness was my number one need. I can pick up languages fairly easy (I'm a software dev by trade), so different languages don't bother me, it was more about feature set. It is annoying, though, that it seems everyone treats Salt as forgettable. I wish I could find things like https://github.com/dev-sec for Salt, but they only have Ansible, Puppet and Chef versions.
I've seen small pieces of utilization like this that looked awesome. I totally agree with the sentiment.
> There are legitimate criticisms of Salt of course. There's a lot of moving parts, there's a lot of half-baked stuff, the crypto has been questioned, but the core idea is honestly pretty great and its always been a pleasure to work with.
I could deal with that, but I have had absolutely misery trying to stabilize Salt minions. I had constant issues with them crashing or just hanging. Complexity is manageable, I can work around the parts that aren't totally there yet, I can live with maybe insecure crypto, but I can't have any stage of the deployment pipeline be flaky. It's annoying for me because I have to fix it; it's annoying to the people that need to deployments because they can never tell if their pipeline will succeed or not, and they have to wait on me to fix it. It ends up building a lot of animosity towards the tool, and by association, to the team that introduced it.
I just use Ansible now. It's not nearly as powerful (the ability to drop into Python in Salt is really nice, and I find equivalent things difficult to write in Ansible), but it pretty much always works. The abstractions are messier, but at a certain point, the less powerful but stable solution just makes more sense.
I really hope Saltstack can turn around on their reliability, because I would love to use it again.
>But, I think people will try to push this further and integrate the infrastructure code inside the actual app. All of it will be managed, by the application itself, at runtime.
I would only adopt this approach if there is an abstraction layer in between, like k8s, or if it would be easy to create an equivalent implementation for a different cloud provider. Being locked into dependence on AWS isn't the worst thing, but I would like to keep my application independent of a proprietary environment.
This point goes by quickly but it's important. One of the biggest factors in my employer's relative reasonableness is IT's corporate network and engineering's production environment have nothing to do with each other. Separate teams, datacenters, stacks, and cultures.
We still have a group that will write a change management plan in Microsoft Word to work on the corporate finance Oracle instance during a scheduled downtime window. But I can release my new API endpoint to customers whenever I want (after code review) using a contemporary CI/CD workflow.
No it will not. Nice dream, but it's not gonna happen. Devops is here to stay, or worse, devops will eventually move into the "real" development. With proper languages, not what we have right now. Either programming languages or something like Dhall.
Heroku was and still is significantly easier to use than any of the tooling or services available. And it's also sufficient for a lot of businesses.
These things are just tools. If your team is skilled enough not to have a dedicated people that care about your infra that's fine.
The real meat of this argument is that in the future DevOps won't be managing infrastructure to hand off to application developers. It will be creating meta programs that are designed to be used by application developers to let them build their own infra securely and reliably. This is what the `awsx` package in pulumi is. A predefined set of best practices that can be used by the application developer to create.
As someone who's worked as various stages of sysadmin throughout the past 10 years, I see this (pulumi-like) as the future. And I agree with the article in general.
I would err to say that Pulumi is not and should not really be used for this, but I was just thinking of how annoying it is setup some open source projects in AWS (especially if they're made for it or any other specific env) and that having the Pulumi config, at least some basic modules, would make it so much easier to get started.
No, this is backwards. Ops in DevOps is more important than it ever has been. Devs honestly have the simplest job in the world: they just have to write some business logic and check out for the day. There's many other teams of people who then take over in order just to make sure that the app will actually work for customers, and will fix it when it stops working. The scope of work covered in QE, Architecture, Data/Privacy, InfoSec, etc teams is massive, and Ops aligns it all so the app works accordingly.
On "configuration languages": Pulumi won't ever take over because 1) some of it requires money and companies are cheapskates, 2) the licensing is not Free (part of #2), 3) there's already this other crap called Terraform that everyone is already using that doesn't have the problems of #1 & #2 and does most of what is needed for infrastructure. For the rest, there's not one thing that can manage it.
CFEngine was where I learned that Configuration Management is the 8th circle of hell. Puppet was slightly better, but now I am trying to burn down every Puppet install in the company whenever I can. Chef isn't any better. Configuration Management as a concept (such as it is up to this point, anyway) is just horrible.
It will take another 10 years, but eventually the industry will wake up to the fact that its current obsession with tooling isn't actually generating any more profit or product quality than a bunch of shell scripts would, and so maybe they should calm down a bit.
The only critical feature of orchestration tools is dependency management. Knowing what changes will impact what other changes and planning for them. Otherwise, a bunch of shell scripts really does work just as well. At that point they'll probably write some other tools, hopefully ones that separate the task of dependency management into a standard independent thing, and let other tools interoperate with it, so that we don't have to custom-code dependency management into anything that wants to use it.
I mean, Terraform is a genuinely crappy product to use, but we literally can't stop using it because of that one feature. Same goes for systems like K8s, where the design is an absolute mess, but nobody can get rid of it because they desperately need something to run a container on multiple hosts for them. And since they have that, they decide, fuck it, let's pile on a bunch of other bullshit while we're doing that and we'll lie to ourselves and say it's better now.
The important parts of modern DevOps are not technology. They are the practices, principles, values, and patterns that have proven to uplift everything else. DevOps is not about Ops, or Dev, or any other team.
Shame as I'm interested in the topic, been trying Pulumi at the moment
I have little experience with either, do you find another tool more reliable?
I guess all of these tools are dependent on the consistency of the cloud provider APIs they're working with
So for time being we are sticking with terragrun&terraform combo.
But the concept of pulumi looks interesting and we will likely re-evaluate in a year or so to see if it has moved in the right direction.
Massive accessibility fail. Font size is one of the few things that usually works fine out of the box with modern browsers across devices of all sizes - unless someone decides to do crazy stuff like defining the font size in relation to the viewport size.