Johnathan Blow on Software and Preventing the Collapse of Civilization

I am a fan of this talk. Working on the sysadmin/devops/sre side of enterprises, I originally found the transition from manual installs to things like puppet and then to docker containers and surrounding tooling refreshing. Building your own services, monitoring error budgets, and all the other principles like infrastructure as code seemed to be “the way”. I have recently become severely disillusioned. On the development side, one need only spend time in the JavaScript ecosystem to get a taste of some of the worst of the baggage we are building for ourselves there. You could spend more time learning toolchains than writing code. On the infrastructure side, it seems we never bother to deploy even a dead simple web service without a full build out of a multi-region Kubernetes setup with ci/cd services, secrets services, auth services, logging, monitoring, pipelines for the data stores and image builds, networking, etc.

At a certain scale these things seem necessary. But I’ve had the most success actually getting things done with small, competent teams where I replaced many different tools with as few as possible. For example I recently turned a complex spinnaker setup into a set of bash scripts that perform helm installs. It makes debugging, knowledge share, and development of the “system” so much easier. Why was the choice made originally, what features does it lack, there are tradeoffs certainly. But I feel we as an industry are losing the thread somewhere.

I often see people holding up StackOverflow and their few servers model as a bastion of sanity. I see them say, “I could run your whole company’s SaaS stack on a single box with systemd”. While I find it hyperbolic, more and more I am considering this not just a casual topic but a vital one.

Sometimes I find people asking - what about disaster recovery, high availability, scalability? When what we need is to slow down and ask, does it work? Is it usable? Or we’re so focused on isolation, deploying security tooling, the systems get complex, and no one understands them to the point they seem much less secure.

Simplicity seems key. Sticking to as few tools as possible, at lower levels, and only growing or adding as really well thought out requirements arise. This isn’t exactly what Johnathan Blow was talking about (abstraction and losing knowledge) but feels strongly related to me.