The heaviness of maintaining systems

> Operations always felt like “the things around the Thing” - a supporting cast that is important to making something happen.

In the short term, it can feel that way.

But in the longer term, at least in the part of the industry I'm in, operations are the core thing. Everybody I know who has become good at building resilient and robust systems that survive contact with the real world has a lot of experience understanding and fixing broken systems. A lot of humility of seeing good plans go wrong, and good ideas turn into bad ideas. The best have all this experience and still deep optimism about what's possible, about what can be built next, and about being able to fix the problem this time.

> How do ops people thrive and grow as those who take care of the systems around us, without letting the systems consume us?

An excellent question. In my career, I've optimized for working with people I can trust, and people I know who care as much as I do. In the short term, that can be stressful, because people express caring in different ways that aren't always productive. In the long term, its all worth it. The times in my career (careers?) I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. That's exhausting.

> In what ways are the systems we maintain mirrors of ourselves?

Another excellent question.

Maintenance is problematic if things aren't built with maintenance in mind, this is true for IT projects, but it is true for other things as well. If you barely manage to make your thing work with the provided resources, how much thought are you really going to spend on the maintenance?

Especially in the IT sector many people never have to maintain the things they built themselves. There is always a shiny new project and off to greener pastures.

Part of this has to do with learning on the job. You build a thing and once it is done one very easily could have a good idea how to build the same thing, but better. Part of it is scatter brain. The IT guy is interested in toying around with a project and being the hero that solves it. But once that is done the thrill is gone, shall others deal with that.

I try really hard to build things for maintenance, because I tend to be the person who maintains it myself and I try to keep my own bus factor low. A lot of that has to do with a wise choice of dependencies (less=better), but also with good documentation. I try to write the code in a way it doesn't require a lot of documentation. This means consistency, avoid being clever, always asking myself how one would expect a thing to work and then build it that way etc.

I have only rarely been a part of teams that actually valued its ops/infrastructure/platform teams. I think the biggest factors affecting burnout for me are long, inflexible on-call hours, and the fact that a huge majority of companies see ops as a cost to be reduced/minimized as much as possible, not a core part of the business. I've seen this attitude even when the platform itself is a product. Work done by ops is frequently invisible when it's going well, and extremely visible when it's not. This phenomenon also leads to patterns in ops infrastructure teams where it's reaction driven, and not preventative - it looks way better to management teams to be the one to swoop in at 3 am to save the day rather than preventing the outage in the first place, which can be difficult to quantify.

I am reminded of this classic: https://web.archive.org/web/20190713163433/https://medium.co...

> While a certain minimum of capability is required to do your day-to-day work, what your value usually consists of is in grinding yourself against the piercing pincers of elusive bugs and razor-wire bundles of bullshit code until something resembling progress is made. You are rarely a problem-solver, but rather a problem-endurer. >

The sad reality for me is that most of my value is in my willingness to do glue work which I find so many others lacking the fortitude to participate in.

Ops is not peripheral to the job, ops is an essential part of the job.

The job is always to make that computer box there do X when I the paying customer click Y button. On the inside there is some algorithm design and logic that tells the computer how to do the job, and also there is some stage setting and lighting and camera setup to cue the computer on what, when, where, and why do the job (and write down who told it to do it).

In terms of classic 'developer writes features then leaves it to ops to actually make the code be useful to customers' (AKA specialization inside the software realm), the division of front-line soldiers vs logistics comes to mind here.

If you don't have a front line, the enemy takes the land and you lose. If you don't have logistics, you lose the front line to starvation / lack of ammo, and then you lose the front line, then you lose the war.

So if you are in strict operations, you may be the logistics side of that, and the front-liners may look down on you for not doing 'the thing', but if you weren't there to do your job, then they get to starve, run out of ammo, or do your job as well (aka becoming more generalist / end to end ownership / full stack SRE) as their previous job.

Reader: Ok smart guy, what's your solution?

I advocate to stop splitting the job of dev and ops into separate roles and doing the devops practices for everybody. We lose theorized competitive advantage but gain flexibility and more well-rounded skill sets so we really can call ourselves software engineers.

If you can code algorithms but not properly manage the lifecycle of your code after entering it into the source code file (testing, deployment, runtime, data migration, upgrades, decomissioning), you only have half the toolkit.

If you can do one or more of the other pieces but not do reasonable data structure and algorithm design relevant to the problem space, you only have half the toolkit.

We don't have residential plumbers arbitrarily split between vertical pipes and horizontal pipes...all the pipes are needed, so the plumber learns to handle both. Why did we let the 'expensive computer operators + cheap clerical roles' of the punch card era split up our profession today into 'devs' vs 'ops'?

Generally I’ve seen a big difference in engineering output between those who see their system as a product, with users, and as a reflection of themselves in its ability to perform well, including under load and at scale, vs those who see it as a challenge to product a program that does X.

The product builders tend to understand that no matter how smart the implementation, not functioning means it all counts for nothing.

At the end of the day it has to run, and the ops matters. This means those product oriented engineers build systems that actually scale and perform their purpose well.

It’s a spectrum, not binary. But to a large extent stops it being heavy when performance and availability are seen as worthy features.

And at some point in every org the engineers who keep things running are valued because it’s actually challenging to keep things running, and things running powers the business. Over any significant timeline Ops always matters.

I started, and essentially always did, ops work. It's a lot of grunt work, but it pays off (if anything because there are very few people actually good ad that, and even less people good and willing to do that) and that gives a lot of learnings that are almost directly applicable outside of work.

The first being: if something is not work, speak out and push for the thing to either be fixed or be ripped out entirely, propose an alternative or a solution, if possible. If you see promises that aren't kept, words and words circling about the issue, minimisation of the issues our outright denial of the issue... Just get out, walk away and don't look back.

The above work as well with companies as with people (either friends or significant others).

And in the house... I very often realize i opt for stuff that's low maintenance instead of shiny (99% of the time i don't need the extra features anyway). I bought a lot of "dumb" technology and it's awesome, I get none of the annoyances people complain about nowadays. I particularly love my dumb TV (that I don't watch much anyway). Stuff that mostly does one thing and it's trivial to replace.

Oh and I've learnt to have spare stuff because I know from experience that things break when you need it the most (and stores are closed). So batteries, an extra shampoo, a spare laptop etc...

Ops is truly grunt work but I'm truly happy to have been shaped by it

(I worry a lot about the consequences of my own actions, though).

Sometimes leadership is the simple and infinitely fractal effort of just making it happen.

This is basically a work-life balance question. Over do-it in either category and the synergy is off. A long career in software engineering is by definition more marathon than sprint. Having said that, … we all experience times of chaos and crunch-time / death-march stuff. It’s up to you to decide if the balance is off. Once in a while is ok, but is it the norm?

Apropos nothing: I HATE endnotes (aka footnotes). If you have something to say, don't break the flow. If you want to add a comment, use parenthesis. If you're not writing a scholarly article, don't use footnotes. They're an indulgence that detracts.

> I’ve tried to start reflecting on what forces are motivating me when I take on extra work, or jump into an incident, or engage in glue work. After that, though: what’s next?

I just recently completed an Reiss[0] assessment, a scientifically proven approach to understanding motivation and through this process, I was better able to understand what drives me to make the decisions I do. Biggest lightbulb moment for me was that ALL of us are motivated by the same 16 basic desires (e.g. acceptance, beauty, order, tranquility) BUT at VARYING degrees. How much we are motivated to satisfy those desires depends on both genetics and environment.

So, now, before jumping into incidents, I find it helpful to check in with myself on what the underlying motivation is.

[0] - https://www.reissmotivationprofile.com/

Big shout out to the kind people keeping things running on the internet.

Thank you.

> Is there a safe way to perform this work for a lifetime and not burn out?

Yes, if you destroy capitalism and wash away this weird bourgeois conception of having a "career", so you can actually do work that cares for other people....

This feels like self-pity. One can easily turn this argument on its head. What if operations people are the once that enable a bad situation to persist? As someone who prefers to create new things over maintaining old ones, I sometimes think that is what operations is about, a codependent relation between shitty software and high functioning depressives, but then I remind myself not to psychoanalyse everything and move on.