Invent More, Toil Less (2016) [pdf]

The ultimate example of toil, which I'm sure Google has automated away at this point, is truncating old log files that would eventually take up all remaining disk space. Simply truncating the file does not solve the inherent problem, but buys you more time before you'll face the exact same situation again.

Review past decisions that led to the adoption of systems that create a lot of toil. Avoid that in the future. Change those systems or the incentives that led to them.

There's no secret or quick way, people have to make hard decisions.

The case studies in the first paper are worth reading, but essentially you make it clear to everyone that you are going to take a hit on your primary production output in order to restore capability:

> Policy analysis showed that escaping the capability trap necessarily meant performance would deteriorate before it could improve: While continuing to repair breakdowns, the organization has to invest additional resources in planned maintenance, training and part quality, raising costs. Most importantly,increasing planned maintenance reduces uptime in the short run because operable equipment must be taken off-line for the planned maintenance to be done. Only later, as the Reinvestment loop begins to work in the virtuous direction, does the breakdown rate drop.

The reason that it's called the capability trap is that nobody wants to accept things getting worse. Everyone wants the improvements to be (1) free and (2) monotonic. But once you get stuck in the trap it's (1) expensive and (2) initially backwards. You slow primary production to make improvements and all the skeptics can see is that the numbers are worse, and didn't you promise to improve it? But the way out means going backwards on the primary metrics before you can free up capacity to improve capability.

And it doesn't even need skeptics to be hard. Even with all the good intentions in the world, it's very hard to avoid the temptation to prioritise your primary output over everything else. Sure, we should improve our CI/CD ... once we ship this feature. Sure, we should automate our disaster recovery ... but put out this fire first. Yes, we should create systems to manage the fleet better ... but can't let you refuse anyone's requests, we have a business to run. Yes, of course we need reserve capacity to deal with uncertainty, but don't you dare run anything less than maximum utilisation ... and also don't you dare say we're at maximum utilisation when I give you more to do.

On one hand, good management should realize that automating repetitive tasks and paying down technical debt is how you add to a tech company's capital stock, and is the only reason why tech company valuations tend to go parabolic. If you're just trading money for labor and labor for customer's money, you have a consulting service business. These can be profitable, but you don't build an enduring asset this way, and the valuations that business owners usually think about when they decide to start a tech company are those that come from owning a monopoly asset that you can sell to multiple customers for virtually no additional cost.

On the other hand, the part that engineers are usually blind to is the business context that the company operates in. Tech moves fast. It's not uncommon for basically all of a company's founding assumptions to be obsolete 3 years later. New tech platforms are available; customers want different things; a new competitor has just demoed a killer feature. All of these have the possibility to render large swaths of a product's feature set obsolete. There's no sense investing engineering time in a codebase that's about to be killed anyway; just rack up the technical debt and declare bankruptcy. Worse, management often can't tell engineering about many of these external realities without killing the product: would you continue working for a company that said "Our competitive position is untenable. Pump out as many features before anyone external notices so we can sell the company" or "We need you to keep the lights on for our existing customers while this secret division over there builds the next generation of the product"?

It's often good to ask questions about the company's overall strategy and competitive position (and to do research on this yourself) before joining. While the owners usually won't tell you everything, it'll build trust if they can tell you some things. They should be able to think of these issues in terms of trade-offs: "What's technical debt?" is the wrong answer, but so is "We care deeply about technical debt and give our engineers as much time as they want for code cleanup" (the latter is a bullshit answer, designed to make the hire). Similarly, it builds trust if engineers can also recognize the business tradeoffs and accept that sometimes the right answer is to hack something together and ship it so the customers can get value while management figures out the next strategic move.

Much like the type of fuel injection used in a combustion engine is irrelevant to most people that use their car.

If you are managing a hauling company and a brand type of fuel allows you to save 20%, you can see the value but you also know that it means updating your fleet, infra and all the costs and hassle associated with that. Which means that while you are keen on making those savings, you probably will not roll out fleet update ASAP.

I think this is how in companies which are not pure IT players, management might see those situations. Yes there is some value in reducing the toil but the perceived costs are too high for them to be fully on board, and I guess it's because their perspective is too rooted in the physical world (i.e my example).

Not that there are no costs when automating toil management in IT, it just doesn't map 1 to 1 to the real world equivalents.