The Failed Commodification of Technical Work

I remember when a mail merge literally meant printing out lots of address labels and then manually sticking them onto a letter & envelope. Word 2.0 (?) solved that problem for the 1990s and MailChimp has commodified it for the 21st century.

Double-entry book-keeping was technical work and was usually run by highly trained individuals. Nowadays every shop keeper just scans a barcode and has the customer tap-to-pay.

There's not yet a drag-and-drop like interface for anything more complex than Scratch (wither Visual Basic!) but the hard part isn't the technical work of stringing together libraries; it's requirements gathering.

Speaking of which, it has never been easier to drop in a high-quality cryptographic library, or import an interactive map on a website, or WYSIWYG edit a website.

So, the author is right that you can't stick a bored 18 year old in front of an IDE and have them create you an ERP. But a lot of the "grunt work" of IT is now firmly a commodity.

However, having read the Phoenix Project twice and hating most of Scrum, I disagree that’s what the Phoenix Project is advocating for.

My main takeaways from the PP are:

1. Have clear systems in place to carry out and manage your repeatable work, automate where possible

2. Minimise the time work is in progress for so people aren’t bogged down with a million tasks

3. Share information widely and have multiple members of the team able to carry out the same task

4. Make sure the work being done is what the business actually needs doing

5. Reduce noise and unplanned work so staff can get on with the higher value work they actually enjoy rather than wading through a quagmire of disorganised chaos

The point of PP isn’t to turn people into interchangeable automatons - it is to put a system in place to allow people the headspace and time to do the really valuable work that can’t be automated or systematised.

I’ve run a factory and been a dev so I see it from both sides and making devs production factory workers isn’t sensible but likewise where work looks like factory work (known work, repeatable steps etc) it should be treated in a similar way.

I think the author is talking about "commoditization", eg genericizing tech work so that any replaceable employee can do it.

From https://en.wikipedia.org/wiki/Commoditization:

> This is not to be confused with commodification, which is the concept of objects or services being assigned an exchange value which they did not previously possess by their being produced and presented for sale, as opposed to personal use. One way to summarize the difference is that commoditization is about proprietary things becoming generic, whereas commodification is about nonsaleable things becoming saleable. In social sciences, particularly anthropology, the term is used interchangeably with commodification to describe the process of making commodities out of anything that was not available for trade previously.

Am I being pedantic? I thought the two had different meanings?

Edit: Ah, but wait... from Wiktionary instead: https://en.wiktionary.org/wiki/commodification

> Sometimes used interchangeably with commodification

Guess it's just a common mixup.

Programming isn't work, it's meta work, you come up with a list of instructions once, and then the computer does the work 24/7 indefinitely. Meanwhile you go on to write another set of instructions for something else. If you ever write the same set of instructions more than once, you're basically doing it wrong. So it's hard to know how long things will take, because you're always doing something new. You never do something more than once.

The misconception is that building systems to tackle new problems are easy and thus have been “commodified” meaning nobody needs to write code anymore.

The reality is that building software is rarely as easy as configuring a UI. You end up needing text which represents logical rules and flows, you need version control see how the system changes, rollbacks… which means you need programmers

Coding doesn’t disappear, it just moves up the levels of abstractions

But there’s a sort of essential quality that reasserts itself no matter what you abstract. Despite the seemingly simple requirements paired with high level frameworks, a lot of our software still doesn’t even work well.

As the author notes. The only real fix is talented devs that care.

You can make a career out of that.

Operation, innovation and maintenance. Pick any 3.

Better if done by the same team or teams that are very close to each other.

Ideally, all done to varying degrees by each person on the team because that’s where inspiration for improvement comes from.

We can split the 3 functions up into 3 groups, with 3 sets of management hierarchy. Often the result is mediocre and the people are unhappy (especially those stuck on the maintenance team - thankless but crucial work, that).

We can try to make teams responsible for all 3 functions, but often hire managers who overfit on one of them (coincidentally the one that leads to better rewards next quarter). It’s hard to champion operational excellence, diligent maintenance and hammock time in a single culture.

No wonder it’s hard.

Its got some problems:

1. They forget you need to hire many more (lower paid) people, because your output now linearly scales. Human hands have to turn the crank because its all UI-based work.

2. You still end up with very complex and disorganized business logic transform code, and now its buried in the UI. The PMs or business teams are the only ones who know what that logic is. The engineering org delivers high quality, tested datasets that are pretty raw for the purpose of answering business team questions.

The hard part was always solving the weird way business outputs are obtained from raw upstream data. Now that solution gets stored in a tableau workbook and cant be used as input for something else. It has to be copy pasted from the UI into a new tableau workbook. Well, now we bought the Tableau cloud service and our BI team can build and maintain SQL extracts in a more rigorous way. Tableau now looks like its trying to take a chunk of Databricks business here, but now its a non-engineering team doing it.

Not sure its going to work out.

Thinking about software as a factory is also possible and might be useful. But not everything fits into that analogy, for instance, when you think about integrations in the product or when it is a service rather than an app. Factory implies something is being made but software is not the end result in many cases but is an enabler in itself.

Consider the humble file copy. Or “automated scribe” if you will.

Copying is automated to the point where the enormous amount of copying our systems do has becomes invisible to us - and also lost as a point of economic differentiation.

And it’s meta useful! All our copying programs are themselves easily copied - with the same algorithms!

The point is that software writing, like mathematics, will always spend its time futzing around the edge of the known and unknown, because every area that gets fully known/characterized will get automated in a way that both solves that problem for everyone and scorches the economic earth of that activity.

Software work will always include an element of quest (research) into the unknown, in search of riches (economic value), in some aspect, whether that new area is glorious or tragically mundane.

Whatever area of software work a manager without software expertise can do themselves automagically, is like a fruit tree tamed so that it is easy to pluck fruit from, because it no longer has fruit.

The manager would no longer be doing anything differentiated or valuable if their problem statement wasn’t upgraded back into difficult to automate territory requiring pesky creativity and expertise.

Software dev is still at the craftsman[0] level. It might move out of that, eventually. But not yet, and probably not in the next 20 years or so. We haven't solved some intrinsic problems around defining a problem completely, precisely and succinctly without having to write code[1]. And getting five engineers to write a single piece of software is exactly as complex as it was when Fred Brooks wrote about it, I think the only improvement we've had since then is Git.

[0] craftsperson? that doesn't feel like the right gender-neutral expression. I guess "artisanal" but that looks rude. Suggestions?

[1] The "I got ChatGPT to write this application without writing a single line of code" phenomenon is interesting, but it seems like an alternate skill path - you can write code, or you can write prompts. The complexity is the same, and the amount of effort is within an order of magnitude. I'm not sure, though - I haven't managed to get ChatGPT to solve a single technical problem successfully yet.

Nice idea by maybe software engineering wasn't/isn't that formulaic as more well-established engineering disciplines.

You are fundamentally building items , not reproducing them in software. A factory reproduces the same items.

There is a crisis in industry where legal contracts and business forecasts do not align with what is reasonable and predictable for software development .

I recently wrote a post "Will Artificial Intelligence Replace Radiologists": https://anordinarydoctor.substack.com/p/will-artificial-inte... that explains why I think the answer is "Yes, when it replaces engineers and everybody else".

And that is the trouble with software in a nutshell.

The assumptions are too many and the subjective opinions too deep.

I rarely submit to assumptions any longer without the other person submitting at least some evidence for their claims.

Experience is important but gathering information on everyone's experience is more important to form some sort of experience based evidence.

I know everything everyone writes is well meant but damn I miss someone wrote a book and made a course on how professionalism shoukd ne based on evidence gathered from our fields experience.

The amount of managers incapable of commitment towards serving towards that are just what fred brooks said it was - they rarely exist.

CEOs think they are buying an appliance. e.g., I buy a coffee maker, I pour water into it, I put in the coffee pod, I press the button, and coffee comes out. In the best scenario (from a CEO perspective), they are getting an assembly line someone else put together for them. e.g., the CEO hires a general contractor who lines up all the subcontractors, installs the different pieces of equipment including the "glue" that connects them, and will maintain equipment going forward. The CEO simply needs to pay for the capital expense and provide the operators for the assembly line. In the typical scenario (bad from a CEO perspective and bad for the software industry), what the CEO gets is a stack of re-purposed software modules that have been pressed together in a slip-shod fashion that works when it wants to and fails without a trace as no logging exists. The CEO has to hire IT developers just to get the thing to run. And he will now look for his next job because the board of directors told him, you failed.

Software development to me is a financial investment strategy. Some risky investments, some safe bonds, some good debt to invest, some bad credit card debt that will be cleared later.

To me, these are the same as a risky overhaul of a critical system adding some automated tests, skipping some type annotations, or launching what was supposed to be a demo/MVP to be first to market.

a piece of software that took months to build could be worthless tomorrow if a competitor comes out with a better solution. There's a time value and strategy to it.

speeding up development with CD lowers the investment cost & time to market of an individual strategy. Automated tests lowers risk. Feature flag & AB testing diversifies the portfolio.

One line of code be the most valuable piece of a product, like Doom's Fast Inverse square root. and millions of lines of code could be worthless.

the value of a software engineer's work is extremely contextual. Two automation can be structurally the same, developed by the same person, written in the same language, take the same amount of time and resources; and still be COMPLETELY different in value.

until IT/software strategies are treated like an investment strategy, companies & leadership will continue to flounder when managing technical teams.

https://govleaders.org/rickover.htm

The counter-example of a software to abstract SQL queries is weird. This is exactly what we have been doing with other levels of abstraction, happily so. Why write Python instead of just using a compiled language? Because it's easier, and allows you to hire different types of people and focus on higher order problems. Maybe that's offensive if you are a world-class programmer like the author seems to think he is.

Delightful.

> Well, you're wrong, and you can fucking bite me.

Hear, hear!

I've lately found a middle ground that seems to work well - licensing source code or libraries instead of entire closed web solutions. It enables a small team of good engineers to be really productive and ship something to production rather quickly.

You pay for maintenance and bugfixes for what you license and can still retain control of your data and interfaces.

(Note: I'm using the term "manager" broadly. It could be an executive, a team lead, an entrepreneur... basically any contributor making choices.)

I personally love being around people who are willing to get dirt under their finger nails, smell the soil, ask questions, and read the documentation (ie, willing to step into complex systems). It's inspiring. It also is a good indicator that the environment I'm in is being well-cared for.

Think of a human caretaker – those with empathy (a beautiful example of willingness to engage complexity) far surpass others in both ability and impact. Think of a barista observing details like humidity while adjusting their process. And now keep this analysis while moving to higher levels of abstraction.

(BTW, this is a phenomenal essay. I love how the author left it to the reader to realize all the lessons contained in the McDonald's anecdote.)

The image of a Burgermaster 5000 operated by a Highschool kid producing burgers is one that really gets stuck in your head.

I am wondering to which degree something like this might or might not come true with the advent of AI driven coding.

I have built a full swift app using Ai (not writing a single line of code) but it took a lot of coding knowledge to pull it off. I wonder if this will change.

It's the business version of the get-rich-quick scam course hucksters. The truth that there's no silver bullets can't compete.

I’m still very early on in my career in software dev (~3 years in) and you articulated the idea I’ve been grappling with extremely well in this piece. I could never find the word for it but “commodification of technical work” is the perfect way describe it.

I've been in tech long enough to remember when you had to carve literally every line of code out of the firmament of the heavens to get anything done. Today the bulk of technical work is plumbing together literally millions of lines of "free" (as in freedom and beer) code to get some behavior that checks all the requirements boxes.

Going from the post some more, modern developers are more akin to line cooks that put ingredients together. There's value to that skill, but it's a reduced value as its easier to learn to plug together APIs than to craft a cache aware data structure and reduce the big-O of some algorithm while fitting it into limited RAM.

In most restaurants line cooks are almost a commodity (even the sous and executive chefs are to a point), customers don't know if the cook from yesterday quit in a rage and a new guy got picked up this morning who had also quit in a rage from the place next door. Follow the recipe and you get the same product to the customer.

News for the tech folks here, line cooks (even experienced ones) make shockingly less than the average fell off a bus tech-bro with a couple bootcamps and a github repo.

The average developer is today thousands of times more productive than the developers from 30 years ago because they don't have to build the ingredients anymore -- the industry is well bootstrapped by now. But if some future Co-Pilot LLM can turn one developer into 4, and effectively "auto-plumb" with minimal human direction then...

In my experience this was building a process for data scientists (math PhDs) to train and deploy ML models, and currently chemical engineers to build and deploy process simulations.

Data scientists and chemical engineers will have to excuse my comparing their work to flipping burgers :)

In the decades since, the 'progress' we've made can be summed up as dropping the J in JVM.

My bet agrees with the author: I am betting that I can spend time and brainpower designing and implementing a good solution to a problem, and people will pay enough to support me and my wife.

You can't commoditize design, and that's where my work actually is. This is also why I'm not scared of AI taking my job.

A founder so dumb he cannot understand the basics of one app equals one domain...wanted to fill each domain he owned with a part of a major app and then sell each domain and not understanding that would destroy the full app if we could ever break the one app one domain equation in the first place!

But to the author's point, I remember checking out a book from 1980 on software engineering from a highly well-known author (I forget who). I was shocked to read the author state unsarcastically that software development labor would be fully itemized in a standard catalogue like auto mechanic by the year 2000.

Then the acquisition came, hordes of new C-Levels and directors added, and suddenly, the system everyone wanted was no longer enough; we needed more out of the Support/RND team for some reason, more sales, more renewals, more special contracts with ENT clients with exhausting demands from the company. And the expectation was just "be more efficient". All of this came over slow time with small changes (back porting features for ENT clients who refused to upgrade, forbidden solutions for the Support team because "it upset the major clients", allowing sales/renewals to force RND engagement if they demanded it, new CRM because it was too hard to do marketing campaigns in the old one, even though people were calling _us_ and asking to just send them a quote they liked the product so much)

I honestly think trying to commodify and extract even more from what was a very successful system financially and just overall ended up ruining pretty much everything. Sales are slumping as are renewals, we're churning RND and Support folk, and because of slump in sales, there's belt-tightening everywhere.

We've implemented so many new systems needlessly with huge implementation/consultant fees that absolutely no one knows how to use -- workday is a prime example of this as out of the box it doesn't do _anything_ we needed, but we had to use it anyways, and absolutely no way to pay for the features we really needed. Same with ServiceNow implementation, it was supposed to be a near 0-code experience we were sold, but naturally that was not the case. Why did we get it? No idea except that it came down from on-high from persons that don't use the CRM and now we're stuck with it.

For me what it comes down to is too many people having to be like the CEO and Bill from the article -- for some reason such Clevels need to show they're doing "something", but I didn't think that something should include implementing wild changes to workflows in the company the C's know nothing about, or even worse, responding to customer complaints and demanding we "fix the issue" without knowing what the issue is.

The conference experience from the article resonates with me heavily, cause we have all these systems that do "everything but nothing", all these workflow changes without listening to the people having to use the work, it's really awful.

In reality most of the world operates somewhere between “we’re a factory” and “we are unmeasurable Picassos.”

It’s not a huge stretch to combine thinking that “software is unpredictable” and “people pay up for predictability” so we should still try to tilt that way.

Consider the evolution of human weapons. It was very easy to make a crude throwing spear (break off a branch and sharpen the wooden tip against a rock) but a ton of strength and precision was necessary to take down a target with it.

It took a while to figure out how to make a decent bow, but once invented it spread all over the world because momentum could be imparted to the arrow instantaneously when releasing the string. Strength was still required to hold the bow at full tension; this requirement was eliminated with the crossbow. As a tradeoff, the crossbow required more significant manufacturing expertise and economies of scale, particularly with regard to precise metal locking mechanisms.

The evolution continues with guns replacing crossbows, tanks replacing horses, and ending with nuclear weapons. Each technological development requires a much greater level of manufacturing organization, and in turn bestows more power on an unskilled user. The resulting economies of scale are why only a small set of huge states are militarily relevant today, and why the idea of nomad bands posing a threat suddenly seems laughable (whereas it was a dominant worry for civilized society up until a few hundred years ago).

It is natural to glorify the work of the artisan whose disorganized but brilliant insight has not yet been commoditized. But for better or worse, progress usually takes the form of painstakingly systematizing those insights, detail by detail, until they are reproducible and robust, so that a broader range of people can consistently churn out similarly good work. It is this systematization and rationalization of the manufacturing process that really brings quality of life improvements to large numbers of people. It is also what can seem to take the magic out of the original insight, eg as Edison’s discovery of electricity becomes as mundane as flipping on a light switch. Progress happens in the transition from magic to technology.

> For example, one pitch in particular was for a product which promised to remove the need for me to write SQL in exchange for being able to set up all my dependencies from a drag-and-drop editor, with the sales pitch consisting of "You can get rid of thousands of lines of all that SQL you hate!" - no I can't, fucko, because your application is still connecting to Postgres so it's just writing the SQL for me with another layer of licensed abstraction on top of it. Why would I pay to have more abstractions designed for you to sell software to multiple clients, you blue-suited dementor? Eight times out of ten, I want to pay you to remove them from my codebase.

See, the problem is that you can't fully remove the original SQL. It's still there, in the code. The database is still there. So instead of having one interface to access the database - SQL - now you have two, the low-level SQL and this bastardized abstraction. Some of your employees will write SQL, and some other of your employees will use the abstraction. They use different tools, so the tools will fall out of sync, and the employees will fall out of sync, and you'll get discord.

Pick one tool. As far as it's technically feasible, pick one database, one programming language, one UI framework, one wiki vendor, one CRM, one ops visualization framework that is right for the business. Don't pick up some fad-of-the-day, and don't let your "artisan" engineers do "research" projects to see whether some other framework might suit their needs "better". Tell your engineers, here's the business problem, here's the pre-existing stack, solve the problem with the already-existing and already-supported tools. If the engineers are truly artisans - guess what, an artisan doesn't blame his tools. Pick new tools only slowly, deliberately, when you have no other choice, and only with a solid plan for standardizing the tool for full adoption across the enterprise.

Why? Because high-performing organizations ensure that they are always building upon prior work. Five engineers with five years' experience with React will be a stronger team - more productive, faster, more accurate at forecasting, have an easier time reviewing and supporting each others' code - than will John with 5 years React experience, Sally with 5 years Vue experience, Wang with 5 years Svelte experience, Cynthia with 5 years Angular experience, and Ilya with 5 years jQuery experience. If Ilya gets hit by a bus? The other engineers will have difficulty picking up the load. Not much of that React, Vue, Svelte, Angular experience will help with supporting the jQuery parts of the codebase.

Commodification requires a standard. We appreciate the value of standards when we succeed at establishing them - tabs vs. spaces, Docker containers, infrastructure providers, project management tooling - because they eliminate discussions that do not revolve around the more fundamental question of how to deliver value to users. Shouldn't you ask yourself, if you're evaluating a new language, tool, or framework, whether you really benefit from breaking the company standard?

There's still no better training for programming than the equivalent of master/journeyman/apprentice. Apologies for the gender specific terms, but they are specific to how tradespeople operated from medieval times.

The worst thing to ever happen to the practice of business is the invention of the MBA. MBAs are imbued with the misleading axiom that management is a craft and science of its own, independent of the type of process or practice that is being managed.

Combined with endless selling of the latest buzzword theories by consultants is why we end up with JIRA-Driven-Development, nonsense like t-shirt sizes, 2 hour wankfests called "Sprint Reviews", let alone all the scrumming and standing-up and backlog-refining and endless make work.