The Wetware Crisis: The Dead Sea Effect (2008)
brucefwebster.com
brucefwebster.com
A single talented carpenter with an energetic but otherwise totally unskilled helper can accomplish more in a shift than two talented carpenters together. If you don't have someone to run to the truck for nails, hold shit, move ladders, etc. productivity suffers because your talent is wasting time on menial tasks.
Same thing applies to IT departments. The folks the author is busy sneering at are at least as valuable as the engineers they're actively trying to retain because (among other things) they soak menial tasks that would otherwise cut into their rockstar's productivity.
I actually have no idea what working construction is like, but I imagine it’s hell. I hear about jackhammering causing permanent damage to people’s nerves and all sorts of random accidents.
Automation is so easy in software development that your analogy to construction becomes pretty weak—we don't need someone moving around ladders because when we realize that's a need we just write a script for it.
So there are menial tasks, but menial like “yeah just redraft this architectural drawing for me mate, to take account of the new 2021 fire regs, I got actual brainy stuff to get on with”.
Pairing senior and junior devs works. But not because the junior is an idiot, but because they have stuff to learn and raw talent and can figure stiff out. They probably already know how to code well from their passion or maybe bootcamps or interning. Not from university usually unless there is lot of coursework.
I am not saying construction workers are idiots. But I am saying the OP article is implying the worst employees are left by this dead sea effect. Are you saying you can absorb untalented people on work sites. In dev teams they slow things down it would be better to pay some of them garden leave.
(were one to have suitable lame duck tasks, how would wetware lame ducks compete in an LLM world?)
Run to the truck and get nails also requires explanation/experience to get right if you have never done it before — what nails?which truck?how many?etc.
It is completely the same thing. The first time around you might have to explain the menial tasks in detail - btw a good exercise anyway, since you need to be able to explain aka understand the task anyway, after all you need to review later. And the first time around there will be some mistakes or misunderstandings.
But it gets better over time. With everyone.
Remember, for us the machine already plays the gofer role: handling menial tasks.
After retiring from IT I volunteered at a raptor conservation centre. My previous experience counted for nothing and I was given numerous menial tasks: clean that water bowl, chop down that overhanging branch, sweep out that pond, re-attach that broken perch, put a trailer on the ATV and take this rubbish to the dump area, clean that aviary back wall, operate the music for the 11.30 display, etc.
After doing this for a while I had great experience of how the place actually ran, where things were kept, when it was safe to do things without interrupting flying, etc. I knew I was getting somewhere when my tasks started being looking after work-experience students, inducting new volunteers, helping in the displays, etc. It was actually quite humbling being a complete newbie again and having to go up new learning curves. I wish I had done it sooner in terms of resetting my objectives of what work actually involved.
You say there are some codebases that need over 9000 IQ to do anything. While that might be true, it is not the norm and a small fraction, so it’s not relevant to the broader argument.
You say it’s completely different in programming. Sure it doesn’t help if one of your workers is a complete idiot — but that was not the premise - it was unskilled but energetic.
And there are always menial tasks to do — you pointed out some example yourself.
So to sum up my points are: - most codebases do not require gigabrain for every single change
- there are always menial tasks to be done in a team environment
- “untalented” people don’t necessarily slow things down in development
- in fact talent is irrelevant when writing code - I would even argue that being talented is not always an advantage, in any field or profession - but that last point probably would need some more explanation.
Example 1: I got hired to work in the dev department at a large regional printing company. One of the suites of software built in-house took large retail clients orders for printed store display material. From the client's initial order the software would automatically detail print batch sizes, assign these to the work queue, and would then go on to figure out how many of each type of display needed to be boxed and shipped to each of the client's retail locations and would create shipping labels, packing lists, etc. from this. At it's heart was a 10 page long SQL query that I couldn't make heads or tails of after two weeks spent studying it. I left the company certain of two things: 1. the dev that wrote that was one of the 3 smartest people I've ever met and 2. when he left the company they had literally zero chance of ever hiring someone to maintain that system.
Example 2: I got hired by the 2nd largest print media conglomerate in North America to help manage a fleet of 31 newspaper websites and a fleet of 125 niche verticals. This was at a time when the industry was just beginning to come to terms with their revenue models being gutted by craigslist and the shift to online marketing. To say the least money was tight. It was decided that the back-end systems that drove the integration between newsroom terminals, printroom layout qeues, and the newspapers' websites needed updating.
They hired #2 on my list of top 3 most intelligent people I've ever met to drive a complete overhaul of the system. It was decided a complete rewrite in a cutting-edge language was called for. The guy in charge didn't sleep for 3 days after which he'd provably onboarded and synthesized a complete understanding of the chosen language and it's ecosystem despite no prior experience, then went on to single-handedly rearchitect (feature complete) a system with roughly the same LoC as an operating system. In six weeks.
Over the course of the next 18 months the dev department responsible for implementing this masterwork (no sarcasm) floundered and the project was eventually scrapped having blown through the entirety of it's original 2M budget with literally nothing of use or note to show for it. Apparently nobody else on the team was capable of implementing and debugging the blizzard of microservices the new architecture called for.
My key takeaways: proposals to make codebases "interesting" should be met with deep skepticism if not outright hostility, and a roomfull of mid-level developers are significantly less likely to get an organization into deep trouble than a single rockstar with the bit between their teeth.
And in the majority of cases, I would say that this is "accidental complexity" - i.e. it can and should be otherwise, and calling out the unnecessary cognitive load, is a benefit.
I agree so much. The "dead sea effect" or whatever is just a lack of knowledge transfer. This problem remains over time too, eg look at the use of Cobol in the banking sector. Newer talents dont tend to learn old stuff. So what else is left for management to do in the long term than document really good and train newbies. If you think "migration", consider the timing, when the lack of knowledge already becomes pressing.
1. The author said IT departments. This ostensibly includes everything from software development to systems admininstration, vendor management, and end-user support. Plenty of drudge work to be found there.
2. If your codebase requires someone to be on the edge of the bell curve of human intelligence to work on it is almost certain that mistakes have been made by very smart people.
3. The author is absolutely implying that the worst people are left, which is entirely my point. They have most likely never encountered the idea that managing folks to their strengths is more effective than trying to cram the room full of edge-of-the-bell-curve unicorns.
There's also value--sometimes hard to capture--in getting an external perspective on things. In a sense, they are a new user/customer for internal systems and processes, and can identify pain points that veterans have largely learned to ignore or bypass.
The users are actually amazing and seeing what happens is awe inspiring - in a bad way.
I'm usually one of the smarter people in the room when doing software development and spend a lot of time helping others. In construction, I'm the enthusiastic idiot as the skillset is different. Most tech workers underestimate how high skilled tradework and many other positions outside their experience are, especially management work.
Having said all that, I feel uniquely qualified to offer a rebuttal. The article is describing a clear pattern that I see at large companies that are not focused on software development or lack solid engineering leadership. Ironically, the problem gets worse the more a company tries to outsource IT as they face all the same challenges with less control. It is an organizational health issue however, not some universal truth. If tech workers lack anything, it's experience with organizational growth, change, and group dynamics, not construction experience. Organizations are like people, they grow, change, figure out who they are, have identity crises and need different things in different phases. Ask the highest level/most competent manager you have access to for book recommendations if you're interested in this.
Anyway, back on topic, software "gruntwork" typically implies a department lacks the agency to automate away said gruntwork or lacks the skills to do so. As an example, I work with many organizations using the JVM, but none of them use Scala, Kotlin, Clojure, or any other "nice" JVM language. They use Java. In some cases Java 8. If you're writing Java, there is absolutely stupid gruntwork. I've written example applications for some of these organizations and I had to create code generation tools to stay sane in this environment. In software, you eliminate stupid gruntwork with tools, not people.
We do need average enthusiasts in software development, but it's not the same as construction. In construction the less talented person fetches things and does setup work. In software development, the less talented workers spend most of their time using libraries, plugins, frameworks, compilers, interpreters, databases, and languages while the more talented workers write them.
My experience isn't universal, and I'd be interested in hearing some dissenting opinions on this.
Absolutely, (in particular) MEP (sheetmetal workers, pipefitters, electricians, plumbers) tradespeople and project managers are highly skilled/knowledgeable and the work requires a lot of coordination. You basically have to understand how an entire building/space is constructed from start to finish to properly plan and schedule a project.
For the record, plenty of other trades are highly skilled, but their trade/craft is more specialized/focused and doesn’t require labor from start to finish.
This was modeled in the essay after "surgical team", with the most senior programmer being the "head surgeon" leading the operation, his direct assistants who could have major tasks delegated, and the juniors who could take on simple but still important tasks. (anaesthesiologist would probably be example of the cross-cutting support task)
> A single talented carpenter with an energetic but otherwise totally unskilled
> helper can accomplish more in a shift than two talented carpenters together.
> If you don't have someone to run to the truck for nails, hold shit, move
> ladders, etc. productivity suffers because your talent is wasting time on
> menial tasks.
Joking aside -- if you find yourself and your team spending too much time on menial tasks, you need to focus more on automating them. Of course, how much can be automated depends on how much hardware is involved in your work; as long as we're in software world, pretty much anything can be automated.
E.g. they won't do anything productive until CI and dev setup are "idiot-proof" - which is a feature! Noone wants to need arcane, private knowledge about how to fix that one problem with that one container in dev.
I manage electricians, and this is highly dependent on the task. In most cases I’d rather send two JWs vs a JW and apprentice.
I’d send two JW to pipe a new feeder, but I’d send an apprentice and JW if they’re just pulling wire and the pipe is installed already. An apprentice can easily feed wire while the JW pulls, but a JW is going to be much better at piping than an apprentice.
I'd like to preemptively emphasize the distinction between "important to have" versus "marginal price to acquire", before folks start talking past one-another with different interpretations of "valuable."
"What's wrong?"
"Sam asked me what a traceroute was."
Sam was the longtime Novell admin. Yes this was a while ago...
"The worst part was, I patiently explained it to him, and he just asked me why that was useful?"
Just one example. There was more, of people of low competence who were very entrenched.
Novell NetWare was most commonly used on a single, flat network with no routers or firewalls. The networks of the era were smaller, and the setup and routing of IPX is more automatic and simpler than IP. It didn't scale "up to the Internet", but it also needed less configuration and troubleshooting.
Notably, the "tracert" tool is specifically for IP, not IPX!
That's why Sam's Novell-expert friend hadn't used it and wasn't familiar with it.
It's like being surprised that someone that had used NoSQL for their entire career wasn't familiar with 3rd normal form and relational algebra.
The amount of random trivia in tech is a vast ocean. Nobody knows all of it. Specific cultures swim in specific waters, but out of habit more than anything else.
While you present a plausible sounding explanation for his lack of knowledge about TCP tooling, networking duties were under his purview as well. He spent a lot of time on the clock not hungrily learning technology, but sneaking off to smoke weed. I watched him once pack a bowl and smoke it while driving on the highway with his knees, coming back from a remote campus. He was very skilled at smoking weed.
Sometimes you need the guy who has burned his fingers repairing things, over and over, to review your design - he's going to tell you why you are going to burn your fingers. Other times, the guy doing things in a semi-glib fashion, needs to be pulled off the production line and enlightened as to why things have to be done a certain way.
To consider this relationship hierarchical is to weaponise the subject - instead, these are key relationships. A single great scientist with 2 or 3 good technicians can build amazing products.
3 scientists with no technician, rarely ever do .. but there is of course the odd exception where a technician, by making something really great, becomes a scientist, too - and scientists, without technicians and therefore having to get their fingers toasty, often make amazing technicians. (This is why the hierarchical subjugation of the subject must be resisted.)
The Wetware Crisis: the Dead Sea effect (2008) - https://news.ycombinator.com/item?id=23166786 - May 2020 (127 comments)
The Wetware Crisis: The Dead Sea Effect - https://news.ycombinator.com/item?id=22376468 - Feb 2020 (2 comments)
Internet Blogger B: "What is wrong with hiring process? We keep getting these clueless noobs showing up. They mess up every time we ask them to maintain prod. All they want to do is some random resume-driven development and then leave us with all the problems. Why can't we hire some real software engineers for once?"
I’ve mostly settled into a place with a good pay rate-to-work expectations ratio, but I’ve also contemplated optimizing for pay alone.
in terms of optimizing for pay vs lifestyle, i spent some years optimizing for lifestyle but im definitely in optimize for pay mode right now. Make a bit of a bag and get out and go back to optimizing for lifestyle. Things are better that way.
Does this approach also apply to age? E.g., older programmers either take some higher-ranking position (management, product owner etc) or they get replaced?
Really, that's all the article has to say. You're welcome.