Understanding how bureaucracy develops
dhruvmethi.substack.com
dhruvmethi.substack.com
“ I divide my officers into four groups. There are clever, diligent, stupid, and lazy officers. Usually two characteristics are combined. Some are clever and diligent — their place is the General Staff. The next lot are stupid and lazy — they make up 90 percent of every army and are suited to routine duties. Anyone who is both clever and lazy is qualified for the highest leadership duties, because he possesses the intellectual clarity and the composure necessary for difficult decisions. One must beware of anyone who is stupid and diligent — he must not be entrusted with any responsibility because he will always cause only mischief.”
Some feedback must exist. (calm, obective and possibly private) Eventually it is up to the manager or manager’s manager to be aware what is happening and take action if critically needed.
If single small mistakes have disastrous consequences, the system is probably too brittle. Approaching it from a blameless angle gives you a better chance of fixing it, as people will cooperate to fix the issue rather than be busy not getting fired
You can still identify and fire/relocate/retrain incompetent people, but that is better done as a continuous thing than as a response to a mishap
Holy crap, I'm going to save that quote forever. I have a co-worker who treats every line of bad code committed as a reason to add ever more layers to CI. Yo, we caught it in testing. There's no reason to add another form we have to fill out.
I’ve encountered a similar phenomenon with regard to skill as well: people want to ensure that every part of the software system can be understood and operated by the least skilled members of the team (meaning completely inexperienced people).
But similarly to personal responsibility, it’s worth asking what the costs of that approach are, and why it is that we shouldn’t have either baseline expectations of skill or shouldn’t expect that some parts of the software system require higher levels of expertise.
Conversely, one skilled senior can often outperform a hundred juniors using simpler tools, but management just doesn’t see it that way.
However, management tends to align with reducing the baseline level of skill, presumably because it’s convenient for various business reasons to have everyone be a replaceable “resource”, and to have new people quickly become productive without requiring expensive training.
Ironically, this is one of the factors that drives ever faster job hopping, which reinforces the need for replaceable “resources”, and on it goes.
Management is correct, if that's the question.
In some very rare bleeding edge cases it is true. Everyone wants to think their company is working on those areas. But here's the truth: your company (for any "you") is actually not.
If you're writing code that is inventing new techniques and pushing the hardware to limits not before imagined (say, like John Carmack) then yes, a single superstar is going to outperform a hundred juniors who simply won't be able to do it, ever.
Asymptotically close to 100% of software jobs are not like that (unfortunately). They're just applying common patterns and libraries to run of the mill product needs. A superstar can outperform maybe 3-4 juniors but that's about it. The jobs isn't that hard and there are only so many hours in a day.
This is made worse today because neither quality nor performance matter anymore (which is depressing, but true). It used to be the software had to work fast enough on normal hardware and if it had bugs it meant shipping new media to all customers which was expensive. So quality and performance mattered. Today companies test everything in production and are continuously pushing updates and performance doesn't matter because you just spin up 50 more instances in AWS if one won't do (let the CFO worry about the AWS bill).
There do exist (I would even claim "quite some") jobs/programming tasks where superstars are capable of, but a junir developer will at least need years of training to be able so do/solve them (think, for example, of turning a deep theoretical breakthrough in (constructive) mathematics into a computer program; or think of programming involving deep, obscure x86 firmware trivia), but I agree with your other judgement that such programming tasks are not very common in industry.
3-10 juniors can make a massive expensive mess of a crud app that costs $x0k a month in amazon spend and barely works, while someone who knows what they're doing could cobble it together on a lamp stack running under their desk for basically nothing.
Knowledge/skills/experience/ can have massive impact.
Yes! Absolutely. It will be faster and more reliable and an order of magnitude (or more) cheaper.
Alas, I'm slowly (grudginly and very slowly) coming to terms accepting that absolutely nobody cares. Companies are happy to pay AWS 100K/mo for that ball of gum that becomes unresponsive four times a day, rather than pay for one expert to build a good system.
The other day, two of our juniors came to see me, they had been stumped by the wrong result of a very complex query for 2 hours. I didn´t event look at the query, just scrolled down the results for 10 seconds and instantly knew exactly what was wrong. This is not because I'm better at SQL than them or a Carmack level talent. This is because I've known the people in the results listing for basically all my life, so I instantly knew who didn´t belong there and very probably why he was being wrongly selected.
Trivial, but 10 seconds vs. 4 man hours is quite the improvement.
Sure. But now try sustaining that impact multiplier every minute, 8 hours a day for a year.
Did you mean something else?
Extreme B: every collegue is required to read and be able to recite x86, C specifications, postgre manual, and must have IQ 190+.
What is obscure or hard to understand is subjective.
The problem with indiscriminate application of "code has to be easy to understand" is that it can be used to make pretty much anything, including most features of your language, off limits. After all, a junior developer may not be familiar with any given feature. Thus, we can establish no reasonable lower bound on allowed complexity using such a guideline.
Conversely, what’s too simple or too difficult is very specific to the person. Somebody who’s coming to a junior developer role from a data science background might have no problem reading 200 lines of SQL. Somebody with FP background might find data transformation pipelines simple to understand but class hierarchies difficult, and so on. So the "easy to understand for anyone" guideline proves less than useful for establishing an upper bound on allowed complexity as well.
Therefore, I find that it’s more useful to talk about a lower and upper bound of what’s required and acceptable. There are things we should reasonably expect a person working on the project to know or learn (such as most language features, basic framework features, how to manipulate data, how to debug etc.) regardless of seniority. On the other hand, we don’t want to have code that’s only understood by one or two people on the team, so perhaps we say that advanced metaprogramming or category theory concepts should be applied very sparingly.
Once that competency band is established, we can work to bring everyone into the band (by providing training and support) rather than trying to stretch the band downwards to suit everyone regardless of experience.
Great point. This would also apply in the context of DEI hiring initiatives.
I told them it was hard to become a C expert without understanding and using pointers and they didn't like that answer. https://daedtech.com/how-developers-stop-learning-rise-of-th...
The blame game drives the exact same bureaucratization process, but faster, because all of the most capable and powerful players have a personal incentive to create insulating processes / excuses that prevent them from winding up holding the bag. Everyone in this thread at time of writing is gleefully indulging in wishful thinking about finally being able to hold the team underperformer accountable, but these expectations are unrealistic. Highly productive individuals do not tend to win the blame game because their inclinations are the exact opposite of the winning strategy. The winning strategy is not to be productive, it's to maximize safety margin, which means minimizing responsibility and maximizing barriers to anyone who might ask anything of you. Bureaucracy goes up, not down, and anyone who tries to be productive in this environment gets punished for it.
"Blaming the system" doesn't prevent bureaucracy from accumulating, obviously, but it does prevent it from accumulating in this particular way and for this particular reason.
I'm never going back to that kind of culture, it's soul crushing.
In modern corporate blameless culture, nobody takes the blame. Now this has its own variety of issues, it’s not perfect. But if you look at blame culture, then exactly like OP said, you have to stop building and start protecting. You know who has time for that? The underperforming lazy employee.
If, as an accountable leader, you realise that someone ignored the processes and protections, you still have the right to hold them accountable for that. If someone is being lazy, it’s your job to identify that and fire that person.
I won’t pretend it’s easy, and I fully appreciate organisations struggle to make that happen for the reasons you and the article raise.
However, the broader problem I have with blame-focus is that it only applies to Individual Contributor roles. I’ve never heard of middle management being held accountable for any actions whatsoever. And obviously not for less “egregious” misconduct like toxicity, workload, favoritism, etc. Heck middle managers can be completely ignorant of their reports’ actual work and survive for decades.
In my experience at FAANG, the worst of managers will get reassigned to a different team, and maybe have their promotion delayed. Occasionally, I’ve seen VPs get put on nearly a year of gardening leave after major misconduct like sexual harassment - and then they leave and become a C level at a smaller company. And of course, CEOs are fired only for complete mismanagement and company failure - and that’s a very high bar and can take forever until shareholders loudly complain.
Basically, my point is that you can only blame the actual workers at the end of the chain - everyone else along the way is easily shielded and escapes blame.
I didn’t think you were advocating for the situation that occurs. I was merely proposing that “blameless” processes are possibly mis-assigned blame (heh) for company cultures that become centred around ducking accountability.
Sometimes the correct answer is, "We accept that this was a low-probability event, and we accept the risk of it happening again. No change required"
IME most "system improvements" elongate the feedback cycle so they need to be weighed against risk/reward.
Some people want to be holding the bag, if the bag is full of money. All risk no reward won't attract accountable people.
If screwing up my job meant getting fired with a $5M golden parachute, I would be more than happy to be assigned individual blame!
Everyone's time is finite. Would you rather spend a few years to make high five figures slogging through a failure of mid 6s succeeding? It's the same mental calculation but with more 0s in to the left of the decimal.
Ye. You quickly learn that with Scrum. Pad alot and report the amount of hours estimated at an even pace. Make the burn down line straight.
People doing actual work have a big disadvantage versus those that put their time into manipulating the setup, if the bosses are oblivious. It is really frustrating as a newgrad until you realize how it is done.
Someone has no idea how modern human psychology is the only thing creating any of these super structures and their frailties.
We aren't ever going to be your super ant organism, get over it.
Eloquently put. Also, likely false.
E.g., soft-realtime audio synthesis applications like Pure Data and Supercollider have had essentially the same audio engines since the 1990s. And users see any missed audio deadline as a negative outcome. More to the point-- for a wide array of audio synthesis/filtering/processing use cases, the devs who maintain these systems consider such negative outcomes as systemic failures which must be fixed by the devs, not the users. I can't think of a more precise example of "blameless culture" than digital artists/musicians who depend on these devs to continue fitting this realtime scheduling peg into the round hole that is the modern (and changing!) multitasking OS.
While there have been some changes over the last 30 years, in no way way have any of these applications seen an explosion in the number of systems they employ to avoid negative outcomes. There's one I can think of in Pure Data, and it's optional.
IMO, there's nothing noteworthy about what I wrote-- it's just one domain in probably many across the history of application development. Yet according to your "law" this is exceptional in the history of systems. That doesn't pass the smell test to me, so I think we need to throw out your ostensible law.
> devs who maintain these systems
What is meant by “system” here? Computer application? Hardware?
You can't stop personal failures from happening because people are people. You can design processes to minimize or eliminate those personal failures from yielding negative outcomes.
Every check has a cost. For some checks, the cost is more than it prevents. Don't add those checks, even after the negative outcome happens.
When everyone is responsible, nobody is.
Even in aviation and surgery, improving the technology tends to be more effective than firing the pilot/surgeon or adding more paperwork. If you find there's a button that crashes the plane, fix the button. Don't fire the pilot or add another hour of education on how to not press that button.
It's easy to quantify the harm of any specific failure. It's hard to quantify the harm of incentivizing people who can fly by the seat of their pants (metaphorically and literally) and generally succeed out of an industry and incentivizing button pushers, checklist runners and spreadsheet fillers into an industry. Say nothing of the fact that a bureaucracy built of these people has every incentive not to study it and to find in their own favor if they ever do.
In order to publish a research paper, it has to be reviewed for suitability for public release. This process is more than a little silly, because it requires seven levels of review, of which exactly one - my immediate supervisor - will have any idea what the paper is about. But fine.
There used to be a paper form. You'd fill it out and either route it around for signatures, or if you had a time crunch, walk it around yourself. Eventually they replaced the paper form with a web form, so now there's an automated queuing system that emails people when they have a paper waiting to be reviewed.
The web form has all of the same info as the paper form, with one addition. They scanned the paper form and turned it into a pdf, and they make you fill out both the web form AND the pdf version of the original paper form. So to sign off on a paper, you now have to download the pdf, digitally sign it, upload it again, and hit the "Approve" button on the web form.
Because God help us if anybody does an audit and we don't have all of the forms correctly signed.
There is indeed a list of rejected papers. The system logs all of them. Generally they're recycled, updated, and published elsewhere.
We've all had those experiences where, the problem we are trying to solve isn't inherently difficult - it's difficult because three separate teams are in charge of parts A, B, and C of the problem, and getting them to talk to each other, align their roadmaps, change their processes, etc. is impossible, unless you can go above their heads and force them to do it.
I think about organization design similarly to software design. It's tempting to think about your software design from the top-down, and design a hierarchy of siloed interfaces with encapsulated private data and strictly separated concerns. This looks beautiful on paper, but then in practice you now have to navigate through a sea of objects and abstractions and redundancies - getting anything meaningful done often requires "punching holes" through the siloes so data can be shared.
Organizations are the same way. Paul Graham wrote an essay[0] recently about the differences between "founder mode" and "manager mode". In a nutshell, managers usually think about organizations as silos - we divide up the company into a hierarchy of departments and teams and levels, so that only directors talk to middle managers and only middle managers talk to supervisors and only supervisors talk to the individual contributors. Again, it looks great on paper, and is what most people are used to.
But "founder mode" is when someone with a lot of political capital can step in and say, "you know what, I want to talk to the people on the ground. I want to find out what's actually going on below the surface in the org, not just the pre-packaged PowerPoint version I hear from my directors. I want to pull together people from across teams and across levels and across departments - whoever is best suited to making this project a success." I think that sort of "hole punching" can be really powerful, if the company's culture is amenable to it.
John Ousterhout explained "it's easier to create a new organism than to change an existing one...": https://web.stanford.edu/~ouster/cgi-bin/sayings.php :
"I don't have to know how what I tell you to do fits into the overall flow of what you want to accomplish between multiple different teams. That's your problem, go talk to all the different teams you have to work with to get approvals and figure it out yourself, that's not my problem."
"I acknowledge absolutely nothing has changed, and I will never use what you create, but format A has changed to format B because everything new is in format B, and thus you must spend time to re-create everything in format B so everything in my filing system is consistent."
"Yes, this is recording the information twice in two different ways. By hand. And with no enforcement to keep it up to date. But I'm only measured on recording the information, and as long as Number Go Up on the recordings, it is not my problem."
"Yes, the new form has not been published. Yes, the new requirement for these approvals have not been put into existing processes that take months to grind out a decision, much less the processes debugged. But in six months, everyone must follow what we publish in several months leaving only two months for you to implement in 3-4 months a new, untried process. No, we will not talk about this possibly creating a bottleneck and exacerbating the flow, that won't happen."
If there was a chargeback of effort to this kind of mindset and decisions, then a different political battle would emerge, but at least it would make these kinds of unilateral, unaccountable decisions in their aggregate form visible to the organization. A tremendous amount of this friction goes away if organizations had the capacity to automate many processes, but the cost hurdle is currently too high for most organizations. I'm hopeful LLM's in the hands of non-technical staff can mitigate that, but I'm likely dangerously naive there.
Systems too focused on defeating corruption as a main objective tend to miss their original intent, and become overly restrictive, to the point of having to rely on rule breaking to actually perform their function.
But once a particular rule is OK to break, every other rule is in jeopardy.
This way you end up with systems like the Spanish access to public jobs: extremely punishing for the participants that are subject to humongous nonsensical competitive examinations, ostensibly to select the very best strictly on their merits, but still rife with corruption.
How much of a full time researcher’s time is bureaucratic overhead? In my case no more than 10% but it feels like 30%.
Even if it takes up only a small percentage of time, it probably takes up the majority of frustration (that's at least true for me)
Which are only involved with much bigger institutions harboring their much larger, dissimilar but more dysfunctional and less-professionally-developed "inadvertent" bureaucracies which are most often layered on under emergency conditions.
Hint, the key word is work around.
Don't even think about it until you've spent however many years it takes to "master" the bureaucracy itself, working from within.
Before you even attempt your first workaround, or you could be shot down in flames without a parachute.
I've said it before but you can't herd cats unless you're a cat already.
These leaders will have to be held accountable. Generally one want to do the dirty job. It boils down to people and nothing else. Any attempt at analysis the system without considering people's intelligence/character is premature.
My partial take on it: https://quberoot.wordpress.com/partners/
Organizational malleability is key. But it wouldn't work in FAANG style standardized performance review style of work.
It's because of the principal agent problem.
As organizations grow, people inside it become less and less oriented towards the organizational goal. The rigidity appears to fight that.
But many of our current problems are more directly related to Taylorism and an intentional separation of design from production.
GMs failure to learn from Toyota at the NUMMI plant is a well documented modern example, with Japan intentionally targeting the limits of Taylorism to basically destroy the US industrial market is another.
The centralized command and control model also ignored the finite congitive ability and assumed the relational omniscient actors.
The funny thing is that multiple Nobel prizes have been awarded related to some of the above, and we know that Taylor faked some of his research and most business tasks are not equivalent to loading pig iron onto train cars anyway.
Even TOGAF and ITIL recently made changes after the Feds changed the Klinger act, moving away from this model and every modern military uses mission command vs C2, but management is still teaching the pseudo scientific management school of thought and not addressing the needs modern situations.
The incentive models are largely a reason for this and recent developments like 'impact' scores push things back even more.
You can still have a principal-agent relationship, but delegate and focus on intent and avoid this trap, but it requires trust and bidirectional communication.
Really IMHO, or comes down to plans being safe feeling, high effort and compatible with incentives.
Those plans never survive the real world, because of the actors bounded rationality and knowledge.
A book that is potentially a useful lens into this is 'The art of action', but it is just a lens.
Organization 'goals' are often poorly communicated and useless because 'planning' is insufficient and not durable enough.
Being way past any horizon that can be planned for, actionable concepts of shared intentions and purpose are not communicated.
Toyota gave teams concrete goals to obtain, allowed them to self organize and choose how to deliver.
GM meticulously copied what those teams had done and forced Detroit teams to follow those procedures and it failed.
It was allowing the teams, which understood the context and constraints of their bounded problems that worked, not the procedures themselves.
Amazon's API mandate resulted in a product mindset and scaled better than almost everyone until culture erosion killed that.
Delegating works, but centralized process needs to be restricted to the minimum necessary.
Unfortunately the negative aspects of bureaucracy seem artificially successful in the short term, but the negative aspects of setting things in concrete are long tailed.
The growing disengagement problem is one of those long tails IMHO.
Taylorism is actually an attempt to make organizations flexible given that the more subordinated people are all completely unaligned with the organization goals and the management is in closer alignment. It's a very direct consequence of that.
Of course, the irony on it is that reality is often closer to the other way around.
All too often the front-line workers do not have a direct voice that will be responded to from above in anything but a generic way.
So without guidane from above, or in spite of it, the lower echelon ends up aligned most closely with the macro vision of what they think the company is supposed to be like traditionally or as a unit, and it's often quite a bit different than some leaders toward the top who are in position to identify and gravitate to directions that are more lucrative for themselves than the company as a whole.
It could all be represented by a graph but it is a huge measurement problem.
Minimize the variance of what with a business unit? It is not clear to me there is one measure that would apply across business units. I suspect that is part of the problem and something that gets amplified with scale.
I'm not sure when Scott did his work, but this reasoning (at least after 2019) would have been wrong. IRB review only considers "generalizable" research, and is exempted from IRB review according to federal regulations and as encoded into specific institutional procedure. [1] Federally binding standards about specific exempt categories exist, but they aren't exhausitive. [2] Part of the point of IRB training – well, human subjects research training specifically – is to get drilled into you what's within and not within the scope of IRB.
[1] https://ori.hhs.gov/content/chapter-3-The-Protection-of-Huma...
[2] https://irb.ucsf.edu/sites/hrpp.ucsf.edu/files/exempt%20cate...
Maybe the real problem is that there is (at least to a degree) a trilemma between effective, safe from research misconduct, and respectful of individual privacy.
If a signature is meant to represent both intent and identity, what is it about the physical medium which makes it more ideal than a digital signature where you're prompted to enter in your login password or something similar?
Is it the belief that its less forgable, that electronic audit trails are more easily duped and spoofed while signature blocks and paper/pen is somehow immutable (despite the decades of forged signatures easily traced from other sources)?
Never understood this idea whatsoever, it just strikes me as a form of pearl-clutching over some nebulous hackers that could easily destroy our well-oiled pen/paper/document machines.
Electronic signatures are an entirely different (and interesting) thing to consider.
> IRREGULARITY #3: Signatures are traditionally in pen. But we said our patients would sign in pencil. Why?
> Well, because psychiatric patients aren’t allowed to have pens in case they stab themselves with them. I don’t get why stabbing yourself with a pencil is any less of a problem, but the rules are the rules. We asked the hospital administration for a one-time exemption, to let our patients have pens just long enough to sign the consent form. Hospital administration said absolutely not, and they didn’t care if this sabotaged our entire study, it was pencil or nothing.
When in doubt: don't imitate the DIV.
The rest is talk.
In a startup, there is some natural way to choose the 1 right person for the job, say, the Developer. They probably wouldn't be chosen unless they have a high level of ability, communication skills (hopefully) and a lot of productivity and proactivity. This could be directly related to the potential rewards of equity etc. but maybe not, maybe they are happy to be free to work at high and low level and to make educated decisions on everything. I did this job for the same money as my older job without much expectation of a large payout (never got one!)
But as soon as you hire 1 other person, things immediately change. It isn't just about their ability, although we often talk about that both in terms of testing it during interview but also measuring it using some KPI but a person is much more than ability. You can have ability and be lazy or lack ability but are a quick learner. You might be really easy to get on with but your stuff is slightly above average, or a complete a-hole but produce rock-solid code. You might be reliable, you might not, you might be motivated all the time, some of the time or none of the time. As much as we like to think we can have regular performance reviews, you can't put numbers on those things but you can put process. I can arrange a daily standup and weekly progress reviews to get slightly better at making sure you are on-track. I can try and count things like tasks completed or LoC or Stories completed or bugs in production etc. but these are also a rich tapestry and after-all, do you compare them to the 10x coder that you are?
Some people need to be told to do something properly, they need process, others will do it properly without being asked, they have passion. How can you tell? Mostly gut feel, and maybe a checklist! I think there is also a myth that if we only had tonnes of cash, we could be much more picky with hires and only get the best, that would remove a lot of bureaucracy but look at FANNG companies, they have this problem in buckets. People can fake their ability and passion to a point, they have 1000s of applications to process and they still have tonnes of meetings. Even if you hire someone with the same passion and ability as you, they will be still be different. They will be insistent on VueJS instead of C#, TDD instead of DDD, Unit Tests for everything or UI tests for some things. Each of those is OK but you still then create that Architect position to ensure consistency between teams, the Project Leader to ensure delivery times are balanced with technical perfection.
Personally, I have never believed that a company with more than, say 200 employees can ever be efficient. They can be rich but eventually they will buckle under their own red-tape or at the next culling that takes place under the "new trendy CEO who had successes at previous companies".