Why is everything so hard in a large organization?
graphthinking.blogspot.com
graphthinking.blogspot.com
* You don't get to be a large organisation without accumulating generations of previous products. Well, unless you're google and can regularly fire your customers without going bust. but part from them, you start to need processes just to track what's going on. (The ineffective startup accumulated previous product attempts too, and is still paying for a lot of pointless infra because no-one still there knows which ones can be switched off).
* At a large scale, it starts to be pretty difficult to maneuver if each dept of 30-50 people has built or contracted all their own infra & services. Some divergence is useful for agility, but a lot of it is just waste and actually slows you down when you want two depts to work together on something - or even prevents useful collaboration. An effective organisation will make the tradeoff consciously.
* When you complete dozens of projects per year, it starts to be possible to invest in researching and rolling out best practices in areas where a startup just has to go with the simple/obvious answer. When my startup employer moved, it was pretty onerous, and there was a bunch of stuff we never found again. The large org had so many offices that it had a full time dept just for moving offices. (They weren't dumb, each move consolidated offices -but they kept buying other companies). When they moved us, it was like magic - we went home on a friday and went to work on a monday in the new office, and everything was set up. Well, we had to set up the lab ourselves, but
I work in a large organization, in an industry that is naturally slow. The work (in the industry) is slow because it's critical to deliver stable and reliable solution because large part of population and economy rely on this industry SW/HW. I have heard a lot of complains on how the process is slow, but in many cases, building on top of existing work takes a long time. It takes long time to understand legacy work, to go through older studies and projects, and to find spots to add contributions. Not all of us work on green field projects, but many of us maintain code/work that doesn't look "cool".
Just today, I logged into Google Calendar on my iPhone and the mobile UI looked like it hadn't changed since 2008 and it was missing a ton of features.
As in any other company, headcount is always limited, and some products are funded by nothing but the willpower of particular passionate engineers/managers, which results in a very low lottery bus factor.
This is only true for external products. When I was at Google it always seemed to me that the majority of engineering effort is focused on the inside, and that the same effect exists there.
An effective responsibility culture and structure. Those who are responsible for something must have the means to effect it. There must be a means of agreeing who is responsible for something and an effective way to find who is responsible when someone is blocked. Things should not languish because no-one is responsible, nor be trampled in because everyone wants a piece. (This is culture as well as structure, because everyone needs to internalise what a good responsibility structure looks like so that they negotiate it into place in a distributed way, rather than having it imposed top down.)
Prioritise making intelligent decisions over raw speed. In my effective employer, if your project was late, you were required to take a step back and really work out when it was realistically deliverable and make sure you genuinely understood what was now needed - better to say ONCE 'it needs another 3 months' than 'just a another 2 weeks' 8 times. (The ineffective company had a 5 day horizon and would burn everyone out constantly trying to deliver next week) Related to this is that profligate agility can work against intelligence. The effective org could afford what would seem like a huge amount of process, but it actually took less of our time than the agile process in the ineffective org because we didn't need to change our minds as often.
Making effective use of staff's skills. The means everything from: Leaders need to recognise what they aren't expert in, not having a destructive people culture, giving people larger pieces of work that they can master rather than splitting everything into a million fragments, allowing for development rather than people getting type-cast.
> profligate agility can work against intelligence
it seems in our chase for "agility" above all else, we are sacrificing our intelligence along the way...
these two factors are extraordinarily hard to cultivate consistently, especially during high growth. this is incidentally why management is so hard, and so often “bad”, because managers are often encouraged to focus on metrics, like financials, rather than cultivating trust and fostering effective communication.
kanban (in japanese manufacturing) is a quintessential example of a management system designed to foster trust (no-fault problem-solving orientation) and communication (simple, immediate, and well-known signaling).
edit: should also have mentioned kaizen as a companion to kanban in this context.
One thing I'm realizing is really useful, and which we're doing more and more, is putting "program managers" in charge of projects, totally separate from "project managers", which really should be called "people managers". With that split we can have a senior engineer who is interested in management work across multiple teams without worrying about the day to day personnel issues. This only works if everyone supports this approach but here it's working well.
It's gimmicky and a pain in the arse. I know management does do a job, but it's hard to justify needing two instead of one.
My view I that matrix style management is a solution to a communication problem you haven't figured out.
Instead of fixing the communication throughput problem, your adding more bandwidth and scaling out the number of nodes in hopes it magically fixes the problem.
And it sort of does fix the communication problems, that's why people like it. The trade off is that as an IC its arese to have to jerk two cocks at the same time.
With a program manager/people manager split, it might be the case that the person leading the project has to truly lead, because the actual authority sits with the people manager. May or may not be a better structure.
maybe the first conversation leads to the second, but its not me :)
I don't think this is that different; people have more static line managers, and are assigned to projects, which project managers then govern.
I was at an effective startup that turned into an ineffective large organization before my own eyes. Your three (excellent) points ring very true.
However, the company actually tried to implement each of your three points and recognized their importance. The problem was in the execution.
Old products: These were neither retired nor given appropriate resources, so they instead decayed over time. Empty promises about longevity were made and then broken. Attempts were made to outsource maintenance to countries with the cheapest engineers, which backfired when each update become more buggy than the last. The end result was a lot of customers and early adopters getting burned and a destruction of our reputation in the industry.
Shared infrastructure: The C-levels saw shared infrastructure as an opportunity to reduce costs and speed up development, but they took it too far. At the worst, they wanted to have one gigantic team of back-end engineers, one gigantic team of front-end engineers, and so on, spread across every project across the entire company. Individual engineers were assigned to many projects and each had to deal with multiple managers fighting for their time. End result was that nothing got done and politics reigned supreme as the way to get attention on your project.
Best practices: The company hired “subject matter experts” to enforce best practices. They were expected to be involved in everything across the company. Imagine being hired as the “security best practices owner” and then told that you’re accountable for the security of everything at the whole company. Instead of promoting best practices, they went into full defensive mode and interfered with the shipment of every product and feature that they didn’t have time to work on. We would have products ready to ship and have the best practices experts panicking to halt it because they wouldn’t have time to look at it for months after completion. Getting them involved at the start was a pipe dream.
The overarching issue was that they tried to do big company practices in the move fast and break things startup style. For every well-intentioned move toward being a proper company they managed to screw it up by trying to inject startup-style speed and improvisation.
The overarching issue was that they tried to do big
company practices in the move fast and break things startup
style. For every well-intentioned move toward being a
proper company they managed to screw it up by trying to
inject startup-style speed and improvisation.
The problems all came from the top-down direction. Replacing the CEO with an experienced big company leader could have solved so many issues.
The CEO was good at founding startups and running small teams, but that’s all he really wanted to do. He was trying to run a big company like a small startup and actively fought big-company organization styles.
Companies like Microsoft, Facebook, and Google don’t try to pretend they’re still startups. They embrace the fact that they’re a big company and they make it work. They’re not perfect, but it’s better than a slowly failing big company that can’t accept that change is necessary.
Or worse, a big company that is in a growth/high profit area that takes in enough money to not fail despite dozens of missteps.
If you do the wrong thing long enough and don't go out of business, then the wrong thing becomes "our way".
A big established company can get away with a shocking amount of incompetence for a really long time, while a very well run startup can easily fail.
You just described most of retail banking and insurance (and I assume, utilities).
It took the 2008-prompted investment-retail bank mergers and ACA admin costs caps to nudge these industries into the 21st century.
The financial side of health care providers is still a nightmare (even though the actual medical side is amazing at continuous improvement).
Somewhat related, I've been in effective teams that turned into organizations, a couple of times. One of the main issues I've seen is that they're especially prone to survivorship bias, on both accounts of process and people.
You get the usual "this is how we managed to get there, it must have been right, let's do more of it", and people get protective or suspicious if you propose anything else.
There's also a tendency of over-attributing success to the people who happened to be here while the team/product became successful. They tend to be promoted to responsibilities, their voice is over-amplified and their opinion becomes dogma.
The result is that rather than adapting to the turning point both organizationally and architecturally, you end up with dungeon keepers protecting their babies under the cover of "this is our culture". The growth is not painful and rather than re-thinking things from the ground up, we correct by patches and ad-hoc fixes - approvals, reviews, build queues, more planning, backlog sign-offs and such.
How do you fix this? Assume all success is pure luck and ban dogmatic things like style guides and code reviews and force entire rewrites of sowftware and infrastructure every few years so that best practices change and don't become dogmatic?
One of the allures to getting in early somewhere is that you get to build the foundation and become the thought leader and define to an extent which dogmas shall be adhered to because you have the context to make those decisions. I'm sure it can get overbearing when abused, but I also think it's quite presumptuous to join a mid-stage startup thinking you're going to come in and uproot everything and make things happen your way. If that's what you want go make the sacrifice and join a small place and earn it. And grow the business with it. And then incorporate it into what you can bring to the table as an experienced industry professional so you can be hired as a director somewhere else when you're ready to move and do it at larger scale. IDK...
Startup pivots are failures made good. Big org failures means people get fired or arrested.
The misunderstanding I saw across several organizations of different sizes was the mistaken rationale that surely, I should be able to take up "just" 25% of your time. When there are 5 other groups, there is nothing at all sure about that statement. Either you are lucky to get 10% or you're riding that employee into burnout and don't understand how they just up and quit one day.
This is the biggest one where I work. Call it technical debt, legacy systems, or aborted migrations - we have multiple generations of products doing the same thing.
Management comes in bright eyed with a desire to migrate to the new system. Not too bad at first, we get some quick wins. Then the 80/20 rule hits, people realise how hard the rest will be and we decide to put it on hold temporarily (i.e. forever).
This is the worst of both worlds, as we now only have the most complicated and ugly applications using the old system. There is nothing easy to train new people on. Over time fewer and fewer people truly understand the old system, work gets slower, and more error prone.
But hey, there's another new system coming along - maybe we can move to that, along with all the fresh easy work we setup in the last few years. Cue xkcd comic about a solution making more problems.
The other side of this is policy and procedure cruft: having so many different systems makes mistakes more likely. Every mistake means an investigation and an item added to a multi-page policy document. You'd never get everything done if you fully adhered to it, but it makes people feel safe.
This is not to say that innovation isn't possible. In my experience, the way to forge ahead in large orgs is to achieve progress by example, advocacy and influence.
Not what I have seen. Big companies with big profits will throw millions away on dumb ideas on the chance they make money, when they do not.
Often an effective strategy for this to purposefully diverge and converge. Allow a new product to strike it out on its own, and if it survives follow back up and homogenize the infra so that maintenance is cheaper.
Takes a lot of labor, but that’s usually something large organizations have in spades.
This is one of the biggest risks I've seen, for both large organizations and small ones.
Large organizations are built of small organizations, and a bunch of effective small organizations will each be moving and shaking, and building or buying things that become part of their infrastructure. Then one day you wake up and discover that your overall organization is paying for 27 different solutions to the same problem.
Then, any attempt to collapse that down to something reasonable results in zero productivity for quite a while and whole lot of pissed off people.
To the extent that the spaghetti of infrastructure prevents effective interop, that's a problem.
And as the need for interop grows between two orgs the cost of the various hacks, shims, and other cheap adaptations, escalates, until unification of infrastructure starts looking awwwwfully tempting.
Of course, unification comes with its own set of problems.
One of the things I've seen cripple large organizations is waiting for the common infrastructure / services becoming mature enough to be used in production.
Teams are kept in a holding pattern, sometimes for years, while waiting for the teams to deliver their common infrastructure / services solutions, and they're actively discouraged from building their own solutions, because the enterprise-wide solution is "just around the corner".
They don't fire the customers (advertisers) they fire the users.
I don't think it's impossible for a company to get large and still be somewhat effective.
Once you have a large org that's making money, you don't really know exactly why it makes money. Sure, there's annual filings and there's still the elevator pitch of what the org does. But the actual how of how it works is difficult to describe in causal terms: if team X or Y was not there, the business would do better/worse? How does this team interact with that? It's really a bunch of informal relationships between different people, and both the people and the relationships change over time.
So what you want to do when you know the whole sorta-works and you want to keep it that way is you pour glue all over it. You try to formalize processes, you give people titles and you make hierarchies. That way you attempt to stop the firm from inadvertently slipping into dysfunction.
The side effect is of course that everyone who works there can see ways to improve things, but they can't see a way to get those improvements implemented.
No, this requires you to constantly steer left (or right, of course). So you are not traveling on a straight line.
A large enterprise such as Google, GE, or IBM can be poorly managed for decades before being economically forced to change.
I observed a team making 25 MM per engineer in a large company that made one change per year. There are products at google that require 400+ pages of documentation/proof to change.
[0]https://www.technologyreview.com/2018/04/30/143150/computer-...
On the other hand in these orgs you have people crying "iceberg!" all the time.
Well at least operationally it was a good product, and imo better than Netflix's, but I have no idea if helped or hurt them financially.
But Blockbuster's established sources for content/disks should have been an advantage against the stories of Netflix having to go buy retail copies of movies in cases of uncooperative distributors.
It was just Phase 2 and moving to streaming where they fell so far behind. (Plus Redbox's uprising didn't help, which is a separate failure to pivot.)
But failing can be good and bad.
(If they are not succeeding, they can't afford to hire a consultant, of course)
It's my firm opinion that there is a U shaped curve of chaos - new companies do not yet know what they should be doing to make money, and large companies are inherently too complex for any one person (or subset of people) to actually understand what is happening.
This is why I prefer to work with and for medium sized companies - companies that can have goals and metrics that lead to success, but are not yet so large that they are inherently unknowable.
I’m reminded of SpaceX, who is run by Gwynne Shotwell and Elon Musk. Gwynne keeps the whole business machine running and manages existing programs so well that Elon has the bandwidth to dig down to a fractal level and address a lot of the weird issues & bottlenecks that everyone knows about while leading new programs extremely fast. Or at least, that’s the story (doubtless the CEO meddling has negative effects, too, but overall seems to work at least as well as traditional organizations that big). Also, maybe that can only happen in a business like SpaceX which is filled with a bunch of fantastic workers who are just really driven to make things work at all levels.
SpaceX Falcon 9 and Falcon Heavy vs Vulcan and New Glenn and Ariane 6.
SpaceX Starlink vs OneWeb.
SpaceX Dragon Crew vs Boeing Starliner.
SpaceX Starship HLS vs Blue Origin HLS.
I fantasized about what if there were some kind of internal company web app, a portal or dashboard, where people at the highest levels, if they cared (I am not even if sure they did), could drill down to a "fractal level" (as you called it) and see what the software is actually coded to do, presented in a way that makes sense to a non-programmer/higher-level understanding.
The elevator pitch for this kind of IT product would be: "Software runs your business, and you don't know how it works."
Unfortunately I never could figure out how to actually turn this into a real thing. None of the common approximations of software (modeling) do a very good job of capturing subtle aspects or of being more understandable than the code itself. The one conclusion I did come to is that it would have to be something where low level developers interpret and model the code to digest it for whatever format the portal uses, and not some kind of automated distillation or analysis of the code.
I think it might in fact be impossible but if you ever figure it out you'll win all the marbles.
Your target customers might end up being other programmers or managers though.
So people are inherently skeptical, and by the time it becomes obvious that they need some outside heavy hitter, it's too late for that strategy to be effective.
I'd really like to systematize it, to be known as the solution for that kind of problem, but that's a different skillset than actually solving the problem, of course.
its not like the solutions are usually that hard to figure out, it's always a problem with people and their incentives ultimately.
the fixer role is basically to give confidence to the CEO that disempowering certain people is safe and there's a path out.
Out of curiosity, how do you define a medium sized company? How many people, how much revenue, etc? The way people define this varies widely, so just trying to understand how you define it.
In startup terms? Private companies, usually post Series A (usually B or C) but not yet private equity unicorn rounds. In market cap terms it's 100m+ to <2b or ARR ~2m to ~25m for a sass-style company. still a "small cap" in the grand scheme of things.
Even if the optimum is somewhere in the middle, the middle region is a spectrum and the part you picked might be worse than the ends. (Picture a sine-wave shaped graph of quality vs size.)
Feedback loops of communication is what I view as the biggest inefficiency. If you CC 10 people on an email, you’ll still be chasing 3 of them for a response next week.
And thereby deliberately push it into dysfunction. At least when seen by an impartial observer.
Someone smart who knows something wants to add it and gets tangled up in "useless" beaurocracy. Why can't they just let me do it!?
Because if they did just let people do stuff, then lots of other people you consider idiots would have already done things you consider stupid and or bad.
Basically everyone wants rules for the idiots so they don't ruin things and at the same time to not be included in the list of idiots.
That would sound harsh though, so mostly people just moan about rules, rather than them not being exempted, but I don't think anyone genuinely wants the rules to be dropped for everyone.
It would seem fairly easy for Wikipedia to examine its own pages for broken links and flag any such link (and its related content) as unreliable. Wonder what that would look like...
Exhibit A: me writing ;p
> we have multiple on differing levels of the idiot scale depending on the topic at hand
alot of things can affect the idiot scale...
The problem is not rules/"idiots"/novices. The problem is when people who make/enforce the rules are not fit to do so, cling to the rules and apply them with sweeping generality. The people who create those problems are typically incompetent but eager or even zealous at the same time.
They are resistant to learning, addicted to power and/or afraid of change and certainly afraid of appearing incompetent. So they never get past that level where they start to recognize when rules need to change or need to get broken.
You need some brakes on the organization to assure that the rate of change is manageable and that marginal ideas are implemented more slowly if at all.
It might also be better to have a much faster pace and more freewheeling "default don't care; if you can make it work, it'll stay" which also then requires a relatively ruthless culling mechanism (even if slightly unfair) for the things that don't work.
Ask me to navigate a bus station in a provincial Chinese city and I'm a complete fucking moron though. Ask me how I know.
Point being, there's no such thing as idiocy without context.
For example, consider two smart Wikipedia editors working on the climate change and global warming pages... one from the fossil fuel sector, one from the renewable energy sector.
2) If she exists, and is as smart as she should be, she will realize quite quickly that her power can be used for personal gain and she will become corrupted. Power corrupts, etc. and you can't really entertain any kind of organizational solution without considering game theory of the situation.
Sociologists have quantified these sorts of relationships almost a hundred years ago with graphical modelling.
It is easy to nag when they never had to secure anything or operate at scale of 100's of employees and 1000's of users.
It is just hard even if you have good practices in place, because checking that everything is in place and training new people to follow rules is a lot of work.
https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
> . A software developer could write a simple e-commerce website in a few days; Amazon employs 1,335,000 people [source].
Those 1.4 million people are not coding the e-commerce app but making e-commerce happen as package handlers, drivers, finance teams, etc. on top of that Amazon has many non e-commerce related projects: building hardware (Kindles, Alexa, etc) , AWS and support, and on and on.
You misread it. The author is not equating them; he's contrasting them. Simplicity-of-proof-of-concept vs complexity of real-world-AmaFaceGoog.
Wasn't this precisely the focus of the article?
"A software developer could write a proof-of-concept search engine in a few days; Alphabet employs 130,000 people "
This suggests that those 130,000 people are all software developers, and that they all work on the search engine. I think clarifying that ambiguity is a pretty appropriate response.
The reason Alphabet employees 130K people isn't because "running a search engine at scale" requires it, it's because they're in 30 other lines of business.
My first gig was at a small software retailer, straight out of high school and balancing study at uni. Picture a GameStop competitor but selling home/commercial software and licensing as well as games.
The owner’s approach was to employ a series of teenagers who saw this as an opportunity to get that elusive “foot in the door” first job experience despite being paid less than half minimum wage for the hours we were actually paid for (hard not to envy today’s junior Devs walking into jobs out of uni today, yet alone the eye watering salaries, but then again there’s something to be said with romanticised rear vision for working your way up the ladder during that golden age of the early modern web).
Two of us, while balancing sales and service to customers, and keeping the store running (receiving and unpacking stock, boxing up and posting customer orders, phone calls, stocktake, cleaning, directing the occasional casual staff), without any formal education, with tools that don’t match today (no especially useful libraries or tools, eg pre-jQuery, Ajax just catching on and feeling like magic, plain PHP, .NET 2.0 without an ORM etc.), built and maintained a website that in my view rivalled Amazon’s UX at the time (we developed some novel ways to group and present licensing and editions that surpassed Amazon in this space).
On top of this we developed and maintained a bespoke WinForms POS/inventory/accounting system that handled all sales and ordering etc., supporting tools to handle pricing/forecasting, automation scripts to keep customers updated etc., and a series of integrations with suppliers to pull near-live stock availability and pricing for perhaps 40k SKUs from around 80 suppliers (none of whom had APIs so we were screen scraping custom portals, parsing the odd PDF price list, or best case pulling CSV email attachments).
The net effect was with minimal operational effort we could effectively compete with far larger businesses and provide a better experience.
This included the ability, with near zero-touch, to surface a comprehensive catalogue of stock while maintaining minimal inventory, automatically take customer orders, place corresponding supplier orders based on the best price at that point in time (rationalising and aligning across price lists, with consideration of preferential contract terms or shipping timelines), publish near-live pricing and ETA changes, administer a customer loyalty program, and provide automatic updates to customers.
We just got stuff done because we didn’t know any better.
I know this is several scales lower in complexity than Amazon, eg fraud detection, volume of catalog and sales, reseller accounts etc. but again we were not 1000s of Stanford and MIT educated engineers.
Incidentally this has shaped my view of the undervalued power of having bespoke software supporting your core business operations, despite also driving home the risks and perils of reinventing the wheel.
I was recruited by a small consulting firm that had a specific project in mind that aligned with a fake project I had on a resume. My resume was fake because there was a BCIS project in high school to create a resume and post it, but I forgot to take it down, but I didn't let them know that either.
I bluffed my way passed two interviews, was given a $30k salary and benefits. Within a month I was running the team, and within 6 months I had shipped my first product (an ocean shipping contract management system). This is after they had been trying to write this software for over 6 years with no success.
The company then decided that I was worth more than the rest of the team, took me into a meeting told me that I would be getting more responsibilities, and asked if I cared if they laid off the rest of the team. I agreed with them thinking that by laying them off I'd get a big raise... I did not, I just got longer hours and even some weekend work!
So after a month of that exploitation, I quit to focus on a couple of side projects that were almost making enough money for me since I still lived with my parents.
First jobs (especially in "industry")as an undereducated 18 year old kid are crazy exploitative. I didn't know the difference between salary and hourly, or the fact that salary means you work as long as we want you for the same pay as if you didn't work at all.
Despite knowing I'm more effective at avoiding major pitfalls, guiding the team in the right direction, building software that lasts.... I miss this feeling, so much.
You then realise that you are the main part of the success so you now want your own office and a team to manage. Of course, they are not as sharp as you so you have lost your skill in return for trying to train and manage them - some don't even really want to be there.
Then the boss decides to get an investor to help scale the business - you need the support people on-board before you can start charging for support - and that person is now putting you under pressure because you are not updating the website quickly enough, there are bugs that people are complaining about etc.
It's a fairly classic story. You either stick, fold or go FANG!
Take FB as an example. FB have 60k employees, 12X the 2012 IPO number. It also costs 12X to run facebook today. They don't do that much more today than in 2012. They mostly just do things a more complicated way.
IMO, the reason for that is actually revenue. Beyond a point, there's no market discipline reason for mitigating costs. FB is profitable enough with its >30% margins. Investors don't want them to cut the salary bill. They want FB to hire people, invest, do more stuff & grow.
If FB revenues had plateaued around $10bn, I'm pretty sure FB would still exist. FB would have 90% fewer employees and users would not experience much difference.
Once you have all these extra people, and basically the same set of tasks at hand... sluggish ways of doing things and inefficiency are basically guaranteed. If 500 devs are now working on an area that 50 people previously ran, it's going to look like ineffectiveness at the ground level.
A lot of the software industry is like that. Half the industry, like most internal dev orgs are not very capable or motivated, and things look sluggish at ground level. The other half is flooded with cash and great devs, but teams are bigger than actually necessary.
You'd find the same sort of answers at FB that you'd find at citibank. They have regulatory challenges, content moderation challenges. Corporate structure challenges left over from acquisitions or whatnot.
I'm sure there are a ton of items on the todo/doing/done lists. There are "new and exciting" projects, oculus VR, whatever is happening with the marketplace, etc.
Same thing at citibank. From the perspective of a manager there, I imagine it looks like they're doing a ton of stuff all the time.
There's no useful measure of FB's efficiency, what 10X more FB means... so we can't externally measure the claim. Whether or not someone agrees with me or you probably relates to what we think about work & efficiency broadly.
FB doesn't "do that much more today than in 2012" - completely resolvable. I hear statements like this all the time. FB/Google/Amazon don't do more, they just do it bigger, or they just hire engs to keep them from the competition, etc. FB had 3000 employees total at IPO. I can easily name 10 brand new things launched/acquired since IPO that each have a massive user base and at least 1000 engineers on them now. I'm sure people could do the same for Google, Amazon, Apple.
The unresolvable part - are the 60K employees at FB today individually doing as much as the 3K employees were at IPO? Personally, I doubt it. The luxury of being big means you can try more ambitious stuff that may never launch, overbuild on infra and process, etc. The flipside is that niche features you've never heard of can drive millions in revenue bc of the scale of the userbase and business customers.
There's also the fact that in software, you have something once you build it. Software is cumulative, so you'd expect it to grow as long as someone is working on it.
There are various points of evidence, but they're all arguable.
The former implies inefficiencies of scale. The latter implies efficiencies. I think both exist.
To the larger points here, this illustrates another major failing of large organizations that achieve de facto monopoly status due to network effect: arrogance towards the users that made them so large in the first place.
They do.
In 2012, their advertising revenue was $5B. In 2020, it was $86B. They are "doing" 17x more now.
I don't mean that from your perspective of revenue leading to wasteful spending, but from an advertising product perspective. If you're making $86B in advertising revenue, then every junior data scientist whose entire job might be to optimise ad click-throughs for cat food in Mongolia by 3% is a net win for the company. If they are getting better at cat food ads, they are doing more.
That was my point. Revenue rises, and everything else grows with it regardless of need.
Whether or not the marginal junior data scientist is a net win for the company.... I suspect not at a company like FB. Efficiency is hard. Factories are efficient because margins are slim and capital costs are high. FB is the opposite. There's no external disciplinarian forcing managers to find ways of minimizing junior data scientist headcount.
FB profit margin this year is 37%. When margins are that high, there's not much incentive to cut fat. If profit margin is 3%, a 1% efficiency gain is a major win. At 37%... different game.
A business person would know all of the things you are talking about that make that e-commerce site as successful as it is.
I have stumbled across many existing companies that do things that I have thought about developing but they employ 2 people, have no marketing and no-one knows about them.
That said, I also agree that there is a weird exponential growth of team size with a product that isn't always much bigger. Our SaaS company turns over about 3M/year with a dev team of 6. I was looking at another similar company the other day (granted slightly bigger/more complicated but only 2 main products) and they had about 200 devs in their about page!
The article does not claim that. You misread it. Right there, in the quote you included:
> A software developer could write a simple e-commerce website in a few days; Amazon employs 1,335,000 people
It says "simple" right in the middle. Then it immediately says afterward that Amazon employs 1.3M people, which clearly indicates that the author does not consider Amazon to be "simple" (and this holds regardless of whether the 1.3M people are factually related to e-commerce), and that he does not expect his readers to, either.
“given their organization's internal objectives, they are forced to build huge workforces to accomplish what seems like "simple" tasks.”
“internal objectives” is not why Amazon needs 1.4 million people to complete the “simple” task described. And if you redefine internal objectives to include everything else that Amazon does to make money then it’s really saying nothing at all.
He started with "if markets are so great and stuff, why do companies run like soviets & chiefdoms in practice? Why doesn't the product team just contract a UI designer, a testing team and such. Hi solution was "transaction costs." IE, it's too hard to find a designer every time you need a button.
Economics and social science generally is prone to a "defend my model" problem. Once you have a model, there's a tendency to see everything in terms of that model. A Coasian sees everything in transaction costs terms. Coasians ran around
This author seems to see everything in incentive & game theory terms. Management theory often prefers group psychology explanations like Dunbar's number.
IMO, you should probably keep all in mind at once, and remember that none are entirely true. Even if you see the problem as "Prisoners Dilemma," the solution may not be "breaking the prisoner's dilemma."
Human cooperation in groups to achieve aims is basically the secret sauce of human civilization, culture & what makes us different from other animals. If it could be predictably engineered, a lot of problems would have been solved already.
Most, seemingly common sense solutions fail miserably. Trying to break prisoners' dilemmas by nailing all responsibility to someone (a) may be a lot more disruptive and aggressive than you imagined and (b) will often result in elaborate CYA responsibility avoidance. See enterprise consulting for a snapshot of a mature system described this way.
One way or another, this problem has occured to many people over the years. It's not an easy one. Perhaps, it's the hard question of social science.
Or, that is what the management chain does and this is the only way to scale it.
There is also a perspective on this which is along the lines of Turner's social evolution ideas.
To recap that, bureaucracy is recast as a gene for a successful organization. For Turner, this is in some sense a very literal survival of the fittest, because he is asking the question, “why do modern militaries and governments regress to The State as their model, rather than the other governmental styles through history?” And his answer is that The State, or bureaucracy, is legit a better gene, it propagates itself and it propagates the organism better.
“But, the thing we know about bureaucracies is that they are super inefficient, how can that be the best way to go?!” Well, maybe we need to revisit that. The modern military that is structured like a bureaucracy is vastly vastly larger than the militaries that are not. Is there a forcing function that would drive efficiency way down at these resource scales? And once he's asking that question the answer seems obvious, the same answer as in OP: corruption. In computing terms, a tree with fan-out 5 has 5/4 as many nodes as leaf nodes, with 10 you can drop to 10/9, so you have an 11% - 25% intrinsic overhead. (Substantially more in terms of cost of labor since they also want to be paid more than the front-line employee, which again suggests incentives.) We look at this minimum 20% waste and are shocked by it, but the claim is that the transition to bureaucracy starts to happen precisely when you are moving so many resources that the 2% of bad actors that slip through your vetting can sipon off 5x-10x what they actually need and produce a 10-20% corruption. This combination sets a Dunbar’s number, rather than it being some aspect of human evolutionary psychology.
The cost was built into the game rules at these scales, it stops being waste and starts being opportunity cost, in the economic sense... Put another way, maybe it is not the hard question of social science, but a constraint in which social science necessarily operates?
One of the point Dawkins' used his original meme concept to demonstrate is that a meme may not be beneficial to the person or culture carrying it. The meme just needs to persist and reproduce. He is the selfish gene guy, after all.
Also I agree that cultures evolve, but I think the analogy to natural evolution by natural selection can be overstretched. This dynamic exists and can even be dominant, but there are other evolutionary dynamic going on.
For myself, my "Big Theory of Bureaucracy" is simply that it accumulates over time. The larger the scale, the less competition and creative destruction happen so bureaucracy can keep growing undisturbed.
You could analogise corporate bureaucracy to just about any traditional culture: the US senate, the catholic church or some small island tribe.
The US senate does this ritual around debt ceilings. There are written and unwritten rules that make no sense from the outside. It's all based on a rule from 100 years ago, and stopped serving its initial purpose generations ago. Meanwhile, the whole process has gained incidental important roles in coalition consolidation & legislation bundling. Those now depend on debt ceilings. It can be resolved either by vote or a ceremony whereby a small metal object is place inside a large building. The whole thing requires a lot of paperwork.
50 years from now "debt ceiling" might refer exclusively to a quaint ritual where a president gives some guy with a special hat a special coin. At this point, it's impossible to tell bureaucracy from ritual. It's just stuff that needs doing in very specific ways for reasons that are not externally logical. They have history, dependencies, tradition, etc.
Maybe bureaucracy is just an inelegant version of that.
(1) 100 years ago we didn't have reliable instant communication. If you wanted to send a message across oceans it could take days or weeks. As a result having a "chain of command" was critically important to survival. For example a military commander prior to the 20th century could not make decisions regarding an army thousands of miles away, and thus needed to delegate decision making authority. In 2021 a military commander (Or leader) can instantly make a decision from thousands of miles away. Or perhaps a computer could make those decisions instead.
(2) Data-driven decision making the name of the game in most large corporations. The dinosaurs are still making decisions based on reports every week/month/quarter. The most cutting edge organizations are operating on a different level. For example, if a particular product or service isn't performing at a certain price, you don't need a bureaucracy to optimize it. Automated software could easily A/B test pricing schemes in different regions, and come up with the most optimized conclusions, all without any humans in the loop. Ride-share software and food delivery apps are already ahead of the curve here, and like it or not, this outsources a lot of people in the business of decision making.
Not to go too far down a rabbit hole, but I think that "bureaucracy ... as a gene for a successful organization" may be a gene that soon goes extinct. Computers, data, and software can do bureaucracy much better than humans can. And if the trend continues we may see the entire structure of organizations, leadership, and decision making make a complete shift from what we even expect an organization to be, possibly closer to resembling cybernetic organism rather than a government or traditional corporate leadership structure.
oops
I don’t know… Albert Einstein alone makes us pretty different from other animals.
Even though he wasn't on a team, that still is a model of communication. We call that model science.
The scientific revolution was largely a revolution in the way scientists were organised. They became connected to one another via publication, critical dialogue and such. The scientific method is, among other things a model of cooperation. How to replicate, publish, review and convince one another of things.
But if you think chimps would produce the general theory of relativity if we could just teach them to publish their thoughts, then… I don’t buy that.
Although I suppose it’s hard to say exactly what the limited level of intelligence needed to understand a few paragraphs of writing is.
The most damning trend I have witnessed first hand is that directors have extreme agency and are most of the time not incentivized to help other directors, especially if they’re under a different VP or SVP. The high-level leadership are too detached from the details (as they should be — but this is the thing that is most different in large orgs — it becomes more and more difficult to know everything, eventually becomes impossible, and then even knowing “enough” becomes impossible) to make centralized decisions so you’re left with one organization at a standstill with another, which causes attrition in the lower ranks of people who just want to get stuff done. That attrition results directly in knowledge loss and overall reduced productivity, which hampers most teams from breaking out of their set of problems. And it all just gets worse with time.
Large companies have way more guardrails and standards than smaller companies and they also change at a greater rate. Since these policies so rarely help improve or ship features, it’s just an ever-increasing source of productivity loss from the perspective of someone who wants to get stuff done. When internal politics are complained about, this is typically the root cause.
Hiring and retaining talent becomes harder as all companies grow and previous top talents becomes disenfranchised over time. Average talent hits an inflection point and then begins a descent. This just adds fuel to the fire, but I believe things would still become extremely difficult without this trait as well.
All of this becomes only more difficult to deal with over time as the company increases in size and age.
Nepotism, cronyism, etc.. all factor in as well, but those issues also affect small organizations and their impact there is even higher.
"Why do I have to fill in this mountain of paperwork, in triplicate, just to get permission to do this basic activity that is an essential part of my job, and hence the permission being granted is a foregone conclusion anyway?"
"Eleven years ago a guy named Bob tried something similar and screwed up terribly."
"But not the same?"
"No, but similar. So anyway now we have a 'process' to make sure nothing like that will ever happen again."
"Is Bob liable to make the same mistake?"
"Oh no, he's learned his lesson. Also, he was fired a year later, so he's not around anyway."
"So, has anyone made the same mistake since?"
"No."
"Do you think I'll make the same mistake?"
"I don't believe so, no."
"So why do I have to fill out the paperwork to avoid making a mistake we all know I won't make, especially now that I've been made aware of it."
"Just in case."
"Just in case, what?"
"Well, how do we know that you won't make mistakes? We have to have proof, some sort of written evidence."
"But you can't prove ahead of time that people won't make mistakes! People don't make mistakes on purpose, so no amount of paperwork will ever stop mistakes happening."
"You're just being argumentative now. You're not special, everyone has to do the paperwork."
"Why everyone? You just said only Bob ever made the mistake!"
"HR told us that we can't make rules just for Bob, so everyone has to follow the same rules. That's fair."
"But you don't need the paperwork any more, you got rid of Bob!"
"Yes, but the process has been established, it would take a lot of time and effort -- and paperwork -- to change it now."
"It's less than making everyone jump through unnecessary hoops just to do their jobs!"
"Maybe, but I'm very busy. Back-to-back meetings to discuss important matters, like how several staff are behind on their paperwork and may need to be disciplined."
etc, etc, etc...
I've had conversations like that -- nearly verbatim -- at several organisations in contexts such as:
"Why does it take four weeks to add a firewall rule?"
and:
"Why is the minimum permitted change turnaround time two months?"
and:
"Why can't person responsible for system X have access to said system X?"
It's noble to say "ouch! well let's make sure we don't do THAT again!", but yeah ; over time the processes and procedures do expand ad infinitum :-/
I sometimes refer to this as "scar tissue" - an over-response to a trauma that causes ongoing issues more so than the initial injury.
In my experience the 30% is conservative, from my measurements of large orgs. High overhead does not reduce errors, instead it increases both frequency and severity. It also kills innovation, thinking, adaptability, etc.
Before: In the past, companies would bribe bureaucrats in order to land sweetheart deals. There was not enough competition and hence the government was paying too much.
After: There is now a 10% chance of landing a contract after a month of paperwork. Only Accenture, Deloitte, and a handful of their direct competitors can afford to risk the time and effort required to win a contract. They don't actually do anything, they just subcontract everything while skimming their cut off the top. This isn't corruption however, it's just a nice legal business transaction.
I regularly see projects that might take 1 guy 6 months balloon out to 20+ people for multiple years across three orgs and costing tens of millions of dollars, all in the name of "avoiding corruption".
The best part is that because the projects end up costing so much, they have to be "taken more seriously", which means that the tender process must be "thorough", which means more papework and more managers and paper pushers on both sides of the fence.
There's an exponential cost to this a bit like the rocket equation...
My mind boggles, since it’s literally a 5 minute fix, and somewhere over our head some managers have been protecting their little fiefdoms at the expense of this poor guy that does something stupid by hand for an hour every day.
5 minutes later the fix is deployed, and we just saved 200 hours per year of pointless labor.
What you end up with is a half dozen “really nice processes” all stacked up and suddenly what took 1 week now takes 4-6 weeks to complete.
Each process is justifiable on its own but put together make no sense at all.
It's not that simple.
A bigger org is slower often because it does bigger things. No, you can't just buy a tractor even though it'll be cheaper and faster for your project than to do all the work to do internal billing to rent the tractor the company already owns.
Because if everyone did that then we'd have a thousand tractors and there's nowhere to put them, and we don't have time to sell them, nor do we know how to sell them.
And your button on your website is not a one-line change because you're contracting with the US government and the contract has ADA clauses, and it needs 34 language translations.
And no you can't make that change because last time someone improvised in this space we got dragged in front of congress and were made to promise to not do that again. So you need to follow procedures, and talk to product councel for exemptions.
And you can't change that API because 6000 developers rely on that, and they need to launch before cyber monday.
Sure, in that last case that's prisoners dilemma misalignment, but to be clear it's you who are misaligned with the goals of the business, so don't do it.
It's not reasonable to suggest a solution of "why don't everyone just stop and do what I want". If you want to tell 10k people to stop what they're doing to work on your thing, then it'd better be right. Which means you'd better know what those 10k people were already doing. And you don't.
And you can't just do it alone because it's 100 human-years worth of effort even if nobody's in your way, and you need to launch within 2 years or it's pointless to launch at all.
That's what hard about making large changes in large orgs.
Actually, correction: it's not even that simple.
In a small org, Bob down the hall enforces standards. In large orgs, they need to apply to the whole org -- so you have documents and committees. The latter are much slower and more difficult to change.
If you are self-driven and seeking to improve things and do cool work these jobs are psychologically quite difficult.
I was always frustrated, first as an enterprise guy, then as a consultant.
Eventually I started a company and quit my job, and then it clicked.
I was an engineer (high level but still) and spent most of my time helping management not look like complete idiots to their managers.
When it came to actual engineering I had to beg the help of so many shared services that I rarely knew how to proceed. It usually came down to finding someone who had been there their whole career to find out which one of their friends knew how to initiate some process...to get an EC2 instance in AWS.
High degree of chaos and stress and nothing of real value got done. To top it all off, when you find out who your customer is you realize they are guilty of war crimes.
I went there to build data pipelines for high volume telemetry. I sort of worked on that, but not really.
I met numerous people who lamented that they would look back on their career and be able to say they went to some meetings. It is not healthy, on many levels.
"Psychologically difficult" is a very diplomatic take.
I worked at a large org in my early twenties. It sucked. Now I work for a large tech company (FAANG) in my thirties. It is not nearly so bad.
The org might just be a better fit, but also I think people just listen to me more now that I'm older. I'm not just some kid who doesn't know anything. I can talk to people and get things done without fighting uphill so much.
Also, I'm more realistic. In my twenties I just wanted things to be simple and logical. Now I've accepted people aren't simple and that's just how things are, so I'm more tolerant of dealing with politics.
The goals and motivations of large companies are really different to small companies. Large companies are all about risk management - at heart they are about revenue defence - while effective small companies should be all about attack. Being disruptive in a small company is the reason they exist; disruption in a large company is confronting and to be resisted. Going from one to the other is likely to be super stressful.
Unfortunately, as I’ve experienced, you do get these same problems in small companies run by big company people who think they know better. In my experience they just add weight. They don’t understand YAGNI.
IMO the discipline and skills needed to run a startup, especially pre revenue, are very different from those needed for an established company. It’s not that one is less disciplined than the other. But the size of a small company gives them benefits and room to move that a large company can’t touch. Small companies are already risky so a bit of extra risk doesn’t count for much. It’s to be embraced.
I think that large company experience is valuable, but you probably don't want to be the startup they're cutting their teeth on. And if you're going to absorb those types of people, best to do it gradually. Make sure they don't comprise a majority of your workforce.
At least I'm not the only one to experience this! But I have no idea how to tackle it.
Mandating that all EA games be made with their own engine has caused so many delays, rushed games, and broken products by teams who have already proven they can deliver amazing games on their own terms.
There were certainly negatives, an internal transfer was effectively a new hire, just one who already had their email/git/vpn access sorted and probably already spoken to a few of the cross org teams like IT or security, but overall I never felt as much organisational friction as other teams I've worked in large companies.
Anyway, the parent company was acquired by a much larger one, that much larger one acquired a similar size company with a much more top down/unified approach to direction, and given that unified approach they were much more effective at selling a vision to our parent companies execs, so a lot of them were put in charge despite the original parent company being profitable and the new acquisition not being so.
There was a big push to standardise/centralise/etc. and in that time a few products stagnated and we lost customers. The newly acquired company has a pretty poor track record in dead/broken acquisitions from this pattern, so it shouldn't have been news that prioritizing this has costs. There was a lot of grumbling at being made move to "standardised" solutions that were suboptimal in meaningful ways, but were preferred at high levels because they were homegrown.
Everything certainly took longer in the new company, dealing with centralised, barely informed non-stakeholders who still had red tape in place to get anything done, along with a strong impulse to just copy other teams rather than come up with a solution that cleanly and effectively solves the problem. I left during this process, and a lot of the people who were the core of the startup-turned-business-unit also left, the largest group now at a rapidly growing competitor.
Yes. The last place I worked was a UK organisation that had around 5000 staff, many of whom were software engineers. The org was divided into 5 or so different divisions, each of which had its own customers, sales teams, PMs, etc. Crucially, each division had its own resourcing function, so new projects could exploit local knowledge to find engineers who had domain knowledge, and could even consider career development opportunities when picking staff for new bids/projects ('Fred is interested in trying this, he's not a perfect fit but is competent and motivated to learn').
Anyway, a new CEO (who himself had an engineering background) joined the organisation and apparently in an early meeting asked 'how many engineers do we have across the company?'. Nobody actually knew. So after a while he started an initiative to launch a company wide resource management process. Although well intentioned, it was a disaster. We ended up with global resource managers who didn't actually know any of the people concerned. A lot of our work was specialised engineering (e.g. simulation development, signal processing, etc) and it proved impossible to develop a global skills framework with all of the right categories. After a very painful year, bid teams and projects reverted to local resourcing to find the staff who had relevant skills and domain knowledge.
I'm not against standardizing things across organisations (indeed, my main function was trying to drive technical commonality) but like optimization, you have to choose the right things to standardize, and the right things to leave untouched.
The only answer I’ve come up with is that economies of scale are much more important than most people think. And the economies of scale of a Walmart/GM/P&G/AT&T are so big that they override all the petty stupidity in those firms.
Would you entrust your live saving to FinanZly? Would you go on a family vacation in a car from DriveFlow.io?
I think megacorps are mainly successful due to being the safe choice.
People love loss aversion, and you never go wrong with Bank of America or Ford.
General solution is to send "meeting notes" at the end - sounds bureaucratic, but it's exactly what you describe - a chance to document and verify "hey all, this is why we agreed on, right?" As well as provide that summary to others who missed it. Ultimately way more useful than transcripts and thus always worth it's time - yes one person will spend 20mjn writing it up, but it'll save time of at least 20min times number of people missing or distracted , plus any time that would've been lost due to misalignment or poor memory etc.
Even with a noisy transcript you at least dont loose anything, and everyone can still make notes in an async collaborative way
> stifling peoples comfort to discuss and speak openly
I know some people feel this way, which is why I wouldnt force it on anyone. But I think its a bit silly and they should get over it, were not on the meeting for appearances.
It's difference between freely brainstorming with colleagues, and speaking on the formal record that anybody can access at any time under any context or lack thereof for any purpose and with full attribution. No more jokes, informal chit chat, putting out there ideas, or sticking your neck out etc. And linking it to features sounds like a great way to fingerpoint, scapegoat and blame :-(
It's all theoretical and unlikely and paranoid until it inevitably happens :-/
(fwiw, after 20+ years in the field, I am not consenting to recording; and if I do, you'll get the appropriately shortest most formal safest contribution imaginable. Only the things I would be comfortable preceding with "it is the formal position of the company / team to...". Feel free to judge, until or unless you experience or witness the downfall that can be of sufficient magnitude to eliminate all upside of recording :)
A recorded video can be transcribed and deleted, and the transcript can be anonymised and potentially summarised by an algorithm. I think this could at least lead to better (although much longer) notes, while protecting the participants. What would you think about such tool, if it existed?
The technical solution, in my view, would a) not be worthwhile the effort to setup [transcribing and creating summary algorithms - I get that'd be fun, but that's not the same as feasible ;] and b) would have too many points of failure (now we need to both trust it'll delete and anonymize etc). If there was a COTS tool... dunno. Maybe. transcribing is already built into most web meeting software these days, and we value our meeting notes way way way way more.
Thing to understand is - making and sending meeting notes isn't (necessarily:) a burdensome, bureaucratic task that has no technical value; once you've developed the skill, you'll find you're keeping great notes for yourself. I use OneNote, I have a tab for meetings, and every meeting I go to, I keep good notes (I'm a touch typist which helps hugely, to be fair). During and after meeting I cut/trim/organize them. If I need to send them out, it's a few minutes to format them nicely for others. If I don't need to send them, a) They're useful for me to refer to and b) They're useful even if I never refer to them, as I keep track of what's going on during meeting, organize my thoughts, and ask more and better questions because I see the gaps. As we go, I keep bullet points, I order them, make them hierarchical, reorder and so on; frequently I'll screenshare them and it actually makes the meeting better as it prevents looping around, reduces uncertainty, and shows gaps / what's left much more easily.
My 100 Croatian Lipa :->
The obvious answer is, "it's my fault" and that's highly possible.
However, I never really see anybody else accomplishing such things either.
"As though that were not bad enough, in some organizations there's a separate compounding problem: limited resources. For an organization, attention resources are {time, money, staffing}. In that case you get to layer on the negotiation for resources among competing teams within an organization."
While I think the other points are valid, this process of resource allocation alone probably explains most of the problem. As an organization grows, its resources grow proportionally, but the communication required to allocate those resources grows non-linearly. The overhead needed to get $50K from each of two people is more than twice that needed to get $100K from one person. Add in the other issues raised, and this core problem is exacerbated even further.
Each one need to justify the existence of their department. And to get promoted, they usually need to hire more middle managers.
These departments will inevitably have confusing directives and even non-sensical overlapping products.
Eventually the built-up inertia becomes too large.
The best easy book I can recommend about large organizations is Smith & DeMesquita's "Dictator's Handbook" because it provides a practical applied model for determining incentives between groups. Reality is, the people you are dealing with are running their own mental version of the model, where the coalition that keeps them employed and prevailing is made up of essentials, influentials, and interchangeables. They intuit a salience matrix of stakeholders (decision power, vs. how much stakeholder cares), and while everything is dynamic, it is usually within these parameters. It's a tool that can provide fast insight into group/project/org behavior pretty well.
In a truly large organization, it takes institutional knowledge to identify who will have to make a change, who maintains it long term, and who will be impacted.
Then you need to convince each group that the change you want is desirable. Sometimes this is as easy as convincing the person in charge of the group. A large one-time change is often easier than a small change to daily procedures.
As your knowledge of how the organization operates grows, so will your ability to find the necessary people and phrase things in terms of their interests. In a thousand-plus person corporation, I would expect this to take at least a year or two.
The increase in levels is like a game of telephone and reduces the fidelity of the message/strategy/goal from executive down to line level team members.
"Why are large companies so difficult to rescue (regarding bad internal technology)"
http://www.smashcompany.com/business/why-are-large-companies...
For example 1 hour ago I was discussing with my business counterpart about a project we are working on. We just got approval to hire 4 more people (that means we have to hire them, it is not optional) and half of them are diversity targets (meaning: skills are irelevant, we will get them anyway) and the other 2 will be hired and given to us without knowing what are the skills we need. Based on such organization dynamics, we plan to be sufficient without these 4 people and if any of them has any useful skills, it will be a nice surprise. At the same time, we need to give them some work to do, so we need to invent some roles that should be as harmless as possible.
This is just a project, in my department of ~100 people about half are just mouth breathing, but it makes it a lot more complicated for the ~ 20-30 that do work to just do their work: some of these people have invented roles to rubber stamp steps in the projects, so they are adding 10-20 steps and a few days of work just to justify their existence. With the department half the size, everything would be much simpler, faster and even more productive.
Thinking alone is easy. Thinking with someone is hard. Now keep adding people. You stop thinking.
Same with hearing. Listening to one person is easy. How about two people. If they don't share their time, you're listening to two people talk at once. Same with actions.
With more people, thoughts become more ideological and less mobile. Same with speech, same with what they listen to, and same with their actions. And each individual is not only physically a fraction of the group, but so are their senses. And to act humane or to simply perform on an individual level is directly hindered by this group dynamic. Individuals then become lethargic, and begin to experience greater stress and pain. You could also think of this as the Group Flu. I know I've experienced it. It's the feeling you get when you know you need to talk to someone higher up just to be heard. It's the feeling you get when you're talking to a customer service rep and you know you're not talking to a person.
Now take a large organization and we have a group of groups.
We already know one group cannot do it all, and one group is too inefficient. But if we're just creating more groups, the problem persists.
So what you need is to undo the group dynamic. But then again, with a big successful organization, maybe change is what you don't want? Ideology is what you do want, and suppressing the individuals leads to your ongoing success.
Say each member has a point, and additional members have counterpoints to balance the group.
What exactly happened? As a member with a point countered, you've effectively been silenced. You have a group where extremes are neutralized. That is either a set goal by whomever controls the group, or works towards the greater ideology that becomes the group voice and group think that no one member can overturn.
You don't need to be in a group to have additional data points.
Thinking alone might be easy, but not thinking in a group is also easy. That's what most members end up doing.
Also you sounds biased against groups. Sharing points of view don't equate being silenced. Having more data alone still means you're the sole interpreter, you perspective / understanding of the world might still be too focused on part of the space. Others can widen the angle, change your position. Of course politics can distort things too, but that's outside my comment. I was talking about conceptual group, small, free of agendas. Just a bunch of people saying what they see. Not every group is ruled by someone (even though most of the time there are such implicit social structures, I agree)
I am biased against group think. But also group speech, group emotion, and group everything else, as they are all inferior to their individual counterparts. Group emotion is a mob.
If you've been in a group where the group changes your independent behavior, or you relinquish your decisions to the group, you've just downgraded yourself.
And if you look at all the stupid in the world, most are done by groups.
The article goes on to describe solutions without actually explaining how the Prisoner's Dilemma operates in large companies.
This makes the article less valuable than it could be because it interferes with the reader developing a mental model of the problem.
For a take on this that does present a very clear (and possibly actionable) mental model, see "The Gervais Principle":
https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
The idea is based on the axiom that "organizations don’t suffer pathologies; they are intrinsically pathological constructs."
Lots of make work, "lawyers don't get paid to simplify". Growing from small to large company you can end up with a lot of bureaucrats that tell you things like "this policy is what big cos do", but is really entrenching their position and keeping them employed. Creates adversarial relationship between devs and IT/ops/sec.
They don't have it easy, because one leak or bad PR due to data and they're shot into space.
Best to move to small effective teams, move fast, and show value. Often by currying favour of C/SVP you can get barriers removed fast and get things done.
Ecosystem is large, so find your niche.
Seen entire teams destroyed by well meaning PMs and other idealists implementing a "system" leading to dev/project slowdown and even abandon. Adding a couple check boxes to a JIRA process seems simple, but every choice is multiplied by the number of developers you have. You can see this in places that implement scrum poorly.
I'm reminded of this slide: https://speakerdeck.com/holman/how-github-uses-github-to-bui...
Overly complex security policies, JIRA check pointing, and constant meetings can grind you up. A lot of it is mistrust of employees, or employees also don't have enough context to do what they need to do.
It is also entirely possible that I am a particular type, attracted to small/medium fast movers, and then I get tired of battling security/ops/managers for access to systems/data I need to do work, or the new sets of flaming hoops to jump through to do completely mundane tasks.
You can always roll the dice! Bonne chance!
Once you have groups with a leader that are in charge of [something] narrow, they start trying to make [something] more important, more complex, etc, than it should be. For various reasons...exposure for career progression, empire building, etc.
Then, when you need to get something from an idea into production, you're dealing with some large number of these groups. All of which have some form of demand management (jira tickets or similar), some form of requirements you have to meet, different ways of interaction, and so on.
And, some of those groups have to work with other groups to do "task a" that you asked for. So you get this complicated dependency graph of tasks that's not even visible to you.
1) Uncertainty. On larger things, success is often uncertain (e.g., on new products, etc.), so the final consequences of the decisions are not known in advance
2) Generally, running processes well across an organization is not something that comes naturally. It needs investment in skills. For example, most meetings should have an agenda and should end with next steps as well as agreed and assigned actions. Tracking and planning similarly need effort and skill.
It might very well be that on top of it the incentives are wrong for people to take risks in (1) or developed skills for (2) or even to cooperate in the first place, but most people are not naturally good at administration to start with.
I think the skin-in-the-game is very significant as the OP says. In a smaller organisation, you might have a person who does several things and one of those is HR. The existence and success of the company depends on hiring well so they do their best. Also, the more efficient they can be (sometimes the worse they can be) the quicker they can get back onto other things.
In a large corporate, you now have an HR team. There is usually much less feedback on performance and it is easier to blame the org or the market for poor hires.
Another issue is that most employees like most people are only average. A few are terrible and a few are excellent. We can be picky in a small company and find that great person with alignment to the vision etc. What about in your team of 10? You aren't going to find 10 great people so you get a whole range and now some are doing much less than 10th of the work of the team so the excellent people are now picking up the slack.
Maintaining multiple communication paths is hard and risky. I worked with a Bank where each requirement spec was signed off by about 15 people (compliance, legal, marketing etc.) how complicated is it to make 1 change? Holidays, other priorities, change of staff etc. can make the smallest thing take really long.
Ego: Some people simply cannot compromise. If you are the CEO, you might get away with it but what if someone else is throwing their toys out of the pram because e.g. the legal team need your product to look terrible for compliance reasons? Things you should just accept take weeks or months to argue out.
Specialism: The head of e.g. marketing should have the final say on all things marketing ideally. But in many large companies, they might have a disagreement with people who might or might not know more about something but vocalise it anyway. This is magnified if you don't really trust your department heads are good at their job.
Oh, so many more things.
Is there an example out there of a large organisation that does work well or are there just too many variables to make large organisations work period?
I am guilty of that myself. In a small team where I know no one is coming to help me, I step up and become 10x productive.
Cf. https://psycnet.apa.org/doiLanding?doi=10.1037%2Famp0000469
I think the author is conflating prisoner's dilemma with the communication challenges in a large organisation, which arises naturally from their sheer size (as other's pointed out, these companies work in a bazillion products in parallel, not in a single one as the author claims).
As an actual employee of one such company, one of the ways they reduce the impact of this is with Principal/Staff level engineers. This is similar to the concept of hero mentioned there. These high rank ICs oversee projects that span teams or orgs, and align them to achieve global positive outcomes.
Thus, in a large organization, you can only improve the ability of individuals to do what they actually do, which may or may not improve the ability to the organization to accomplish its nominal purpose.
You don’t realize it right away but your processes, methods, and networks are breaking down with the new scale and what was once effective stops being so. But it’s hard to detect right away - it’s the “boiled frog” problem.
The company defines themes or goals or priorities and small cross team groups are assembled but it only gives the illusion of being coordinated. You still have a bunch of teams vaguely working towards solving a greater problem but not actually working together.
There’s a general sense of “losing our way” and disillusion from the veterans who don’t recognize the company anymore and attrition sets in causing even more disillusion and attrition. The company is growing headcount but it seems like everyone is leaving.
I think this is the point (hopefully we’ll before) the company realizes the old ways that got them there don’t work so well as to get them to the next place.
What I’ve found works is senior management that can not only identify what the company priorities should be but also understand that every team should participate in it and allow the teams to determine how to execute. Defining programs and having coordinators work with the stakeholders helps scale communication and coordination. It’s another layer that wasn’t needed in earlier days of the company but becomes critical in my experience.
And the cycle begins again. If the company does achieve the next level of scale similar problems will set in as coordinating 3000 people is just as different than coordinating 15,000 as it was from 700 to 3000. More layers of indirection are required.
What works for 16 works up to 45/50 (3X15).
Next breakpoint at 150 (3x50), the 'classic' Dunbar number.
likewise at 500, 1500, 4K,...
and it all comes down to communication bandwidth. The same scaling laws which lead to the cortical columns in the visual cortex specializing into different kinds of signal detection (edges, motion, ...). You can only maintain close connections with so many people.
One thing I've been thinking lately is, what if everyone had as part of their "goals"/"performance review" structure -- how have you helped another team (in a non-close branch of reporting structure) do their job/accomplish their goals better?
It's probably naively optimistic to think there's a bureaucratic solution to bureaucracy though, which is basically what that suggestion is.
The article doesn't dive much into complexity of problem. Prisoner's dilemma doesn't have much place in traditional organisations. It is more subtle than that.
Similarly if someone worked their way up by creating a really important bottleneck everyone has to go through rather than facilitating productivity in others, the company and many other employees would benefit if he changed but he wouldn't benefit from losing his reports and thus well paying job.
This is different from prisoners dilemma where everyone benefits if everyone cooperated.
Read Working Backwards and it's clear Bezos got it.
Obligatory Tweet: https://twitter.com/RichRogers_/status/1159872097205805056
The reason the businesses are run centrally planned is because the alternative is introducing internal markets within the company which create high internal transaction costs, brutal politics, and duplicative effort.
At my company they reward "impact" and base promotions on that. The problem with that is now we have a zillion internal tools that need to be maintained. Lots of things were indeed built, but did they make things better for the rest of the org?
If they want to dig into why large organizations have bureaucracy, or what that bureaucracy is made of, then a different question can be posed. The prisoners dilemma is only a small part of this.
Why is everything so hard in a large code base?
Same reason.
To me, it's simpler. The people running the organization put the wrong incentives in place. No need to complicate with individuals playing one-shot games and then finding a way to a coordination equilibrium as a solution. In large universities at least, the problem is that most of the people running the outfit start as faculty or low-level administrators, and they move into high-level administration positions without any understanding of how to do their jobs well, or for that matter, what it even means to do their jobs well. Most do not care. As long as the higher paycheck keeps getting deposited in their account every pay cycle, they feel they are doing their jobs.
this assume that the organizations are mishandled. you don't need to posit such a requirement, it's even simpler, as the organization grow, the cost of mistakes increase, while the margins decrease. this is what requires controls structure to be put in place, slowing and weighting down operations.
it's not a bad thing either, just the result of doing business at scale. many startup that find themselves at a significant market size quickly and quietly drop the "move fast and break things" culture, for the very same reason
Bureaucratic inefficiency is a huge drag on profitability. But it becomes invisible in a way that obvious mistakes and failures don't.
If a new project fails, it's an obvious failure. But there's no business metric that tells a company how much it's wasting on pointless processes and paperwork.
So even though the cost of wastage can be bigger than a high profile failure, it never appears in the accounts.
There is a flip side, which is the rando CEO who goes on an uninformed acquisition spree and blows $x bn on companies that don't integrate and end up being a huge net loss. Or starts numerous reorgs and rebrands which add more friction to no benefit.
You gain true expertise only by understanding everything yourself and then you are also extreamly fast at accomplishing your tasks.