The solution to which you speak is microservices. No, not some misguided multi-process IPC technology model that seems to have tried to usurp the name, I mean real microservices. The people model. Where each 4-8 person team continues to act if they are an independent businesses, even if they legally work for a single organization.
It's not particularly difficult. The macro economy, which operates at an unfathomably larger scale, shows how easy it is. But, indeed, it can be uncomfortable for some people to not feel like they have full control over an organization.
Similarly, building a whole product out of pure microservices with teams that barely coordinate with each other beyond maintaining their APIs doesn't really work. There is very much a place for such microservices for certain well understood problems (authentication being probably the clearest), but for most domain specific work you need far more coordination than that to truly evolve a complex product.
For example, you can't build a computer game in this model. A modern 3D engine requires far more than 4-8 people, and there is no clean separation between pieces to afford this. And even if there were, there is no One True Way to build one: the teams designing the 3D engine need to communicate with the teams building the game engine and the game designers themselves and the artists to achieve a cohesive whole. A 100-person project just can't (always) be modeled as 15 4-8-person projects.
Sure, just as they need to communicate to the farmer what food needs to be grown in order to ensure that the right food shows up on their dinner table. These are solved problems, though. We already know how to do incredibly complex things at scale.
In addition, most companies that package food in various ways, including major restaurants, either own or contract farming business for specific quantities of specific crops ahead of time, spending large amounts of time to negotiate contracts to this effect, and sometimes working even closer when a particular result is required.
If you find yourself craving Mongolian food one night, you may often simply find that none is available, at any price you are willing to pay, unless you worked with someone to make some months in advance. And conversely if you open a Mongolian restaurant, you may just not find any customers for it and waste significant money on it. (substitute Mongolian with any other food that is not commonly eaten or even thought about in your area)
And again, most real companies own most of their supply chain, they don't depend on a hundred different suppliers at every level like you seem to be suggesting. And they do this because it's FAR more efficient to have a large monolithic team build something together than to rely on dozens of small teams competing in a market to build the same thing.
Obviously. Like we said, no 4-8 person team is building anything by themselves. For one, most obviously, they'd starve to death if they even tried. Integration and cooperation are literally what defines an economy.
> most real companies own most of their supply chain
As opposed to fake companies, that have no supply chain, with them being fake and all?
> because it's FAR more efficient to have a large monolithic team build something together
How can you know? The problems needed to be solved to see that through were stated to be unsolved. Was there some breakthrough in the past 10 hours that you haven't mentioned and the rest of us missed?
I should have said "companies that can afford to".
> How can you know? The problems needed to be solved to see that through were stated to be unsolved. Was there some breakthrough in the past 10 hours that you haven't mentioned and the rest of us missed?
Because I can look at the price of goods versus their cost. A company that owns the production chain for a sub-component is paying the cost of that component, while a company that doesn't pays the market price.
Also, markets only work if you have competing suppliers for the same product. This in itself is extremely wasteful for software, so trying to maintain an internal market of software where multiple teams build the same service and compete with each other on who has a better version is crazy when there is any alternative.
Again we can see this from the software market: most companies not only don't build an internal market in house to have, say, the best OS, but they actually found that even supporting an external market is wasteful. Instead of buying Solaris or Windows or OS/2, they started collaborating on Linux (and clang, and MySQL, and Kubernetes and others).
But these are not built as a marketplace of microservices. They are built as monoliths with large teams working to add features and collaborating relying on a hierarchy of maintainers to merge them together in a cohesive whole, or on a more elaborate committee system.
So again, I don't see any proof whatsoever that building a company or a product in general out of a marketplace of dozens or hundreds of very small teams has ever worked beyond maybe some niche cases.
I'm not sure I follow. Let's say Acme Inc. produces a widget that is a subcomponent of their larger product. When that subcomponent is within the control of a team of four who works for Acme Inc., the company has to pay the market price (meaning a markup margin?), but when that team is joined with another team who also works for Acme Inc., then they only have to pay the cost?
1. What difference does that make? Even if a markup really was paid on paper, it's just to Acme Inc. itself. You haven't changed anything, practically speaking.
2. If an external entity is willing to pay more for the component, which is what I think you are trying to suggest with market price, Acme Inc. is paying the opportunity cost when keeping it internally and thus is still paying the market price. Are you under the impression that there is a free lunch here? There is not.
> trying to maintain an internal market of software where multiple teams build the same service and compete with each other on who has a better version is crazy
Just as it would be crazy in the macroeconomy. The idea that competition is necessary for markets to work is misguided. You don't need a second road running parallel to the first. One road is just fine. Competition only comes into being when someone thinks they can serve the customer better by doing things differently. Which is a useful property of human dynamics, to be sure, but sometimes things are already as good as anyone is able to imagine.
> they started collaborating on Linux
As individual teams, with another team headed by Linus who accepts the services of those other teams as seen fit. Another great example of the service economy in action. No need for a monolith – if such a thing were even possible, but as pointed out at the beginning of this, of which I agree, we don't actually know how to build monoliths, no matter how great they sound in theory. There are outstanding problems not yet solved.
> They are built as monoliths with large teams working to add features and collaborating relying on a hierarchy of maintainers
A monolith, but also hierarchal? Uhh...
> I don't see any proof whatsoever that building a company or a product in general out of a marketplace of dozens or hundreds of very small teams has ever worked beyond maybe some niche cases.
I suppose you can take the perspective that nothing works, but given the outstanding problems not solved to allow anything else, something that doesn't work beats nothing at all. Worse is better, perhaps.
The way I see it, the discussion began by saying that any team (but in context, referring to a whole company) that grows past a certain size needs a traditional hierarchical system of layers of management to function. Then, someone said that a team/company should just not grow past 4-8 people, and thus you avoid the need of hierarchical management.
My point is that this idea makes no sense. A company can't function as a composite of hundreds or thousands of sled organizing small teams without any management to coordinate them.
My understanding is that you claimed that such a thing is possible, and that in fact it is how the free market and thus the economy of most nations actually functions, without any hierarchy.
I was then rebutting this by showing that the economy does form hierarchies (with large buyers essentially viewing small sellers as subordinates, not independent market actors, when they're not outright buying them).
So, I would argue, the economy is not generally organized as a bunch of independent small orgs, but as a series of huge conglomerates that either own their suppliers, or at least dictate market terms for some period with contracts. The idea of a free market where buyers and sellers meet and negotiate prices based on supply and demand is largely a myth, with some limited exceptions.
As such, the hierarchy in how Linux is built is in line with my original idea: not supply and demand based, but instead organized in a relatively rigid hierarchy.
In support of why I'm claiming that this system of hierarchies arises due to efficiency, I gave the example of how much it costs a company to buy a piece they need versus how much it costs them to produce it internally. This is meant to show that a supply&demand based self-organizing market (the flat, many small teams without managers model) is less efficient than an integrated supply chain (the top-down hierarchical model, with layers of management).
There is a hierarchy of sorts (realistically, something more like a cyclic graph, but close enough for all intents and purposes). The customer gets food from the grocery store -> who gets food from a wholesaler -> who gets food from a processor -> who gets food from the farmer. But at no point, despite being an exceptionally complex system, do you need to exceed a small number of people per team.
> thus you avoid the need of hierarchical management.
Right, so I expect this is where the miscommunication is happening. You're not wrong that the consumer of food manages the grocery store. Absolutely that is true. Of course it is. And in that sense, some kind of hierarchical (or whatever we want to see the graph as) management will emerge. That is a necessary precondition for an economy to exist.
Contextually, though, management was seen in a more vileness way, which does tend to happen when communication overhead starts to overburden the people, leaving actors to be faceless "cogs". This is what small teams avoid.
> but as a series of huge conglomerates
Which don't operate as monoliths. Internally they function as a series of smaller businesses, and inside that even smaller business, until you whittle the teams down into a small number of people. Maybe not exactly 4-8, but certainly within that order of magnitude. A person can only communicate with so many other people before all hours of the day are exhausted.
> The idea of a free market where buyers and sellers meet and negotiate prices based on supply and demand is largely a myth
Okay... Sure? Is there some reason for us to randomly start talking about a "free market" here? It was never mentioned before.
> not supply and demand based
I don't follow. Supply and demand is not some kind of foundation on which something is based. It is observance of social behaviour, typically that of humans, but applies even to animals. Supply and demand is every bit as much present in the development of Linux as everything else humans do with other humans. It is fundamental.
> but instead organized in a relatively rigid hierarchy.
Instead? They are not at odds with each other.
> I gave the example of how much it costs a company to buy a piece they need versus how much it costs them to produce it internally.
Which didn't seem to serve a purpose. If all else is equal, it costs the same. Presumably you are referring to a case where you can buy a subcomponent externally for $10 or produce it internally for $5? But given that buyers are willing to pay $10, you lose $5 every time you keep it for internal use. So, in both cases, it costs you $10. There is no free lunch.
> This is meant to show that a supply&demand based self-organizing market (the flat, many small teams without managers model) is less efficient than an integrated supply chain
The data doesn't support this, though. Statistically, smaller businesses in coordination with each other are more efficient. You'll notice large businesses only become large when they can build some kind of moat. In the case of software, that mostly means hiding behind copyright and patent protections.
I think this is the key to our miscommunication. I don't call that management. That is what I am referring to as the free market model of self-organization: one customer doesn't dictate what the convenience store does, they only give very indirect signals that the convenience store can choose to respond to or not (say, this customer didn't buy any cheese; they can drop the price of cheese, or drop cheese from their inventory, or keep offering cheese and hope that others will buy it).
In contrast, a rigid hierarchy is what happens inside most corporations: your manager tells you to make cheese, you either make cheese or you get fired. Your manager is telling you to make cheese because a different team is planning to make cheese toast, and the manager wants to ensure they have cheese available. I would refer to this as a planned economy model, and call this business a monolith in the sense that, while it has internal parts, they are "fixed in place", they don't get to decide what to work on.
Similarly, Apple's relationship to its suppliers is closer to the planned economy model than to the free market: Apple doesn't go shopping around various suppliers and getting some milk here and some cheese there. They investigate some market and pick some suppliers, and form contracts with them that demand specific amounts at specific times at specific prices. After signing the contract, the supplier is not free to take Apple's feedback or balance it with their own desires or other customers: they have to execute on the contract or risk penalties and fines. They are now effectively under Apple's management, for at at least some portion of their business, and not operating as an independent market agent that uses pricing signals to elaborate their strategy.
Of course I do agree that at no step of the way is there a 1000 person team working on something. But neither is anyone else: the discussion we are having, as I see it, is whether you can organize 1000 people into many small teams that only coordinate voluntarily, where team A decides what to work on themselves and team B decides whether to use team A's work or not; or you need to impose a more rigid management hierarchy that tells team A to work on this and team B to work use team A's output.
The second model is FAR more pervasive in how companies organize internally. And even in the economy at large, contrary to what you're saying, large organized groups like Samsung easily outcompete a network of small businesses. There are almost 0 industrial consumer products supplied by a myriad of small manufacturers, in fact. From phones to cars to appliances, everything is produced by huge conglomerates. And they don't have any special moats, only money and the inherent efficiencies of being able to run a planned economy internally.
Why would they prefer the closed market over the global free market?
On the global free market they are not constrained to 8 employees, they can have 9 or more.
But back to the topic at hand, there is an impossible level of communication overhead to burden if you try to keep millions (or even tens) of people on a single team. Like the original comment pointed out, we've never actually figured out how to do that. It might be a good idea in theory, but the problems remain unsolved.
The workaround is easy, though: Just don't do it. Do what everyone else does instead.
Now I work at a place with not enough managers to go around, and my team basically has 25% of a PM's time, and no dedicated department head (last guy quit, leadership has not decided whether to rehire). It's absolute chaos. No shit shield between us and leadership, nobody to soak up all the excess meetings required to understand what's going on throughout the company, nobody with political weight to throw around on our behalf, nobody to persist a long term vision for our department. We're working twice as hard to tread water, and creating a lot of technical debt while we only barely stay on top of our basic responsibilities.
It may be that, at an organization that is set up not to have managers, it's better than just being the one group in an organization that doesn't have one. I admit that's a possibility.
But, at another organization, I've actually had great experiences with managers, who cleared the road for me and made me feel like I was doing better work more easily than I was without them. So, on the whole I am pro manager, so long as they are very very very good at it.
That is a big "if" because most of the managers I have seen are on the spectrum of "ok" to "very bad"; the really good ones are rare and usually move up leaving spoiled employees that have to endure being managed by the norm of "ok" to "very bad".
In nearly all of my workplaces, it's the upper echelon that has typically been painfully incompetent (poor financial strategy, too much micromanaging in areas they don't understand, etc.), and I've been mystified how they managed to hire and retain such talented middle managers. I'm sure it is survival bias, though, because a company can only scrape by with bad upper management if middle management is unusually good.
Thankfully, my current and previous job were both at companies where the C-suite actually know how to do their jobs well. Even though I've some had fantastic managers at otherwise poorly-run companies, that's not enough to keep morale high. No matter how much your manager supports you and encourages you, it feels like you're wasting your life when you feed your most productive working years into a company you don't believe in.
One reason could be the type of industry one may be in; banks: loads of chair-warmers, mainstream tech: probably better because the scope of hiding in hierarchy may be less (not sure, just speculating).
I've opened your profile and then your website just to find out where is this magic place is. How fitting that it turned out to be on other side of the globe from my place... But good for you :-)
In a functioning workplace, everyone agrees to do their work because it is part of the social contract of working on a team. They don’t need to be told what to do. If someone is falling behind, they’ll talk to other team members and work together to get back on track. You are there to help your coworkers, and they are there to help you. Someone doesn’t have to be your manager to make sure projects run smoothly; everyone can take turns in this role if they feel like it’s something they’re interested in doing and are competent in that role.
It’s only through the distorted lens of corporate ladder climbing and backstabbing departmental politics that the idea arises that you’ll just hire untrustworthy people and then beat them into submission by making a workplace into a prison.
Hire some recent grads and then make that statement. Not all require checking in on, but often entry-level, junior folks do. Don’t ignore them and don’t hold it against them. You, I’m sure, had someone checking in on you when you first started. Not in a “Are you at your desk” way but a “Are you able to complete the tasks? If not, how can I help you?”
Sadly the last few years with remote work and layoffs have made it so companies get rid of those who are “just doing the job” and keeping those that are “always the job”. Brutal.
This is assuming a lot. Whose responsibility is it to get to a "functional workplace" and whose responsibility is it to fix it if it's not functional?
And assuming that there is a "social contract" that everyone that's in the workplace buys into is also assuming a lot.
Just because they have been given some set of rights and social power.
Weve seen how they organize and adapt are what classifies the innaye "good"
What you described smells like communism - might work on small scale in some isolated cases, but usually doesn’t.
One is interfacing the work of a group of people with the needs and wants of others.
Another is doing administrative things like who has done this compliance course or who needs a security authorization.
A third is allocating credit and blame to group members (aka compensation).
A fourth is ensuring that the work product itself is on track (aka technical leadership).
A fifth is ensuring that schedule commitments to other stakeholders are honored.
If you take out the third dimension (credit and blame), then even if a single person happens to do the other 4, they probably won’t feel much like what you’d call a “manager” today.
This is a long-winded way of saying that the various things a manager does can be disaggregated and either spread around the team, eliminated or given to a different person.