The magic of small engineering teams
newsletter.posthog.com
newsletter.posthog.com
In a large org, small teams have no weight. Actually, we have wait - we have to wait for everything: devops resources, marketing resources, even infosec resources. We are too small to notice, not important enough to get quick attention (never mind that our small team's product is profitable).
Anyway, after a month or two of this I started catching significant flak because my team was nowhere near as productive as it used to be. Complaining about slow POs was met with "maybe you should plan better". Complaining about design reviews for one-off boards that would go from idea-to-problem solved was met with "you're engineers, you need to document your work". It was painful and I came very close to resigning.
What ultimately worked, though, was figuring out a catch phrase that spoke the language of the new people: "accountability without authority". This ruffled some feathers but once I repeated "you are trying to hold me accountable for delivering a $500,000 project on time but are not giving me authority to buy $40 worth of stuff to execute on that" enough times it finally got through and the system started to change. But man did it suck for a while.
Imagine, holding a 2 hour, 10 person meeting to decide if an additional one time $1000 cloud expenditure is warranted. Yet, this happens all the time. I can assure you, that this is not how shareholders or customers want you to spend your time.
A traditional makefile like setup would only build one compilation unit per compiler invocation.
Each compiler invocation requires a license checkout over TCP. This was badly impacted by Nagling and delayed ACK. The license server was not managed by engineering rather the IT department so we were unable to get them to enable TCP_NODELAY.
Multiprocess scaling was challenging - you could give the compiler multiple compilation units, but sometimes doing so would effect the order of things in the object file, now you lose reproducible builds. These seemed to always be minor things that didn't significantly effect the executable, but rarely you'd crash the compiler and all its jobs would need restarted on the next rebuild.
For those of us batching our compiler invocations for local development builds, the last straw came when an ops team unfamiliar with the tools was put in charge of license management. They started kicking out license checkouts beyond a few seconds in the name of fairness. Build times increased further, but license checkouts per second went up so they thought they were helping.
Eventually I realized it wasn't the 1970s anymore and since I no longer smoke I was bored out of my skull just waiting for timesharing computer to compute again.
Several teams did move to GCC when it got bad enough - hand tuning loops and such when needed.
Probably shouldn't've had so much code in headers - APIs had a lot of dependencies and when I needed to change CompanyName.h, I was in for a few 90%+ rebuilds.
It was a combination of: not my problem, nobody's problem, I got mine, codebase is already like that, death by a thousand cuts, we've never better so how can you say this is bad, network latency.
I don't have to imagine; I've had it happen within the last month. IT was complaining about dev's storage usage. It would have been cheaper for them to buy 10x 20tb hard drives than pay for the meeting that was called about it. Luckily, when that was pointed that out in the meeting, the dept manager agreed and ended the meeting shortly after.
I suspect accountability without authority might not be as accurate as one might seem when there exists middle management. You might be accountable but your boss probably is more so, thus the reluctance in giving full autonomy. They of course won't be able to take credit in that case either :). Perhaps an overly cynical view I admit.
"The optimal amount of fraud is non-zero"
The optimal number of (minor) crashes when training for a mountain bike race is non-zero. If you never crash, you cannot be sure you were going fast enough around that turn.
People are risk averse, so a known cost is preferable, even if expectedly higher than dealing with abuse. This is in part why insurance is so profitable.
If it's an error, you correct and accept errors as the cost of doing business.
Tragedy of the Commons Ruins Everything Around Me.
Those who act selfishly poison the well for everyone.
That was the most interesting thing about this. I still reported to the CEO directly, but some of his tasks had been delegated to finance and project management teams. So the project management folks were running their MS Project models and I had to keep them updated weekly on where everything was at, and if I spent money out-of-pocket without having it pre-approved by finance my expense claim would be denied.
Even if nothing changes, it still has therapeutic value.
Some organisations (like the one I'm working for) seem to be run like a dystopian daycare. Stay in your box, be "respectful", don't complain about your terrible manager, don't complain about the hoops you have to jump through to get even the most simple or meagre of resources. The internal communications are heavily monitored and that has a chilling effect on cross-team collaboration.
They have even weaponised their "culture" in a way that coddles the offending teams in the structure and still direct blame for missing deadlines at the victims rather than the perps.
There is no fighting this. My small team are all aggressively looking for new jobs which is a shame because they are a tight crew with great skills who have now been beaten down as far as they will go. A great loss to the organisation when we walk too, all that institutional knowledge gone.
I semi-jokingly-but-not explained to them what a Shit Umbrella was and that while all of this was going down I regretted not being as available to them as I had been, but that I was doing my best to give them an environment where they could keep doing the amazing work they'd been doing without getting dragged too much into the bog with me.
There are tradeoffs, but I'll never go back to big organizations.
That's not the main reason I'm quitting this week, but it definitely helped the decision!
I’m not very financially motivated. I’m already making more than 90+% of the US. I just can not fathom the level of greed I saw hiding behind people’s eyes at Google as they try to build empires. I’m in Silicon Valley to learn about and work with computers. I get to work a chill number of hours while still learning every day. If you could get a lot of employees like that you’d have a huge market advantage.
I feel like the insane ponzi scheme of our housing market does this to everyone eventually
But honestly most of the time I’m able to take my best thinking hours of the day, put those into the job, and get 90% of the output with much less than 90% of the time. The worst thing you can to is try to fill the time and create a bunch of tech debt. Artificial time scarcity keeps things lean.
That’s very well written. Poetic even.
> I’m in Silicon Valley to learn about and work with computers.
The industry used to be more like that. Never entirely, of course - Microsoft made a bunch of people millionaires in the 80s and 90s, turned some heads. A few others did too.
Even in the early 2000s almost none of the MBA types were going to the Bay Area after graduating.
But things changed. It basically had to. There were too many companies making too much money - and making their employees too wealthy - for people to ignore it. Zero interest rates added fuel to the fire and everyone who could jumped on it. Late stage capitalism, everyone gotta get theirs. And here we are.
There's so many of these people that it's really hard to avoid working with some of them. I consider myself extremely lucky because I started a business and when these people show up, I can choose not to promote them, or straight up fire them, or find some other way to show them the door. But the broader ecosystem we're in is something I have no control over, and it has clearly changed, so that it's not just filled with people who don't know or care about the tech, it's run by them (and in the long term it shows as the tech degrades).
If you want something done right, you have to do it yourself. That goes for hiring and culture in addition to technical implementations.
Large orgs tend to have a Roman Empire feel to them for most of the people, most of the time. You're stuck out on the edge of the empire, minding Hadrian's Wall, proud to be a Roman but also far, far away from any influence over or interest from the Emperor. If you are lucky they leave you alone. Occasionally a missive arrives from Rome that you have to decipher and implement no matter whether it makes sense or not. Fun times.
You absolutely know you wont ever build a lasting bridge with a six person team, but you could ford a river. This type of stuff is ok when you are in startup mode but doing this long term at big companies creates a lot of existential risk that could be mitigated by better planning.
This is a tradeoff some people make knowingly (like the posthog post clearly lays out) but as usual this "2 pizza team" is cargo culted way too hard.
In many cases I see the product folks with the same vision regardless of the team sizes or org fit.
You could be the smartest guy at the company, have the most well rounded team, have a PM that orchestrates people like a symphony conductor, and have a huge idea that would make absolute bank for the company but your idea won't matter for shit if you can't sell it.
N choose two is n*(n-1)/2, which yields the right answer
In asymptotic analysis, they are the same. Except, that analysis does not make sense, asymptotic refers to the growth of the function as values approach effectively infinite
If you are essential to the company, it is on the company to give you the resources you ask for, or they are asses. If you have to waste time fighting for them, to the point where work can't get done, that is their problem.
If you are non-essential to the company, then it is on you to start looking for a place where you will be essential, either internally or externally.
> Startups ship more per person than big companies – everyone knows this. But how do you retain that advantage as you scale?
You can't! Not really. If you could, we would see companies doing it but...we don't (barring some yet undiscovered engineering process).
Everything you ship, by definition, has an ongoing maintenance cost. The more you ship, the higher your maintenance burden. Over years, this grows and grows. Output (as defined by "products shipped") per engineer must go down, because more and more engineers must be dedicated to maintenance work.
Now, we've gotten really good at disguising maintenance work as product work/shipping things. But it's not reality. Even this post makes this mistake by referring to "data warehouse" and "analytics" as "products". But customers don't care about your data warehouse or your job pipeline.
> Right now, for example, we're in the process of scaling support by moving our support engineers out of the customer success team and into a new customer comms team.
This is not product work. This is maintenance, and is a literal example of the type of thing that larger companies have to do just to maintain their existing products and contributes to lower product output per engineer.
And it comes down to hubris.
Too many people (in finance, mostly - which drives the rest: entrepreneurs, "big" managers, etc.) in the industry (like to) think that maintenance is bad, because it's not new or growth-capable (which is where they expect to get more, and more, and more).
But maintenance/routine is actually most of the work for everyone, everywhere on this planet. Which is good work.
You can/may outsource, or automate (it's the same actually: getting the same output for a lower cost), it. With the consequences it comes with.
You can/may prefer to be in the 1% of innovation too. But once you innovate (on) something, you've got to maintain it. Someone has to.
The good places where to work are those that admit, and implement (in their structure, in how the manage people and their expectations), that maintenance is most of the work, and that innovation is the small part of it.
It's like in art: most/99% of the time, you practice, you learn. 1% of the time, you get to do/find something great out of it.
Which leads me to the dullness of generative AI: you can't have the 1% if you don't go through the 99% - and you may not understand this if you never tried once to get through those 99%.
Depending on the part of the software industry you are in, customers might care very much about data warehouse or analytics products. Particularly, those products are highly important in the ERP area.
Early stage everybody wears 10 hats, work is distributed and prioritized on a day to day basis.
Once you have dedicated people or teams for task domains then the whole thing shifts to having a need for bureaucracy.
You can slice the number of pizzas anyway you want, but imho once people have a “this is/isn’t my job” mentality (which again, is not a bad thing), you really need to focus on role boundaries and coordination. But the “startup” part is in the rear view.
"I have 1000 engineers and I can't get anything done!"
In the worst case the CEO solves this by doing lay offs. Been thinking about this problem for over a decade, making effective engineering organizations that can not only grow, but change shape is difficult, but can also be very rewarding when done successfully.
If you do this all the time for "finished" bits and instill a culture of "getting to done" in your engineering, it'll just be normal every quarter to move some stuff into maintenance and free up time.
But very few companies are able to do this for some reason.
At 15 strong self-directed teams, you can have a few teams focused on the high level directives, and a few entropy repair teams that mostly self-manage.
The way to think about it is maybe like homeostasis. Self-directed product teams will implement new features, fix bugs, and generally keep the thing on track, but the efficiency drops off as the feedback mechanisms of talking to customers reaches equilibrium.
To mix metaphors, a leadership team creates a kind of current flow in that system. When you're small you can go to each of those teams and ensure that current flow is happening.
But at a larger size, that doesn't work. You have to engineer and carefully craft the feedback mechanisms the teams are working off of to induce that current. This is a hard problem, but it's where things like minimum attrition policies, OKRs, etc spring from: leadership trying to have a policy that induces current.
The reasons why small teams work is because the number of communication channels go down, and you spend less time simply talking about the work and actually doing the work.
Or we lack anti-trust controls, so rent seekers are soaking up markets.
Maybe a combination of all three.
Likely. How big each "slice" is would be good to know. I wish someone would study it in depth, if it's even possible to fairly analyze.
I don't necessarily think your analogy with MS word is a fair or correct one, FWIW, only because once you build something the functionality is able to be distributed and replicated.
Employees require active up keep in a way that software really doesn't.
If teams aren’t resulting in profit, they’re gonna get pressured to make cuts or will have RIFs imposed on them. I’ve seen this happen at every company, even big seemingly inept monsters like Oracle and IBM.
That's why layoffs don't happen until the market starts to get tight, but yeah, when the market's tight.
The majority, the vast majority, of executives are not capable of measuring this any way you slice it.
Do you know what a jobs program is? I think a lot of confusion comes from people not understanding that a role that currently costs more than in brings in is absolutely not the same thing as a jobs program.
A jobs program is designed to just keep people employed at a loss by design. No expectation of a path to ever making money. The TSA is an example of a jobs program.
Another one that is questionable are tech evangelists or dev rel. Sometimes those positions can be connected to revenue, but usually it's "mindshare" accounting.
It also feels rich to have a 47 person company tell you they've figured out the secret sauce of people management and team formation.
One thing the article didn't mention is how crucial it is for a team to have focus and to ruthlessly prioritize. It's easier for bigger teams to fall into the trap of doing "busy work" and people fighting for scope on their performance reviews. This is the worst possible outcome for company and employee where you have work driven by optics vs value.
P Thiel, in 0-1 which I just started, says if you find yourself in an org where people spend more time building artifacts of showing progress, than progressing something, quit.
Also I think with open source it has scaled. I don't have to reinvent curl
Hong Kong has over 9,000 high-rise buildings, of which over 4,000 are skyscrapers standing taller than 100 m (328 ft) with 554 buildings above 150 m (492 ft).
A fair number of the Hong Kong skyscrapers appear to be cookie cutter | symmetry (reflection &| rotation) patterns.
Elsewhere:
As of September 2023, fourteen cities in the world have more than 100 skyscrapers that are 150 m (492 ft) or taller
I'm going to suggest that high rise engineers have some pretty solid skyscraper templates today. The greater challenge likely comes from architects and investors wanting unique 'statement' designs.
Sharing is good, but as others have pointed out, folks publishing such things could use a bit more intellectual humility. At this point perhaps authors just expect others read it as opinionated anecdotes.
Typical thought leader dogma aside, using pizza as a metaphor for team size has always been silly to meaningless.
An easy way to see it fall apart is to imagine a team that each eats 3-4 slices of pizza, or a team that only eats one slice each.
Are US pizzas giant?
My point was if you interpret it to mean a full meal for hungry people a two pizza team is like 2 maybe 4 people, because 'a pizza' is an order, unless these are enormous fast food 'party size' type pizzas.
^personally I work from home and rarely eat lunch, so I have no skin in this!
Regional/cultural differences, appetite and body size, pizza styles, etc.
For small teams that makes for a meaningful difference between the min and max. For large teams not so much.
My wife and I have no issues finishing a 16in tavern style pizza. So that's a 4 person team. We're not even large people or big eaters. We just like pizza.
When I worked for amzn.. I was eating approx 3500 calories a day - long bike commute. Pizza was always disappointing as rarely enough (and even then, eating my fill would be too much. I had to go out and still buy another lunch anyways* - and almost always not enough veg pizzas!). We perhaps could get into the variety of factors, yeah, eg: nutrition density to as a function of toppings.
Though.. if you look at serving size and how many people a large is meant to feed, it is pretty simply just 3 to 4.
* worse yet, because lunch was delivered, there was an expectation to work an extra hour that day (ie: working lunch meeting, and certainly do not go home early). Foing out to get enough real food and suddenly I was the bad guy for being the one team member not in office. Actually buying lunch was easier, I would just buy two at once.. Thinking back to that time, holy shit the work expectations were something else..
Startups can quickly change alignment to make the company work. They can throw more spaghetti at the wall to see when it's done.
> But how do you retain that advantage as you scale?
Once the company figures out what works, you'll want to put people and processes in place to keep it working. That means bureaucracy. That slows change. Intentionally.
In big commercial kitchens no one throws spaghetti at the wall. That's waste.
Of course you can keep splitting your teams when you are delivering a well defined product to customers. You could probably also have not split those teams. 50 people is trivial to manage, you could all meet in a room, should you have kept them geographically close. You may even have succeeded despite neutering those teams into triviality. You simply don't know.
Try doing something even slightly more complicated: Build a bridge. Or run a hospital. Or even a pension fund. Or just a software company with a lot of products. Those nice trim teams would quickly run into the brick wall of human communication.
That's the tough part of any organization. And we as a society needs to organize on large scale to get the important things done, unfortunately. Getting the easy things done is not enough.
In seriousness, pizza is not a bad measure since it means basically 3 to 4 people per pizza. Thus two pizzas is 6 to 8. 2 slices is usually assumed to be the serving size. 20 people is not even one slice per person
Are they?
Apple revenue per engineer is $2.4M. It seems hard to beat.
Corporations optimise for profit.
Apple took 2 years to introduce the clipboard to iOS (then "iPhone OS"), and 14 years to port their own calculator to iPad.
You can have millions of dollars in revenue without producing anything, on the other hand, you can also provide a lot of services and products for free
I'm quite sure a number of product people would answer "everything"
Feature factories are a thing for a reason.
If anything that goes against the parent comment, how Apple has more revenue per engineer while not being a feature factory by any stretch
In any case, taking both of the comments combined kinda prove my point, that higher revenue can be attributed to many things completely outside how many products/features was shipped:
- Pricing strategy
- You can have a monopoly with a single shitty product
- You can be middle man/broker with no product to begin with
- You can be running a Ponzi scheme or committing fraud
So it doesn't make sense to use revenue (or revenue per team member) as a way to compare teams between different companies and furthermore possibly across completely separate markets and industry
They don't know how much pizza I can throw down, especially when the company is paying.
> Right now we're 47 people
I'm sorry, but you haven't solved scaling small teams.
Their product is a collection of micro products which is pretty unusually especially for a company at their stage and size
Yes, some people can eat more or less. If you had 10 people, you could order 2 larges but that is going to be a treat or a snack, not something that will be welcome if that is a lunch meeting.