This is really, really important. In my experience, this is not just unknown but actively ignored and disagreed with (either explicitly or effectively) by a large majority of people involved in software businesses. This hamstrings: companies, individuals' growth, and the advancement of the state of the art.
It is important that historic knowledge is respected for the context it provides, while outside knowledge is equally respected for the deeper context and data behind it.
Unfortunately, this change often means generalists lose jobs in favor of specialists as a company grows if there isn't a good place for them, or if they don't rise to management. In fact, even early managers can be displaced by outside managers who come in with more management experience. And this is not inherently a bad thing for the company (while it may be for the employee getting bumped). The people that take a company from point A to point B are rarely the same people that take it from point B to point C.
*citation needed
Edit: to be clear, in my experience the specialists that aren't also generalists on some level are the least productive on the team because they have trouble participating outside their knowledge domain. Maybe I have worked at the wrong places though...
However specialists don't need to be generalists to be productive as long as they come in with the mindset that they are specialists, and therefore may not know everything. And that is especially important, as I highlighted, for respecting historical context of the company and its operations.
There might be VERY good reasons for not doing something the way the rest of the industry does it (which is what the specialist might be inclined to do). However a good specialist should also be adept at factoring that into how they approach problems, and properly weigh the new information against their past experience to determine if in fact this is a special snowflake situation, or if their approach is still the right one. Conflict can often arise when if they push for the latter, so strong communication skills are key (and that is something that never really changes for any role at any stage of company growth).
"Scale" often seems to equate with SF hotshots from one of the big names that will come in and airdrop a brand new architecture. These things take years and you need those generalists that have the tribal knowledge and the ability to jump in (almost) as deeply as a specialized developer, DBA, Network Admin, etc. I think the SRE movement is good evidence of the type of breadth and depth needed to operate a complicated system at scale.
That's the messy part of the transition: There's kind of the implication that past decisions weren't correct. It's easy for people to get emotionally involved and take personal offense.
Example: Backup product. Design started in the late 90s. By late-aughts, multiple CPUs were obviously a serious thing. New guy was brought in, and set on the project of re-architecting the core program into a more asynch data processing model. 6 or 8 months later, the changes were working for base cases, but the senior devs threw an absolute hissy fit focused on how much of their code was changed. We should've banded together and fixed the edge cases. Instead, we spent years fighting over the structure of the program, and that dev transferred to another group.
Bringing in security-focused employees worked out better. Their changes were more around the edges of the various programs, rather than in their cores. Fewer toes stepped on, fewer disagreements.
Thanks for highlighting something I missed in my post. Some of the time yes, the decisions might just be wrong because generalists may not be experienced enough to know the right way to do things. But quite often, they were in fact the right decision at that time and circumstances change.
So any sort of process and culture that helps take the emotion and risk of personal offense out of those situations is really critical to gaining the benefit of historical context in a positive manner.
> Unfortunately, this change often means generalists lose jobs in favor of specialists as a company grows
> And this is not inherently a bad thing for the company
Yes, it is not _inherently_ a bad thing, but I would say that it is _generally_ a bad thing. Sure, if you are raising a Series C and everything is about optimizing the hell out of your product, there might not be a place for generalists.
However, I would claim that if you throw out generalists, you throw out some of the core/most valuable people who could help the company establish new products and help you grow into a multi-product company (diversifying your portfolio is generally a good thing). If the company is only left with specialists it will slowly transform into another one of those industry giants that struggle with innovation.
I think it is more like an organical transition where newly hired personnel are more likely to be specialists. And that is totally okay, because when you start a company there are about 0% specialists onboard ;-)
Specialists aren't as good as generalists at integrating the work of specialists across domains (almost by definition). A lack of generalists can then lead to a situation where specialists are all making locally good moves, but the overall direction is negative -- something akin to Simpson's paradox (though, in the other direction).
I think otherwise trying to be a "senior generalist" or "staff generalist" is going to be very hard, but for logical reasons.
You have to know what you're talking about and know when to defer to experts, and being a generalist might prepare you well for that. But once you've reached the threshold on knowledge and critical thinking, the biggest barrier to being a good manager is mastering the mechanics of people wrangling.
Firstly, besides understanding the work, a manager also has to be able to actually manage subordinates. This is orthogonal to technical ability. Secondly, internal promotion of a generalist carries risk of that generalist being too opinionated based on his own work. They are much more likely to be a conservative force. This might be an unnecessary barrier to good changes.
These days I try to just be a tie-breaker for the team, instead of laying out the plan and looking for "feedback".
And I believe it's entirely possible to be both; to be a better specialist you need to be a better generalist, and vice versa.