Hacker News Reader

Comment by planet-and-halo | Hacker News Reader

The best thing I ever read on this topic came from biology. In "The Diversity of Life," E.O. Wilson describes how specialist species thrive in stable environments. The stability allows them to evolve "deeply" into a niche and outcompete any species less specialized for that same niche. So if I'm really great at plucking fruit from tall trees, and you suck at it, I win. In unstable environments, however—say all of those trees burn down in a forest fire—it's the generalists that triumph, because they have more options and are therefore more resilient. Of course specialization and generalization are abstract terms. Reality can operate on a spectrum, often a discontinuous one at that. So maybe I'm a finch that's best at poking my long beak into a particular kind of tree, but in a pinch I can also hoover up ants off the ground better than most of the other creatures in my ecosystem. So it goes for humans competing in an economy too.

View on HN

I've had a going theory on this - that it's a yoyo between generalization and specialization.

A particular topic starts with generalists but as time progresses and stability (i.e. a path worth pursuing) becomes apparent, specialization is needed to eek out more performance gains.

Then after a while, that particular area gets disrupted because there's a fundamental shift in landscape and the cycle repeats because the landscape is new to everyone. Many times this new landscape was brought on because a specialist applied a specific finding to a new, larger group. Then generalists help build that new space.

Maybe this is obvious...

Pardon me for a bad (and generalist) habit. I’m not correcting your spelling for any reason other than to give you a free Scrabble tip: it’s eke.

And yep, you’re spot on.

I had no idea, good to know!

ah thanks! i'll keep my error for posterity

This is the most pleasantly written spelling correction I have ever seen.

This is touched on in "Beyond the 80/20 principle".

Much like your example, though I think it said specialisation itself can offer up new generalist opportunities, so internal forces as well as external shifts.

One of my favourite books of late.

In economic theory this is well known, and related to S-curves. First innovation (requires integration, which requires generalists/people spanning disciplines), then specialization and price war / commoditization

Interesting. So, as the s-curve begins to climb up and systems demand specialists, then, is it the case that they'd be paid more and hired at higher levels than generalists?

Which is to say, specialists tend to get hired once stability is achieved but also tend to capture more value / status than the generalists that helped the system navigate the s-curve?

Alternatively, are all senior hires at stable systems like FAANGs, specialists? Even if they're generalists, do they get valued less than them: May be hired at Staff level instead of Senior Staff or above?

Specialists need to be trained or train themselves. This is only really achieved by on the job training or very in depth courses. So ultimately, you would hire good generalists and get them trained on the job to specialize in the field.

Nobody is born a specialist. Universities do produce some but it's research, not the same as development and deployment.

And you need generalists as well as multiple field specialists to not get locked in a quibble over comma or a comfortable but overcomplicated and bad solution.

So in this context, do you think generalists are needed to set high level strategy and specialists needed to execute it? If so, would this imply that generalists would be a better fit for senior roles?

> but also tend to capture more value / status than the generalists that helped the system navigate the s-curve?

Are you assuming that being a specialist in a mature market always makes you more money than being an early player? I don't think that's always the case, there are many stories in the news of early employees of Silicon Valley companies who are now multi-millionaires just because they were in early. E.g. one of them IIRC is the cook who was in charge of preparing meals for the early Google team.

Generally agreed, but you're looking at stable FAANGs.

If you look at startups, for the most part, need lots of senior generalists who have a can-do attitude to 'just figure it out'.

Ideally, those generalists get rewarded w/ stock (side stepping the trigger topic on purpose) because they joined close to the ground floor. Most startups don't need and/or can't afford specialists until around Series B. Again ideally, if you needed a specialist early, you've got one on the founding team.

Many exceptions to the above statement, but it fits this topic's narrative.

True. I think every team needs both generalists and specialists to complement each others qualities. For example, my manager is kinda specialists and he is great at solving problems in his domains. But then I put my generalist ideas and solution is sometimes way simpler than what a specialists was trying. The reverse is also true. So a balance in team is needed.

And once in a while, a really large disruption in all areas causes a mass extinction where all specialists are wiped out, and only a few generalists rebuild the new environment on their own.

Does it mean the roles of husband and wife, which were specialized before, are generalized now due to unstable marriages?

I know you’re being downvoted but wouldn’t this be indicative of an unstable economy not an unstable marriage?

E.g., both parties may have to work and do household duties because a one-salary household is much harder to manage financially now due to wages, lack of employment stability etc.

I've been wondering something like this lately in relation to apparent species die-offs - we view and catalogue numbers in a timeframe that makes sense on an immediate human scale.

What if species have always gone through collapse/regrowth cycles and we happen to be viewing things through a too finely-focused lens that skews interpretation?

I'm not trying to make light of Humankind's horrific environmental impact, rather checking that we're not simply missing a fundamental natural law and so misinterpreting things.

This came to the fore of my mind particularly recently, listening to a podcast wherein part was talking about a rebound in the population of Peregrine Falcons, with them taking up homes across London.

- ed lol at me - 'taking up homes across London' sounds like the falcons are gentrifying. I meant, obviously, 'their populations are growing across London'.

The key is right their in evolution - adaptation.

These days I’m far less interested in what skills someone has. I’m far more interested to see if they have a track record of being able to adapt to changing circumstances. I’d much rather have someone who has shown they can adapt - or better still, will embrace the chaos openly :)

You don’t want to have everyone like that - you need a few “slow and steady wins the race” types to keep fundamental stuff going too. But I know which side of that equation I’d rather be on :)

My nickname for this phenomenon is the "robustness/efficiency spectrum." In this case, it would be the robust generalists and the efficient specialists on opposite ends of the spectrum.

(I'm not sure it has to be a two-dimensional spectrum in all scenarios —I believe there may be cases in which robustness and efficiencies are at least partially orthogonal/independent to each other.)

Taleb discusses this in his writing, if I recall correctly.

In reinforcement learning which is used for things like AlphaGo, there is a similar duality called "exploration vs exploitation":

- do you keep searching in breadth for new avenues

- or is it time to dive in depth in what you found so far.

And this is even optimized through something called "regret".

- example: https://rlss.inria.fr/files/2019/07/exploration_lazaric_1.pd...

Tech recruiter here. What you are saying fits neatly with my observation that during a bull market the specialists tend to do great, amazing compensation, lots of contracting options.

When the market turns these are the first to be downsized because they are very expensive. The generalists are more sought after because firms try to produce the same output with fewer resources, and for this you need the jack-of-all-trades types.

I've also found that in suitably large enterprisey orgs, the generalists are the first to be let go under a cost crunch because no matter what they do, they have in their stable of people someone else who can do it better. And its very, VERY rare that a generalist is actually DOING more than 1 or 2 things, no matter how many things they CAN be doing.

The problem with this line of thinking is that there is a distinction between you and your current skillset. In nature a finch can't deliberately change the shape of its beak, but a human can alter their behavior. A fast learner can be deeply specialized in terms of knowledge during times of stability but then quickly switch over to being a generalist when conditions change - on the other hand someone who took a long time to become a jack of all trades may struggle just as hard as anyone else to adapt to a genuinely new development. When most of us picture a specialist, we think of someone with perhaps an advanced degree or years of experience in a narrow field, while a generalist has invested much less time in a niche, but really a specialist is someone inflexible in their thinking and focused on details whereas the true generalist is someone more focused on broad concepts and open to new ideas, regardless of career history.

> The problem with this line of thinking

I don't think it's a problem. It's just that specialization can happen in skill AND in genetic adaptability to the environment. We think of skill and genetics as separate things, but in this context it can be helpful to think of them as existing along a continuum of how we learn about the environment and adapt to it. Genetic adaptations 'learn' about the environment over multiple generations, and behavior changes happen within a single generation.

So for example, the finch is specialized exactly because it can't change its beak, and the human is a generalist exactly because it can alter its behavior. I think you could argue that any human behavioral 'specialization' is still probably less adapted to any given environment than animals who evolved for a million years to be suited to that environment.

But it is not useful here to think of skill and genetics as being on a continuum, it just produces confusion because these two orthogonal concepts have very similar terminologies, but its a mixed metaphor.

It's good to be a genetic generalist during times of instability specifically because you are incapable of changing your genes during the upheaval - and thus while you may not thrive in any circumstance the risk of extinction is also mitigated. If we had advanced genetic engineering, and could rapidly adopt different genes to suit the circumstances without having to wait for generations of random mutations, there would be no reason not to specialize genetically. While this is a pure hypothetical for genetics, it is reality for behavior - no matter how much you've invested into learning one skill, you can instantly switch to learning another, and you can choose exactly the skillset you want to curate. When a new field emerges, your experience in other, unrelated fields is irrelevant - the specialist is at no disadvantage compared to the generalist, and so they sacrifice nothing by specializing in a field.

In contrast to genetic evolution, which has very discrete goals (reproduce, survive, etc) and is agnostic about how these goals are achieved, skillsets can have a much wider range of objectives and the 'how' is integral. A finch might optimize for eating nuts or insects, or could remain a generalist to handle both, but is unlikely to come across a nut-insect hybrid where the generalist beak outperforms either the nut or the insect beak. But such interdisciplinary situations happen all the time - for example the roboticist with understanding of mechanics and software can do better than either a pure mechanical engineer or pure software engineer. The roboticist has a wider, shallower skillset, but that is only beneficial for tasks which happen to overlap - they won't have any easier time switching to say pharmaceutical development. It simply doesn't make sense to think of the roboticist as a generalist and the other engineers as specialists in the evolutionary sense.

Certainly any direct comparison of specific behavioral to genetic specializations is illogical. By what metric would you compare how well adapted a lion is to a big game hunter with a gun? How could you compare a dam-building construction crew to a family of beavers? Again, we might use similar words to describe what they are doing, but we are describing totally distinct concepts.

I love video games and this makes perfect sense to me.

Games could be seen as different universes, they have different rules. Generalists looking for 'meta' ideas that can help them understand the game rules quickly and take advantage first. Specialists tend to put more effort into the same game or the same genre practicing (like typical professional players) and then slowly outperform generalists.

In the real world, unstable or dynamic societies are continuously changing rules, and it feels just like jumping from games to games. Once the society becomes stable or static, specialists slowly come into play.

On top of that, there's also another dimension of the game: speed. If the game rule is very simple, then the specialists would win very fast. Otherwise, the process would be very slow when the game is very deep. I believe the same rule would also apply to different industries.

So if I were to apply this to my career, I should be a T-shaped engineer, right? :P

Joking aside, I love this comment.

To expand on this further - maybe the question of generalist vs specialist is not the right question. Rather, I come to this conclusion when exploring that topic - adapt to the environment.

So, the 3-step plan to become successful at life (TM):

1. understand what it takes to be successful in the current environment

2. adapt yourself to be successful in the current environment

3. understand that your environment will change, so iterate over time by going back to step 1

Honestly as a human being I think you also have to be careful what you're optimizing for. This is a nice framework if you're thinking about survival and to some degree career, but as human beings we also have a very complex psychology and those problems are mostly "solved" to some degree for many of us. Money and careers are obviously important to some extent, especially if you're struggling to pay rent or eat, but beyond that if this is the only framework guiding your life, you could easily make a bunch of money and still end up miserable. Treating ourselves as economic units maximizing utility I suspect is a big part of what has exacerbated societal ills like alienation and depression.

We have this in physical engineering too: a robust design continues to perform well away from its optimum state configuration, on-design state, etc.

Something in a deep minimum may well go to heck when y’all he situation changes just a little bit.

Just a restatement of K vs r theory.

https://en.wikipedia.org/wiki/R/K_selection_theory

it doesn't seem so... can't either selection strategies be used independently by specialists and generalists?

Yeah it's not quite a 1-to-1 comparison. Close, and I can see the overlap, but K/r is about low effort spamming vs. high investment intensive building.

Nothing says you can't spam specialist, niche offspring -- works well for certain kids of ants. Likewise you can spam generalist spawn as well, hoping that they'll be well rounded enough to get by in several different scenarios.

I thought it was K&R ...

Really it's more akin to succession I think.

https://en.wikipedia.org/wiki/Ecological_succession

And our environment has been very stable for a long time. But the past mega trends are no more: Covid, global warming, end of boomer dominance, China dominating and collapse/stagnation of the asset bubble may change this for a while.

From a career perspective specialization pays better but in tech few specialities last a career. So also build skill sets and assets that can be leveraged into other tech or even non-tech domains.

It's like training an ML model.

By that argument...the faster the world is changing, the bigger of an advantage generalists have. There's even a book about it: https://www.amazon.com/Range-Generalists-Triumph-Specialized...

Would you believe, this article is a review of that very book ;)

Reply on news.ycombinator.com