Hacker News Reader

Comment by M0r13n | Hacker News Reader

I'm honestly not that surprised about the size. In my experience, efficiency and the number of engineers have not necessarily correlated.

For example, I used to work for a company with about 500 employees (relatively large by German standards) and now work for a competing company in the same industry with less than 150 employees. The core engineering team only consists of a dozen engineers.

The output is noticeably larger. The number of new features and especially the stability is significantly better. It is noticeable that due to the small team, each project has one or two "heroes". They are responsible for most of the commits and know the system inside out.

These "heroes" have been working on the same or similar systems for over 30 years in some cases and have experienced a lot in their careers. As such, they can bring experience and advice that is just out of reach for others.

Given the choice, I would always prefer a small team over a large team.

View on HN

Having worked in a range of companies, there is definitely an inverse correlation between company size and speed, especially for individual teams. At 500 people you're well into enterprise level organisation, where a lot of processes are put in place to minimize the amount of damage a single person can do to the organisation. That also puts a speed limit on those efficient and knowledgeable contributors.

Some companies spend a lot of time & energy trying to implement operating models that try to overcome this but actually end up putting in place more decision processes & oversight. Sometimes the real solution is to reduce control and let the efficient teams lead the way. But then what would all that management do to fill their days...

I think this is more correlated to maturity than size. It's just that size is highly correlated to maturity.

I've been in large companies where I was the admin/owner of their AWS account. And I've worked at companies with 200 people where I have to submit tickets to get anything created in AWS. The major difference was how mature the product was.

I work on a suite of products over 50 years old that have been incumbent for about that long and still have a submit tickets to get just about anything done.

My current employer has a nice balanced approach to this: everything is a code review. I may need a gatekeeper to approve my change, but I can always submit it.

It’s also safer this way. Not even the proper owners should routinely be on production shells or tools with live write access. They also go through peer review with each other. Once the infrastructure is in place for that, it makes sense to open up.

Heroes are a red flag for me, as a manager, trying to build a resilient team. Which is not to say that you can't have a range of experience levels. But what happens when those heroes are unavailable? Maybe they're out on vacation, or medical leave, or they eventually retire or leave the organization because they're tired of propping it up? Heroes tend to be a single point of failure.

You're right up to a point, but I've seen far worse red flags. Like companies that can never acquire these heroes in the first place because no one competent would work there that long.

As somebody else mentioned, these companies end up regularly throwing millions at consultant projects that always fail. In other words, they pay a hefty premium for shitty temp "heroes" that give them less than employee heroes would have given them.

You see this a lot in the traditional finance sector, where managers don't appreciate tech workers and relentlessly fuck themselves over trying to save a dime.

Yeah, somewhat counterintuitively treating tech as a cost center will inevitably lead to wasted funds. But hey, that just makes incumbents die faster and give way to organizations treating tech as an investment. Which are the ones you want to work for as a tech person anyway.

If you have good trust and communication within a team, the scenarios you describe are surmountable.

Every strength is a weakness, and every weakness is a strength. IME, heroes are no more a red flag than pretending that good engineers are interchangeable. It depends on the context.

The expected outcomes of these teams are different, and that’s OK. If you’re a very small company and don’t have a couple heroes, you won’t build anything important. If you’re a very big company, the heroes that built it left years ago, and you need resiliency more than new heroes (unless the business is going sideways and needs saving).

This is definitely a potential risk. However, you have this risk with small teams (less than 10 developers) even without hero roles. The smaller the number of developers relative to the complexity of the product, the higher the probability of a single point of failure.

However, you can reduce the risk through good documentation, good engineering practices and so on. As long as you are aware that the team is partly made up of heroes, you can prepare accordingly. So as a manager, I can try to ensure that a few heroes keep productivity high while making sure that the rest of the team understands the basics of the system through things like code review and the like. In the worst case, a teammate can then at least get up to speed quickly.

The worst team is the one with a single point of failure. All hell breaks loose if they're unavailable. This makes me wonder why managers have teams like this?

There is one type of team which is even worse than one with a single point of failure: those teams without anyone at all capable of fixing the system. Given that alternative, having a single point of failure is better than having nothing at all. It would of course be even better to have multiple capable employees, but you cannot always have the team you want with the hiring budget you have.

Not all managers have a primary concern of "trying to build a resilient team".

Sometimes, the #1 goal is building a high-performance and/or high-quality product. Lack of team resilience, (temporary) downtime, risk of project failure due to an insufficient bus factor...they're all not good things, but they can be compromised in the pursuit of that goal.

If you want high resilience as well as high performance and also high quality, well, you should go work at NASA, and hope that the Senate works out their budget issues...

I was talking about regular tech companies not NASA or startups with funding issues.

I think that your point stays valid there because reality is far far away from ideal case.

Managers fail up. So they can point to their tight budget in their next interview.

Oh, they can also blame the single point of failure for being stupid and lazy. That tends to work a few times before their boss catches on.

Being replaceable tends to make work less satisfying. All the places I've worked at that followed your advice had the most churn and the least productivity/ROI on engineering $ spent.

Being replaceable tends to make work less satisfying.

In my experience the opposite is true. My personal goal on any project is to make myself replaceable. There's nothing I find more tedious than having to work on the same thing for years because no one else can take over from me.

You’re talking about something else, where you are an key employee building a valuable system that doesn’t need you. The other person is talking about a company hiring you to fit into a system that never needed you.

I would hate to be stuck on the same product or set of features, but I very much enjoy getting to a point where I know all of the systems, how they relate, and roughly where all the functionality lies and who's worked on what. It takes a lot of time to develop that experience and it's so valuable, and it seems so strange that companies don't want to select for this and we're dominated by a culture of job hopping every 2-3 years because it's the only way to maintain appropriate pay. Then companies wonder why everything is over budget and never on time

> My personal goal on any project is to make myself replaceable.

So you admit that you aren't replaceable? If the company mandated that you should always be replaceable then you wouldn't need that goal, it would always be fulfilled. And working in a way that makes you always replaceable isn't fun.

Edit: What we learned from your comment is that making yourself replaceable is fun, but being replaceable isn't fun, since as you say you work to become replaceable so you can start working on something new rather than to stay replaceable forever.

If you think being replaceable is unsatisfying, try being indispensable. In the long run, it’s way worse.

I think like most things there is a balance. The sweet spot may be I can go hard on the project but if I feel like I need some time off - there is the ability to step back and have the work continue.

I'm sure I don't know how to achieve it though.

You hire more "heroes". Pay what they're worth. Make them happy.

Or you just pay an order of magnitude on hosting, performance consults, bandwidth, etc. etc. etc. And STILL won't EVER get to the same result, because you can't always pay (I'd say almost never) your way into a stable and performant solution. Look at all the others.. Nothing works as well as whatsapp.

Good luck hiring 500 engineers who have no idea what happens throughout the whole stack.

I'd rather pay what you save on cloud providers and snowflakes to them. (not even taking into account the result of these "heroes".. the business value / dominance whatsapp has because of this)

You can hire 1000 people who can't draw, or maybe just a little bit, or you can pay an artist and get the result you'll otherwise never get.

> Nothing works as well as whatsapp.

Ehh, that's very debatable. Signal, Threema, and yes even Telegram (regardless of E2E enc, etc.) work well and are reliable.

> You hire more "heroes". Pay what they're worth. Make them happy.

i dont think its as easy as you make it sound. these "heroes" generally are as productive because they dont sync and already have a clear idea of what they want to achieve/how to implement what they wish.

if you add a second person like that to the same team you run a massive risk of having two competing ideas which creates a lot of friction.

So make sure you're hiring a hero and not a primadonna?

The difference between hero or primadonna can be environmental. When there are multiple experts for a department/tech stack, you get heroes. If the bus number is one, a hero risks converting to a primadonna.

I find if you hire heroes with different skill sets you can avoid this dilemma. Or don't hire two Supermen. Hire Batman and Superman instead. But if a hero converts to a primadonna then they were never a hero. On the other hand if you're skimping a hero of the equipment and resources they told you they needed from day one, you're the a***.

I agree. I'd add that having multiple experts in competition with each other (due to bad culture, bad communication or even just bad personality) also has the potential of turning heroes into primadonnas.

Environments don't create primadonnas. It's a matter of attitude and maturity. Being an expert doesn't make a person insufferable.

In my experience, insufferable mid-level management can bring out the worst in the best people, and then accuse them of being primadonnas because they are absolutely incapable of any self-reflection themselves.

At my company, “heroes” are given their own projects with them given full creative control over every detail, including the people they work with. I think I’ve read somewhere on HN to never let any two person do the same thing, to avoid the conflicts you mentioned.

Benefit of this is that these heroes can each break new grounds, and their BKM shared amongst themselves making the team extremely productive.

> I think I’ve read somewhere on HN to never let any two person do the same thing, to avoid the conflicts you mentioned.

I’ve read this in Peter Thiel’s “Zero to one”.

Yes, you will always pay one way or the other.

I work as a contractor, thus I have read a lots of code from various different types of projects and the conclusion is always the same, if they had spent more money on skill to begin with I wouldn’t be needed.

> Nothing works as well as whatsapp.

I don't disagree with the rest of your post but Telegram does in fact work as well if not better than whatsapp.

Use heroes to make jumps you can’t make with cog machine teams, then manage the prior tier platform to be resilient.

5 heros on vacation for 7 days, and coming back and finishing the work in next 5 days is more comfortable position than 50 clueless engineers claiming work will be over in 4 days.

hmm, not sure how many customers would be okay with 7 days of downtime, for instance.

System has less of a chance of going down if you have fewer noobs pushing to master all day.

Many/most teams don't have heroes at all, they only have noobs :) I.e. most people stay on a team for a year or three and change jobs afterwards - there's no chance of accumulating knowledge. Nobody is sure how things work and why they are the way they are.

I believe many managers are fine with this, because the team is producing something (i.e. "velocity" is higher than zero), but are oblivious to the fact that team could be way more productive if the knowledge was actually retained in people's heads and not just lousy attempts at documentation.

I don't know if they are oblivious to it - but it might not be possible to pay people enough to get them to stay due to pay bands set by HR etc.

OP mentions that he is in Germany - there it might be more possible as SWE doesn't pay as well as it does in the USA and there are fewer companies so it might be feasible to pay a lot more than the competition, whereas in the USA that's unlikely to be viable outside of companies like Netflix etc.

I understand your point, but what managers fail to understand is that someone usually "carries", i.e. is actually responsible for the success of the product. I have even seen it be a manager...once.

> Heroes are a red flag for me, as a manager, trying to build a resilient team

Heroes are your most powerful asset, but you have to use them responsibly. The best thing you can do when handed a 10x unicorn developer is to try to document 100% of the things they say & do, and also make it a requirement that the hero mentor others some % of the week. E.g. For 1-2 hours every Friday you force them to hold a "no stupid questions" session.

will your 10x unicorn be willing to both write documentation and do mentoring?

My first interaction with this management philosophy was when our partner AOL replaced experienced SRE serving our project with 3 far less experienced people for the same price. What used to take an hour now literally would take weeks on the bright side for the manager he had way more reports now.

I tried to clarify in my top comment that I wasn't saying we should avoid having experienced people on a team, but based on your response, maybe a little more clarification is needed.

When I think about a "hero" in terms of team dynamics, it's a person who is consistently relied upon to save everybody else's butt, often by doing things that are flashy but not particularly healthy, such as working long into the night to meet an unrealistic deadline. When you have this sort of Superman figure who's willing to swoop in and save the day, the problem isn't so much their level of experience but the unhealthy level of dependency that's placed on their shoulders.

I'm all for having highly skilled, highly experienced engineers who are productive themselves and can further increase productivity by helping unblock others on their team. And I agree with you that replacing them with some number of juniors without the same institutional knowledge can be disastrous. But if your team becomes so dependent on heroics that they can't stay afloat otherwise, then when that hero gets hit by a bus or just quits to take a position elsewhere, you're screwed.

> But what happens when those heroes are unavailable? Maybe they're out on vacation, or medical leave, or they eventually retire or leave the organization because they're tired of propping it up?

I don't know why people insist on using examples like this instead of the more obvious one: What if they are hit by a bus?

Any of the other examples have trivial solutions in the event of an emergency: call them and maybe offer them some money. If they are very stubborn then go to their home and beat them with a wrench. But you cannot get information from a dead person no matter how much money or how big a wrench you have, and people die suddenly every single day. That's the worst case you need to be making contingencies for.

It appears you have become the hero and are maintaining that status.

It depends on the company/stage/setup/etc.

Having a "hero" at a 5 person startup can mean life when death was inevitable. Having that at a 500 person org likely means another team is left cleaning up crap.

It's really not hard to be a "10x engineer" if you disregard tech-debt and error-handling. I was part of a very productive, complementary duo where my partner churned out a lot of code that only catered to the happy path, and I'd clean it up/"productize" it.

I actually enjoy that sort of work, but didn't receive as many accolades as the guy "churning out features quickly", even though he'd break the build and block the entire company at least once a week before I paired up with him.

Once I saw how the sausage is made, I'm a lot more sceptical of the "10x" label, either there's an invisible support system, or the code base had a short half-life. Any other scenario is a set-up for disaster.

7x is the sweet spot.

Managers prefer a team reliably working at 0.1x speed rather than have it work at 1x speed 95% of the time. Yes, sometimes people leave and you will have less velocity for a while, but people pick it up and you go back to high velocity. To fix that you ensure the team always works at slow velocity, that way it wont get slowed down since it was already performing as if you lacked those heroes from the start.

> Given the choice, I would always prefer a small team over a large team.

I think that's one of the conclusion of the "mythical man-month" book.

As long as the core system is built on solid engineering, this definitely makes sense.

From my experience in a deadline-constrained environment, things end up being slapped together until they "just work". Adding new features gets more and more difficult and if you're still constrained by deadlines, then the solution is to hire more, which is effectively a brute-force solution to meet the deadlines. Output per employee gets smaller, but total output should grow by some marginal amount.

Absolutely. This is also when bigger and more pervasive bugs enter the conversation. Decisions that kicked the can 2 years ago are now a big problem and no one team can fix it.

I worked at a place that made a group of loosely-related websites. Basically different spins/themes on the same products. There were 4 of us on the development team. There were 10 websites. Some new, some in maintenance mode.

I quit after a few years, and about 3-4 years later they went on a hiring spree. Tripled the number of developers. Refreshed some products, discontinued some, and some went into maintenance mode. They still had around the same 10 websites, but 12 developers.

The products didn't get better. The products didn't get made any faster. They didn't have any less bugs. My wife actually works there now, and when she tells me about the issues they have I laugh a little bit.

It's all the same problems we had back then. Invites don't work. Teams doesn't work. Uploading profile pictures has issues. Entering data doesn't work. Registration is broken.

I had a somewhat similar experience with a large tech company. Grew from <100 to 600 in a few years, but in the meantime they weren't even able to update the homepage. Not even the content changed, in almost three years. Everything was convoluted and over-engineered. The whole thing was rewritten from scratch twice, and when I left there were plans for a third rewrite from scratch. COVID destroyed the company: there were zero customers. But the website still struggled to stay online. That's when I decided to leave.

> efficiency and the number of engineers have not necessarily correlated

Often inversely correlated...

Reply on news.ycombinator.com