Hacker News Reader

Comment by wand3r | Hacker News Reader

> I'm honestly curious if the massive headcount of some of these companies is even necessary.

Patrick Collison (Stripe) talks about this in Blitzscaling and how this is often severely underestimated. He says it takes a lot less engineering to build the original product/service than to maintain it and operate it at scale. There is value in running a lean operation and there is probably some level of bloat in human resources across organizations; but dismissing the reality of necessity within these organizations should only be done carefully. I think the context PC was talking about was speed of feature creation/product shipping but conceptually the same thing holds for the size of the company.

[0] https://www.youtube.com/watch?v=qrDZhAxpKrQ

View on HN

Yeah, I think it's important to stress that.

Building an application is easy. Maintaining it, adding features, scaling it, integrating with other services, building platforms… those activities all take a lot of other work.

I work at a small company whose core product was developed primarily by one or two developers. We now have substantially the same functionality with an engineering team of ~13, but constantly have a huge amount of work we want to do. Things like scaling, refactoring legacy code, paying down technical debt, handling complex and tricky edge cases that could previously be glossed over, and building administration tools – these take up _phenomenal_ amounts of time, especially if you want to work at a sustainable pace. I can totally believe that larger software houses need a surprising number of engineers.

Not to mention that you just can't break things anymore once you have millions of users on your app. You have to move very carefully.

I worked in teams maintaining large projects and I also built (and maintained) a few ones myself after starting my own business, so I can sort of compare. I'd say it's a matter of risk management.

When 1 or 2 people are building a project, making decisions is easy. You consider the options, you pick a strategy and you're the one to blame if it fails. If that happens, you pick up the pieces and try to pivot, but you're still in charge.

When you introduce more people to the process of decision making (even on low engineering levels), it creates a massive political overhead. People compete for visibility, so they will propose ideas even if they don't make much sense strategically. People have feelings, so they will often support ideas of their friends and dismiss the ideas of their enemies regardless of the business impact. People don't want to be responsible for failure, so they will often create lots of unnecessary low risk/low impact work instead of taking risks. So there will be indeed a lot of extra work, but in many cases the customers won't notice the difference.

The upside of this is fault tolerance - if one of the 2 co-founders leaves a company, it is doomed. If the same work is done by a team of 13, you can fire and easily replace 1-4 of them without much impact and investors generally like that.

I'd say the risk management is bigger than that. If you have 10 customers and cock something up, that's 10 people affected. But with a million customers, the same cock-up causes 100,000 times as much damage and might cost 100,000 times as much money. So you can spend something closer to factor more to reduce the likelihood of it happening by up to 100,000 times.

I don't think the only side effect of this type of inefficiency is increased costs. I would love to see some scientific research on it, but my hunch from working in overstaffed companies suggests that if you don't let people self-actualize through taking responsibilities and putting their ego into the problems they solve, they start self-actualizing in office politics, creating a toxic environment and ultimately making the company incapable of smart decisions. Modern work culture does a pretty good job hiding it behind a thick layer of company values, virtue signalling and multiple-page integrity policies though.

Besides, you can actually mitigate the cock-up risks without going crazy on your head count: have focus groups, beta testers, roll out updates gradually, have a clear separation of layers and a mechanism to roll back a breaking change, etc.

Additionally, adding support for advertising at scale is a massively complex effort, and usually invisible to outsiders.

It's one thing to build a WhatsApp-type app, focusing 100% of your energy on the core messaging functionality itself, and little else.

It's a completely different thing to build a revenue-generating business that needs to connect with an entire ecosystem of advertisers, ad servers, video servers, programmatic ad exchanges, analytics, brand safety, reporting, billing systems, and a lot more.

And we're not even talking about services. There's another big layer of complexity to keep those services running 24x7 (well beyond just SREs), ensuring ads are running properly, campaigns are fully optimized, advertisers are happy with what they're getting, etc.

Why not outsource the complex ad work to a company that specializes in it? Why not use Google's DFP platform for instance?

When building an advertising platform that can potentially compete with Google, it's best to not put your entire revenue stream under the control of Google.

> Additionally, adding support for advertising at scale is a massively complex effort...

My interpretation of the GP's comment is that the advertising platform is not the core business but rather a secondary concern. Focus on your core business and outsource the rest.

In that case, I agree with you that outsourcing is a good way to go. However, it should be stressed that the "core business" needs to be the core revenue generating portion of the business. I don't care if your company is dedicated towards making "the best photo sharing/social media/experience sharing platform for users". If at the end of the day your bills are being paid by advertisers, you're an advertising company and are in direct competition with Google.

Supply side platforms take a pretty big cut of the overall revenue, and are often very focused on existing formats like text and video. You’d be losing out on a huge chunk of advertising dollars both from the SSP cut as well as not getting optimal pricing because you’re bidding in the wrong market.

Of course ideally the question would be “why not just have users pay you directly instead of relentlessly monetizing every bit of personal information you can squeeze out of them” but as an industry unfortunately it doesn’t seem like we’re there yet.

An army of full time employees ain't cheap either. I imagine your ad business would have to be very large to justify the investment, especially if building & supporting an ad platform is not your core business.

It also adds human redundancy and specialization, both of which reduce risk for a company. In the "product/market fit" words, before fit, a company is all risk. After fit, during growth and long term profit extraction, companies try to reduce risk as much as possible (and optimize for greater margins/efficiency).

Is there an example of a million-user/big B2B company built entirely on top of FAAS like AWS Lambda and some lind of database with auto scaling, sharding et al. (maybe Spanner or CockroachDB [0])? I suspect once somethink like that happens we could have companies that have almost Google scale while just employing a couple of engineers.

[0] As a side note, I'm sure a blueprint for handling the common DB scaling problem. I suppose we could train a ML model to read some logs/usage stats and handle the remainder of edge cases. But making such a model will probably require hundreds of engineers so I guess we're safe at the moment.

The scaling isn't the problem that requires large engineering teams. It's partner integrations, compliance work, and all sorts of things that are extremely non-core.

this 100x!

Snapchat was built on Google App Engine, so they're probably one of the best large scale examples of this.

This is true, and really one of the more mysterious things about the headcount. This should be an example of how to keep an organization lean as it scales. I wonder why it isn't.

Snapchat and Nextdoor are great examples. The former runs most infra on Google App Engine, and the latter runs primarily on AWS.

I think it’s important to remember that this is all happening in the context of easily raising huge amounts of VC money at incredibly high valuations. If a company was looking to raise money say 10 years ago when, all corporate metrics being equal, their valuation would have been half or even less than what it is today and it was much more difficult to secure funds, the mindset about headcount would likely be different.

Unfortunately, the mindset of ordinary people is reflected clearly in OP's confusion.

Reply on news.ycombinator.com