Comment by deminature | Hacker News Reader

Comment by deminature | Hacker News Reader

Parent Full thread

I wondered this before joining a large tech company. As it turns out, a lot of engineering is done on experimentation to optimize the product. Only a small portion of users might see the experiment, and many don't succeed.

An example might be complying with food and alcohol law in a single region, which you would never see unless you lived there. Another example might be a complete experimentally gated refactor that shaves 1 second off loading time. A lot of effort would go into simply making sure the change can be turned off, and verifying with unit tests, which takes time and effort.

These projects might be pointless on a small app, but in a large business might be worth millions to the company.

Another aspect is your changes often have to be approved by a lot of people, which is not the case on small projects. Your projects are also analyzed by data scientists, to make sure they're not harming the business - and if they are, your work might be thrown out, which contributes to even less visible change to outsiders.

All of these in concert slow things down, and necessitate staff.

That's a really charitable explanation and I think it explains some percentage of labor, but not nearly all of it (not even the majority of it, IMO). Yes, there is more going on behind the scenes in every company than you'd ever know about. But I think the majority of labor follows the pareto principle-- there are maybe 100 employees in every company of, say, 1000 people that do nearly all the work, and 900 employees that do very little to nothing at all at best (and produce negative value at worst).

Why that is the case in every business, I haven't been able to figure out. I suspect it's because when businesses reach that size, the efficient market effects become greatly overstated in the short term, so these businesses naturally acquire a lot of fat.

One piece of evidence for this is that private equity exists. You can buy a business, fire a quarter of the people, then sell it and make a lot of money without hurting the operations at all.

I think you're missing the key part of GP:

> These projects might be pointless on a small app, but in a large business might be worth millions to the company.

With enough traffic and revenue, you'll get a positive return on hiring a bunch of engineers to optimize things that wouldn't be worth it for smaller sites. With enough engineers you'll get a positive return on hiring engineers to make engineering more efficient.

It's only natural in that case that as traffic and revenue drops, the ROI calculation changes and it suddenly becomes better to lay those people off.

And that's just engineering. I can imagine that in this downturn the sales department just isn't generating new revenue. What brick and mortar business is going to advertise right now? It only makes sense to lay off as much of the sales teams as possible. You can hire them back later.

> With enough traffic and revenue, you'll get a positive return on hiring a bunch of engineers to optimize things that wouldn't be worth it for smaller sites. With enough engineers you'll get a positive return on hiring engineers to make engineering more efficient.

Very insightful! As a corollary, you should not staff up until there is something to optimize.

You probably can't "hire them back later". People who you've fired might have an aversion to working with you in the future, or simply be occupied with other work when you want them back. If you mean ripping out entire teams and replacing them with outsiders, that also has costs.

What I've seen is that salespeople are lot more fungible than veteran engineers who have a lot of knowledge of the companies system. Sales armies tend to have a lot of more junior staff too. They might not hire back the same salespeople, but they should be able to hire what they need. There will be a lot of people looking for work.

There's also reputation issues with a "hire them back later" attitude. Engineers are likely to become skeptical of recruiters from companies with a policy like that, and then you start struggling to hire anyone.

That reference was about sales employees, not engineers. I think most people realize that in larger companies the sales team and the engineering team are managed very differently, with the employees having very different expectations about the stability of their position.

> With enough traffic and revenue, you'll get a positive return on hiring a bunch of engineers to optimize things that wouldn't be worth it for smaller sites. With enough engineers you'll get a positive return on hiring engineers to make engineering more efficient.

The problem is that no modern tech company I know actually measures the ROI of this (positive or negative) so it's completely unrealistic to determine the IRR on it. In other words, there might be a scenario where it actually is more profitable NOT to do it.

Check the incentives. For corporate managers, your salary is generally proportional to the number of people you manage. Who cares if your 1000-person org doesn't accomplish anything, you're a big shot now.

On a sports team, coaches are rewarded for championships. There is no room for waste, and underperforming people get cut.

>> no room for waste

Most sports teams rosters are restricted by league rules. However coaching staffs are not, and they have grown tremendously in recent years, as have college athletic department staffs. When someone actually does take a hard look, these staffs seem exceptionally bloated and wasteful.

https://www.baltimoresun.com/news/bs-xpm-2006-10-31-06103101...

When Vince Lombardi began coaching the Green Bay Packers in 1959, he had four assistants. This year, Denver Broncos coach Mike Shanahan has 21.

http://www.espn.com/espn/page2/story/_/page/easterbrook%2F10...

Ohio State lists 458 people in its athletic department. There are 192 faculty members in Ohio State's English department, with a support staff of about 50.

https://www.newsobserver.com/sports/college/acc/article16325...

One assistant coach is called the recruiting coordinator, but he’s backed by a Director of Player Personnel, an Assistant Director of Player Personnel, a Coordinator of On-Campus Recruiting and a Recruiting Assistant. There are three video coordinators.

Slightly off topic, I noticed the first article you linked is from 2006 and I was curious if that number had grown in the last 14 years.

2019 Denver Broncos had 23 assistants, 24 total coaching staff including the head coach.

https://www.denverbroncos.com/team/coaches-roster/

Bullshit Jobs is a good book about this exact topic. Managers hire people so they feel like they have someone to manage.

Read the original article, it makes good points. I bought the book going on that and found it to be unreadable waffle.

I found the book to go more in depth to the article. What didn't you like about the book?

That is a charitable view of private equity - that it improves margins without hurting operations.

Another explanation lies in reputation mining. Think of it like doing something profitable but unsustainable.

Curious what kind of experience or data sources you base your first graph on — the Pareto principle piece makes sense. But isn’t it possible the very end of the tail is still profitable, albeit less so, than the head?

> Why that is the case in every business, I haven't been able to figure out. I suspect it's because when businesses reach that size, the efficient market effects become greatly overstated in the short term, so these businesses naturally acquire a lot of fat.

i think it's more likely that no business will know in advance who those 100 employees that create value are. It might even be that the combination of those 100 creates value beyond the sum of them individually (lets say, they are in one team).

There are also mundane work - fixing bugs or upgrading libraries, as well as keeping up the infrastructure running.

And then there's the middle management - that comes from the mis-trust that upper management generally have for the "grunts", and this happens all the way thru the organization.

> private equity exists...

and if you look at those companies that did this - very few of them becomes the next big thing. Very few of them actually innovate. Very few of them, even survive.

Private equity is really about taking risks with a failing business, e.g. buying it really cheap, and cut the fat and ride out the storm for hopefully a better future. I dont think any private equity will buy a company that isn't failing in some ways, and try to "make it better".

Companies doing well generally aren't for sale at a reasonable price anyway, so yeah, private equity by nature buys failing firms and tries to turn them around. That's the goal of the business.

Yes, you're probably right to some extent. A lot of efficiency and velocity is exchanged in these large scale environments for risk-aversion in the form of heavy testing, supporting the ability to revert changes and requiring approvals.

I'm not personally a fan of it, but I get why it works this way. The financial loss if you break something adds up in an excruciating fashion on a per hour basis.

Just like the stock market having bubbles, businesses don't get efficient except during a downturn.

They don't get efficient during a downturn. They might shrink but cuts for the most part will fall on those that lost the political jousting regardless of actual contributions to the top or bottom line.

The real answer is that it's a publicly traded "tech" company that has to maintain a charade of growth. They can either give their profits back to shareholders (violating the growth charade) or they can hire more people and hope they figure out how to create actual growth.

Corona spoke up and pointed out the Emperor had no clothes.

>Another aspect is your changes often have to be approved by a lot of people, which is not the case on small projects.

So you need more engineers because you hired more middle managers?

That's a little absurd in my opinion but I am likely wrong in this assumption.

It just seems like the reaction to "We have lots of approvals" should be figuring out how to reduce the amount of approvals needed, not hiring more engineers to increase velocity.

It sounds wasteful until you've been with a company that outgrew its management structure. I spent many years at a place that grew from 200 to 800 people in about 3 years. We prided ourselves on the flat org and high level of responsibility. But as we approached the 500 mark, the cracks really started to show:

- Things truly requiring management handling (like raises, big expenses, hiring, etc.) took FOREVER in the best case because of the bottleneck at the top. More commonly, they got lost/dropped without explanation.

- Flow of sensitive information was slow and inconsistent because the only way to communicate with a flat org is all-or-nothing.

- Specialist engineers found themselves trying to track too many projects before they realized the overload was a problem (this was me)

- The president (and effective sole owner) was still deeply involved in all projects and regularly exercised veto rights. As the project load increased, the vetos landed later and later - some times after we'd already spent 50% of the budget! This left our customers up a creek and our reputation suffered.

It's not just approvals from managers, sometimes you need approval from a group of peer engineers just to create new routes in an (internal) API, or to launch a new cron job that will upload an Excel spreadsheet in marketing once a day with data from Google Forms.

Also, UX wants to see the final result on their screen before they approve your build, so before deploy you have to push into a test cluster, but there are only 10 of those, and you need to wait until one of them is free.

And of course your Jenkins server takes 2h to build and fails sometimes, so pushing small changes takes 2h because you have to babysit the build. The testing cluster above also takes 2h to build. CTO believes in engineers taking care of their code, so SRE is not allowed to handle the builds.

And then there's an average of 2h of meetings per day (I'm not including the 40 minute daily scrum because your squad has 15 people). Half of them to discuss the production incident you had because you didn't see the Slack alerts when you were in another meeting. The other half is chapter and guild meetings where you don't talk, but participation is mandatory. On Mondays you have sprint planning, so forget about coding.

Engineering fact: A 300 watt amplifier is only twice as loud (+10 dB) as a 30 watt amplifier. The same math works for engineering teams.

Right here. Everyone read the above, this answers the question of why so many?

The best way I’ve ever heard this explained is to imagine a software business as a machine. Early on, it’s easy to trust - the machine is small, there aren’t many working parts and dependencies are all manageable. Making things even easier, the founders generally have control over hiring and they’re often still at the hiring friends/colleagues stage of their evolution.

Then, the machine gets bigger and the dependency graph gets more complicated. Suddenly, if you change one small part of your module, you run a very real chance of ruining something someone in a faraway department relies upon. So, the machine evolves to have things like coding standards and more formalized review processes. Unfortunately, the machine is composed of humans and humans aren’t as simple as just pulling it all apart, greasing it up and putting it back into service. So every single time you add a review step, there’s a chance it genuinely adds to quality (or removes risk). But, there’s also a chance that that change was about ego and organizational power. Then, in five to ten years, they hire MBAs to come in and lay people off.

Edit - Good heavens, this sounds bleak.

I think you need as many approvals as your risk tolerance allows for... they're not pointless. If they are pointless then sure, get rid of them. But I don't know how you'd guess that number without deep knowledge of the business.

It is sometimes better to go slower than you could and be more confident that the product is acceptable. Approvals can but don't necessarily help with that.

You start out with a very light process. Then you have outages, or you discover that not every part of the system with appropriate care, etc. Then you put some process into place. That slows everyone down a bit... your business keeps growing, though, so "more slightly less efficient people" can capture more of the market than "fewer more efficient people" can because you have a calculation where something like 1000.4 is still greater than 101.0, so if that additional stuff getting done is worth the cost of more people, you keep doing it!

And you can cycle through this a few times. And yes, people will also cycle through "we need to reduce bottlenecks" and such, but at this point, you've been successful, you've got a bunch of money coming in, and pissing off your clients is something to be avoided, since individual efficiency is not the end goal by itself.

> So you need more engineers because you hired more middle managers?

Companies often implement more process for good reason.

The antithesis of old Facebook's Move Fast And Break Things is to have policies that ensure good security, stability, accessibility, privacy, internationalization, meeting legal requirements, etc. and that usually means processes to ensure that eager engineers and product managers actually meet those policies, instead of rushing to launch.

It's easy to scoff at process when you're a small company or startup whose mistakes will go largely unnoticed unless you suddenly go viral. If you're a Big Tech Company, accidentally messing up security or privacy somewhat means getting a front page article on The Verge detailing your crimes.

Working at Google now, there's plenty of process involves in developing and launching a thing, but I can't really say any of the pieces are without merit.

You're not hiring more middle managers because you want too, no large companies have to promote engineers to senior engineering managers to handle the complexities of approvals required by compliance such as SOC2,SOC3,ISO27001,FedRAMP, GDPR, etc.

I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance.

> I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance.

Jesus, what company was that? And how about libraries included... Will this lawyer now also need to go through that?

The approvals usually come from other engineers. It stems from having to modify highly-sensitive code which usually has an owner you need to work with to get an approval.

The approvals often exist for valid reasons. You can't just get rid of them. You can try to streamline the approval process but that only gets you so far.

It appears that Middle managers are downvoting you in force.

Middle managers have nothing to do with the approvals I was talking about. It's entirely eng-driven.

Reply on news.ycombinator.com