I know this is a very tired topic, but does anyone have any good blog posts or articles about "what exactly is it that you all do?" Yelp having 6000 employees. I'm not surprised, I'm not mad. I just want to understand.
An example might be complying with food and alcohol law in a single region, which you would never see unless you lived there. Another example might be a complete experimentally gated refactor that shaves 1 second off loading time. A lot of effort would go into simply making sure the change can be turned off, and verifying with unit tests, which takes time and effort.
These projects might be pointless on a small app, but in a large business might be worth millions to the company.
Another aspect is your changes often have to be approved by a lot of people, which is not the case on small projects. Your projects are also analyzed by data scientists, to make sure they're not harming the business - and if they are, your work might be thrown out, which contributes to even less visible change to outsiders.
All of these in concert slow things down, and necessitate staff.
Why that is the case in every business, I haven't been able to figure out. I suspect it's because when businesses reach that size, the efficient market effects become greatly overstated in the short term, so these businesses naturally acquire a lot of fat.
One piece of evidence for this is that private equity exists. You can buy a business, fire a quarter of the people, then sell it and make a lot of money without hurting the operations at all.
> These projects might be pointless on a small app, but in a large business might be worth millions to the company.
With enough traffic and revenue, you'll get a positive return on hiring a bunch of engineers to optimize things that wouldn't be worth it for smaller sites. With enough engineers you'll get a positive return on hiring engineers to make engineering more efficient.
It's only natural in that case that as traffic and revenue drops, the ROI calculation changes and it suddenly becomes better to lay those people off.
And that's just engineering. I can imagine that in this downturn the sales department just isn't generating new revenue. What brick and mortar business is going to advertise right now? It only makes sense to lay off as much of the sales teams as possible. You can hire them back later.
Very insightful! As a corollary, you should not staff up until there is something to optimize.
The problem is that no modern tech company I know actually measures the ROI of this (positive or negative) so it's completely unrealistic to determine the IRR on it. In other words, there might be a scenario where it actually is more profitable NOT to do it.
On a sports team, coaches are rewarded for championships. There is no room for waste, and underperforming people get cut.
Most sports teams rosters are restricted by league rules. However coaching staffs are not, and they have grown tremendously in recent years, as have college athletic department staffs. When someone actually does take a hard look, these staffs seem exceptionally bloated and wasteful.
https://www.baltimoresun.com/news/bs-xpm-2006-10-31-06103101...
When Vince Lombardi began coaching the Green Bay Packers in 1959, he had four assistants. This year, Denver Broncos coach Mike Shanahan has 21.
http://www.espn.com/espn/page2/story/_/page/easterbrook%2F10...
Ohio State lists 458 people in its athletic department. There are 192 faculty members in Ohio State's English department, with a support staff of about 50.
https://www.newsobserver.com/sports/college/acc/article16325...
One assistant coach is called the recruiting coordinator, but he’s backed by a Director of Player Personnel, an Assistant Director of Player Personnel, a Coordinator of On-Campus Recruiting and a Recruiting Assistant. There are three video coordinators.
2019 Denver Broncos had 23 assistants, 24 total coaching staff including the head coach.
Another explanation lies in reputation mining. Think of it like doing something profitable but unsustainable.
Curious what kind of experience or data sources you base your first graph on — the Pareto principle piece makes sense. But isn’t it possible the very end of the tail is still profitable, albeit less so, than the head?
i think it's more likely that no business will know in advance who those 100 employees that create value are. It might even be that the combination of those 100 creates value beyond the sum of them individually (lets say, they are in one team).
There are also mundane work - fixing bugs or upgrading libraries, as well as keeping up the infrastructure running.
And then there's the middle management - that comes from the mis-trust that upper management generally have for the "grunts", and this happens all the way thru the organization.
> private equity exists...
and if you look at those companies that did this - very few of them becomes the next big thing. Very few of them actually innovate. Very few of them, even survive.
Private equity is really about taking risks with a failing business, e.g. buying it really cheap, and cut the fat and ride out the storm for hopefully a better future. I dont think any private equity will buy a company that isn't failing in some ways, and try to "make it better".
I'm not personally a fan of it, but I get why it works this way. The financial loss if you break something adds up in an excruciating fashion on a per hour basis.
Corona spoke up and pointed out the Emperor had no clothes.
So you need more engineers because you hired more middle managers?
That's a little absurd in my opinion but I am likely wrong in this assumption.
It just seems like the reaction to "We have lots of approvals" should be figuring out how to reduce the amount of approvals needed, not hiring more engineers to increase velocity.
- Things truly requiring management handling (like raises, big expenses, hiring, etc.) took FOREVER in the best case because of the bottleneck at the top. More commonly, they got lost/dropped without explanation.
- Flow of sensitive information was slow and inconsistent because the only way to communicate with a flat org is all-or-nothing.
- Specialist engineers found themselves trying to track too many projects before they realized the overload was a problem (this was me)
- The president (and effective sole owner) was still deeply involved in all projects and regularly exercised veto rights. As the project load increased, the vetos landed later and later - some times after we'd already spent 50% of the budget! This left our customers up a creek and our reputation suffered.
Also, UX wants to see the final result on their screen before they approve your build, so before deploy you have to push into a test cluster, but there are only 10 of those, and you need to wait until one of them is free.
And of course your Jenkins server takes 2h to build and fails sometimes, so pushing small changes takes 2h because you have to babysit the build. The testing cluster above also takes 2h to build. CTO believes in engineers taking care of their code, so SRE is not allowed to handle the builds.
And then there's an average of 2h of meetings per day (I'm not including the 40 minute daily scrum because your squad has 15 people). Half of them to discuss the production incident you had because you didn't see the Slack alerts when you were in another meeting. The other half is chapter and guild meetings where you don't talk, but participation is mandatory. On Mondays you have sprint planning, so forget about coding.
Engineering fact: A 300 watt amplifier is only twice as loud (+10 dB) as a 30 watt amplifier. The same math works for engineering teams.
Then, the machine gets bigger and the dependency graph gets more complicated. Suddenly, if you change one small part of your module, you run a very real chance of ruining something someone in a faraway department relies upon. So, the machine evolves to have things like coding standards and more formalized review processes. Unfortunately, the machine is composed of humans and humans aren’t as simple as just pulling it all apart, greasing it up and putting it back into service. So every single time you add a review step, there’s a chance it genuinely adds to quality (or removes risk). But, there’s also a chance that that change was about ego and organizational power. Then, in five to ten years, they hire MBAs to come in and lay people off.
Edit - Good heavens, this sounds bleak.
It is sometimes better to go slower than you could and be more confident that the product is acceptable. Approvals can but don't necessarily help with that.
And you can cycle through this a few times. And yes, people will also cycle through "we need to reduce bottlenecks" and such, but at this point, you've been successful, you've got a bunch of money coming in, and pissing off your clients is something to be avoided, since individual efficiency is not the end goal by itself.
Companies often implement more process for good reason.
The antithesis of old Facebook's Move Fast And Break Things is to have policies that ensure good security, stability, accessibility, privacy, internationalization, meeting legal requirements, etc. and that usually means processes to ensure that eager engineers and product managers actually meet those policies, instead of rushing to launch.
It's easy to scoff at process when you're a small company or startup whose mistakes will go largely unnoticed unless you suddenly go viral. If you're a Big Tech Company, accidentally messing up security or privacy somewhat means getting a front page article on The Verge detailing your crimes.
Working at Google now, there's plenty of process involves in developing and launching a thing, but I can't really say any of the pieces are without merit.
I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance.
Jesus, what company was that? And how about libraries included... Will this lawyer now also need to go through that?
Surprisingly, or hopefully not, the vast majority of the company is more directly revenue generating; sales and customer acquisition. Its spread very globally; at least as of a few years ago, the SF HQ didn't house any sales people, though I think they had an office in Oakland which did.
Point being, I always got the impression that their engineering team was very "right sized" given their revenue and product scope.
Engineering was mostly based in SF but with satellite offices in London and Hamburg. It looks like they were expanding in a large way into a Toronto office too.
Infrastructure was around a fifth of that total but included verticals like search and machine learning infrastructure.
Yelp turned off their last datacentre in early 2018 (IIRC) - and is entirely on AWS.
As an example: a retail transaction database. Or a calendar booking system. That should be really simple, right? Just record who ordered what, when they received it, who sold it, etc. Who reserved the room, who to send invites out to, simple?
No, it turns out that you have to also take into account people whose orders got delayed by the human-based shipment system, people who got coupons and they expired/want to extend the coupon, people who returned the merchandise and never got any confirmation that it was received back. Or what if someone modifies the meeting location after people accepted -- does a minor change to the meeting description trigger a re-invite or update, or not? Can meeting rooms be held by more than one person at a time? Or what if you want to tie it to the email marketing system that wasn't properly integrated or planned to be integrated -- that's another couple of engineers who have the thankless task of maintaining ETLs that constantly break whenever a change comes along.
Or in the case of Yelp, I'm sure there's small teams who are responsible for the mundane tasks of how to keep track of when a business closes, or reopens, or temporarily shut down due to virus situation -- how are the entries for those businesses supposed to be updated? We never had a field for "closed by mandatory government order" -- that's gonna take a refactor of xyz to implement, etc. etc. We have users who review things, and then those users someday die/go idle/get banned. What happens to the ranking of their reviews? It goes on and on.
(and usually, the people who take the time to think about these things in advance set themselves up for much less pain, and far fewer people needed to fix it, later)
I am not sure people downvoting me understand what all goes into an EHR/practice management software. Calendar booking? Yep. Billing. Oh hell. Don't get me started on the byzantine mess that medical billing is! Then there's charts with icd-9/10 hell. And then everyone you sell it to wants their own charts a little different. Patient records, insurance, I mean, it goes on. The database had several thousand tables. It was insane.
Maybe it's not a good thing that only 50 engineers were working on that system.
An old EHR system from years ago does none of those things, doesn't do engineering heavy things like live updates with thousands of A/B/C/D tests running simultaneously with multiple clients for multiple operating systems and so on.
http://d18rn0p25nwr6d.cloudfront.net/CIK-0001345016/360caaf2...
Page 5: "Our sales force consisted of 3,844 employees as of December 31, 2019..."
Page 12: "As of December 31, 2019, we had 5,950 employees globally."
For example, just one marketing VP with a salary of $120,000 could have an advertising spend budget of a few million.
But in Yelp's specific case, they do have a ton of sales/customer service people who most definitely are among the first to go, along with engineers who are working on projects that are not related to keeping the website from 404ing
Even most programmers -- who are well aware of the hidden complexity of the products they themselves work on -- usually seem to believe that this must not hold true for the products of other companies.
For every feature you're aware of on a random consumer tech product or service, there are probably ten or twenty more that don't concern you, that you haven't seen, or that you don't care about.
So if you’re designing your own CTA, just copy Wal-Mart’s or Amazon’s or Airbnb’s and voila.
In a way, I wish I got to work at Wal-Mart or McDonalds just to get a 1st-Hand view of their processes. Would beat any “LEAN” course.
When I was contracting I was always surprised how outwardly similar businesses could have vastly different IT functions. Sometimes an order of magnitude larger for no apparent gain.
It begins innocuously enough. It takes just one or two mid-level manager empire-builders to join the company. Thanks to the funding, the product and business want growth at all cost which means everything needs to be built and launched yesterday. The engineering-managers say well sure, I need 100 engineers. And so the game begins; the early empire-builders get promoted to dizzy heights.
It doesn't take much time for others to take notice that they need to build an empire to get promoted and every mid-level manager is hiring like there's no tomorrow. In no time you have an incredibly bloated org. In this environment not building an empire is not an option; teams with fewer than 10 members don't get new-shiny projects and are brutally sidelined.
[1] I've been on both sides. The start-up I was working in got acquired by a behemoth that had skilled empire-builders. My team got massacred. In my next gig I had learned my lessons so hired like crazy and in no time had 40 engineers reporting (direct/indirectly) to me 15 of whom had any real work to do. The rest were building random CRUD apps. In the end, realized that empire-building is not for me; I enjoy running a small tight knight team producing high throughput and impactful work.
Logically, that would extend to people whose job revolves around signing up restaurants to marketing portals. The restaurants that have an online business are already there. Those that don't have an online business are already closed.
Programmers arent as easy to replace because they already understand the system. Training takes time, and if there's not enough documentation then you'll end up contacting the previous programmers for help (I've seen it happen) anyway.
The thing that truly influences "hard to hire" is supply and demand. With the influx of bootcamps Software engineering supply has increased and with COVID-19 decreasing demand Engineers may no longer have the luxury of picking and choosing like they used to.
I'm an engineer btw. Just viewing the situation from a realistic lens.
While the supply has increased, the number of senior engineers hasn't grown at the same pace. I know of companies who've fired even their senior engineers, but they've been hit particularly hard by this crisis (think 'business model is literally rendered invalid' level of affected). Most people who think they'll be able to weather at least the next six months will be reluctant to fire more experienced engineers, because they know how hard it was to hire them to begin with.
While some of the options might be gone, chances are people who get laid off or are still in the market will look for companies with better chances of surviving this (think Google, Apple, Facebook, etc.), so the pool of experienced talent might actually be reduced for startups.
In other words: people with 5 years of industry experience are playing in a completely different arena than more junior engineers, who are unfortunately in a more precarious situation.
The logic is sound, you have a point. But I think it doesn't play out this way in reality. Maybe it's like this for your current company, but is that the general trend?
I know that during the dot com bubble bursting many years ago what you said was not the case. Could be different now but do you have any other supporting evidence?
Mind you the situation is different now so what you say could be right in a certain way.
More generally, it's kind of amazing that any company with 178 million unique "customers" only has a few thousand people working there. I know this is the norm nowadays due to the power of computers and the Internet, but I imagine how many people would have been employed to perform the same tasks fifty years ago and suspect it would have been three orders of magnitude higher (and completely uneconomical to do).
Progress is a pretty impressive thing.
I know you're not saying this, but I'd caution anyone against saying things like "<company>'s product is just a glorified <something simple>". It's very hard for people not working directly on a product to truly understand its complexity and everything that goes into it.
EDIT: here's where I read that https://danluu.com/sounds-easy/
I'm sure the Owner of Botto Bistro [1] is having a proper side-splitting laugh right about now hearing this news given all he [2] has gone through.
In short: they have 6000 because that's what leadership thought they could afford. Obviously a 10-year bull market leading up to the current lockdown situation puts them in a rough spot.
There is this canon dogma about 'market efficiency keeping things lean and fit', which is so obviously false and disproven by every single Corp out there, and yet is allowed to stand. Curious.
Yelp's business model requires a lot of cold calls to small businesses.