Hacker News Reader

> Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times.

I used to work on a Unicorn a few years ago, and this hits close to home. From 2016 to 2020, the pages didn't change one single pixel, however there we had 400 more engineers working on the code and three stack iterations: full-stack PHP, PHP backend + React SSR frontend, Java backend + [redacted] SSR frontend (redacted because only two popular companies use this framework). All were rewrites, and those rewrites were justified because none of them was ever stable, the site was constantly going offline. However each rewrite just added more bloat and failure points. At some point the three of them were running in tandem: PHP for legacy customers, another as main and another on an A/B test. (Yeah, it was a dysfunctional environment and I obviously quit).

> Yeah, it was a dysfunctional environment and I obviously quit

What do you think could management have done better to make it not dysfunctional and have people quitting?

I think just common sense and less bullshit rationalisation would have been enough.

They had a billion dollars in cash to burn, so they hired more than they needed. They should have hired as needed, not as requested by Masayoshi Son.

They shouldn't be so dogmatic. Some teams were too overworked, most were underworked (which means over-engineering will ensue), but no mobility was allowed because "ideally teams have N people".

They shouldn't be so dogmatic pt 2. Services were one-per-team, instead of one-per-subject. So yeah, our internal tool for putting balloons and clowns into images lived together with the authentication micro-service, because it's the same team.

Rewriting everything twice without analysis was wrong. The rewrites were because previous versions were "too complex" and too custom-made but newer ones had an even more complex architecture, but "this time it's right, software sometimes need complexity".

Believing that some things were terrible would have gone a long way. Launching the main node.js server locally would take 10 to 20 minutes to launch, while something of the same complexity would often take about 2 or 3 seconds. Of course it would blow up in production! Maybe try to fix instead of ordering another rewrite.

They were good people, I miss the company and still use the product, but it didn't need to be like this.

> They shouldn't be so dogmatic pt 2. Services were one-per-team, instead of one-per-subject.

Where the heck did this come from? AIUI, the ideal is supposed to be one-team-per-service, not one-service-per-team.

It comes from a dogmatic reaction against microservices. Microservices were problematic in certain ways, but instead of analysing what went wrong and why, they just went the opposite direction and started doing "big services only". It was a misguided approach, plain and simple.

Interestingly due to internal bureaucracy and understaffing in some teams, there was a lot of "multiple-teams-per-service", which yeah, is another issue in itself.

Favorited (https://news.ycombinator.com/favorites?id=akkartik&comments=...)

I don't know your specifics, but I have worked on some large scale architecture changes, and 200 engineers + 2 year feature freeze is generally not a reasonable ask. In practice you need to find an incremental path with validation and course correction along the way to limit the amount of concurrent change in flight at any moment. If you don't do this run a very high risk of the entire initiative collapsing under its own weight.

Assuming your estimation is more or less correct and it really is a 400 eng-year project, then you also need political capital as well as technical leadership to make it happen. There are lots of companies where a smart engineer can see a potential path out of a local maximum, but the org structure and lack of technical leadership in the highest ranks means that the problem is effectively intractable.

>I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future

sounds like a typical massive rewrite project. They almost never succeed, many fail outright and most hardly even reach the functionality/performance/etc. level of the stuff the rewrite was supposed to replace. 2-4 years is typical for such glorious attempt before being closed or folded into something else. Management in general likes such projects, and they declare victory usually around 2 years mark and move on on the wave of the supposed success before reality hits the fan.

>to convince anyone to take engineers out of product development.

that means raiding someone's budget. Not happening :) New glorious effort needs new glorious budget - that is what management likes and not doing much more on the same budget as you're basically suggesting (i.e. i'm sure you'll get much more traction if you restate your proposal as "to hire 200 more engineers ..." because that way you'll be putting serious technical foundation for some mid-managers to grow :). You're approaching this as an engineer and thus failing in what is the management game (or as Sun Tzu was pointing out one has to understand the enemy).

My impression has always been that FAANG need lots of engineers because the 10xers refuse to work there. I've seen plenty of really scalable systems being built by a small core team of people who know what they are doing. FAANG instead seem to be more into chasing trends, inventing new frameworks, rewriting to another more hip language, etc.

I would have no idea how to coordinate 200 engineers. But then again, I have never worked on a project that truly needed 50+ engineers.

"Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog." Probably that's 4 friends in a basement, similar to the core Android team ;)

Your impression comes from the fact that you have not worked at larger teams, as you said so yourself. It's relatively easy to build something scalable from the beginning if you know what you need to build and if you are not already handling large amounts of traffic and data.

It's a whole different ballgame to build on top of an existing complex system already in production that was made to satisfy the needs at the time it was built but it now needs to support other features, bug fixes and supporting existing features but at scale while having 50+ engineers not step on each other and not break each others code in the process. 4 friends in the basement will not achieve more than 50+ engineers in this scenario, even when considering the inefficiencies of the difficulty in communication that come along with so many minds working on the same thing.

GP said they have never work on something that truly needed 50+ engineers. Truly being the keyword here IMO.

I have worked on a 1000+ engineer project and another that was 500+, but I'm on the same boat as GP. Both of those didn't needed 50+, and the presence of the extra 950/450 caused several communication, organisational and architectural issues that became impossible to fix on the long term.

So I can definitely see where they're coming from.

I've long wondered what I might be able to keep an eye out for during onboarding/transfer that would help me tell overstuffed kitchens apart from optimally-calibrated engineering caves from a distance.

I'm also admittedly extremely curious what (broadly) had 1000 (and 500) engineers dedicated to it, when arguably only 50 were needed. Abstractly speaking that sounds a lot like coordinational/planning micromanagement, where the manglement had final say on how much effort needed to be expended where instead of allowing engineering to own the resource allocation process :/

(Am I describing the patently impossible? Not yet had experience in these types of environments)

> a lot like coordinational/planning micromanagement, where the manglement had final say on how much effort needed to be expended where instead of allowing engineering to own the resource allocation process

Yep, that's a fair assessment!

The 1000+ one was an ERP for mid-large businesses. They had 10 or so flagship products (all acquired) and wanted to consolidate it all into a single one. The failure was more on trying to join the 10 teams together (and including lots of field-only implementation consultants in the bunch), rather than picking a solid foundation that they already owned and handpicking what needed.

The 500+ was an online marketplace. They had that many people because that was a condition imposed by investors. People ended up owning parts of a screen, so something that was a "two-man in a sprint" ended up being a whole team. It was demoralising but I still like the company.

I don't think it's impossible to notice, but it's hard... you can ask during interviews about numbers of employees, what each one does, ask for examples of what each team does on a daily basis. Honestly 100, 500, 1000 people for a company is not really a lot, but 100, 500, 1000 for a single project is definitely a red flag for me now, and anyone trying to pull the "but think of the scale!!!" card is a bullshit artist.

Yay, I'm learning :D

> trying to join the 10 teams together

oh no

(insert https://webcomicname.com/ here)

> rather than picking a solid foundation that they already owned and handpicking what needed.

Mmmm.

I wonder if a close alternative (notwithstanding lack of context to optimally calibrate ideas off of) might have involved leaving all the engineers alone to compare notes for 6-12 months with the singular top-down goal of "decide what components and teams do what best." That could be interesting... but it leans very heavily on preexisting competence, initiative and proactivity (not to mention conflict resolution >:D), and is probably a bit spherical-cow...

> The 500+ was an online marketplace. They had that many people because that was a condition imposed by investors.

*Constructs getaway vehicle in spare time* AAAAAaaaaaa

Sad engineering face :<

> I don't think it's impossible to notice, but it's hard... you can ask during interviews about numbers of employees, what each one does, ask for examples of what each team does on a daily basis.

Noted. Thanks.

> Honestly 100, 500, 1000 people for a company is not really a lot, but 100, 500, 1000 for a single project is definitely a red flag for me now, and anyone trying to pull the "but think of the scale!!!" card is a bullshit artist.

That makes a lot of sense, and also filed away.

Also, I recently read this which resonates quite strongly with the economy-of-efficiency scale problem (which I totally agree with): https://rachelbythebay.com/w/2022/01/26/swcbbs/, and the update, https://rachelbythebay.com/w/2022/01/27/scale/

> what I might be able to keep an eye out for during onboarding/transfer that would help me tell overstuffed kitchens apart from optimally-calibrated engineering caves from a distance

The biggest thing I've been able to correlate are command styles: imperative vs declarative.

I.e. is management used to telling engineering how to do the work? Or communicating a desired end result and letting engineering figure it out?

I think fundamentally this is correlated with bloat vs lean because the kind of organizations that hire headcount thoughtlessly inevitably attempt to manage the chaos by pulling back more control into the PM role. Which consequently leads to imperative command styles: my boss tells me what to do, I tell you, you do it.

The quintessential quote from a call at a bad job was a manager saying "We definitely don't want to deliver anything they didn't ask for." This after having to cobble together 3/4 of the spec during the project, because so much functionality was missed.

Or in interview question form posed to the interviewer: "Describe how you're told what to build for a new project." and "Describe the process if you identify a new feature during implementation and want to pitch it for inclusion."

Of course. Wow, I never thought about management like that before. But particularly in software development it makes so much sense for people to jump toward this sort of mindset.

There really is an art to scaling problems to humans so the individual work (across management and engineering) falls within the sweet spot of cognitive saturation. TIL yet another dimension that can go sideways.

The signal to noise ratio is very appreciated.

Yeah, exactly. There is overhead simply because of the (necessary) cross-communication at that scale, and there's overhead from legacy support, but here's a thought experiment. Imagine that you've built the most perfect system from scratch that you can think of. Fast forward five years, and the business has pivoted so many times that system is doing all sorts of stuff it just wasn't designed for, and it's creaky and old. It just doesn't fit right anymore and even you want to throw it away and build a new one. So you form a tiger team full of the smartest people you know to greenfield build a new one, from scratch, but that's gonna take two years to write. (You think, hey, maybe we could just take this open source thing and adapt it to our purposes. To which I say, where do you think large open source projects come from‽)

How do you bridge the two systems? You build an interim system. But customers want new features, so those features need to be done twice (bridge+new) if you're lucky, three times (existing+interim+new) if not. Could a smaller team of 10x engineers come in and do better? First off, thanks for insulting all of us, as if none of us are 10x-ers. But no. There's simply not enough hours in the day.

We've all heard of large IT projects that failed to land and said "of course". But we don't hear about the huge ones that do. And plenty of them do land, quite succesfully, with these 200+ person teams where I, as an SRE, don't know the code for the system I'm supporting.

None of this is visible from the outside.

> I've seen plenty of really scalable systems being built by a small core team of people who know what they are doing.

There is huge difference between building a system that could theoretically be scaled up and actually scaling it up efficiently.

At small scales, it's really easy to build on the work of others and take things for granted without even knowing where the scaling limits are. For example, if I suddenly find I need to double my data storage capacity, I can drive to a store and come back with a trunk full of hard drives the same day. I can only do that because someone already build the hard drives, and someone stocked the nearby stores with them. If a hyperscaler needs to double their capacity, they need to plan it well in advance, allocating a substantial fraction of global hard drive manufacturing capacity. They can't just assume someone would have already built the hardware, much less have it in stock near where it's needed.

Which FAANG is rewriting to another hip language and chasing trends (especially when it comes to infra services??)? I don't mean to be rude, but it doesn't sound like you are talking about any of the FAANGs, this sounds completely made up.

https://blog.pragmaticengineer.com/uber-app-rewrite-yolo/

FAANG is an acronym for Facebook, Amazon, Apple, Netflix, Google. Uber isn't in the same ballpark as those companies (arguably Netflix isn't really in the same ballpark as the other four either...).

Heh, I wish they still looked the same. They added an order of magnitude of HTML and JS bloat while removing functionality.

Had that issue in my previous job.

Higher management decided to migrate our properitary vendor locked platform from one cloud provider to the other one. Majority of migration fell on a single platform team that was constantly struggling with attricion.

Unfortunately I was not able (neither our architects) to explain the higherups that we need bigger team and overall way more resources to pull that off.

Hope that someone that comes after me will be able to make the miracle happen.

I usually move on to a different project/team/company when it gets to this. E.g. my new team builds a new product that grows like crazy and has its own set of challenges. I prefer to be deliver immediate customer value vs. long term hard to sell and hard to project the value work.

"That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user"

It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill something explodes?

a manufacturing company is designed the ground up to works whit machine but isn't the same whit software, is hard to understand that triple data isn't only triple server but a totaly different software stack, and exponentially more complex is not only put more factories like textile.

There's still order of magnitude change analogies to real world processes, if people are willing to listen (which is the hard part). Use something that everybody can understand, like making pancakes or waffles or an omelet. Going from making 1 by hand, every 4 minutes at home for your family, to 1,000 pancakes per minute at a factory is obviously going to take a better system. You can scale horizontally, and do the equivalent of putting more VMs behind the load balancer, and hire 4,000+ people to cook, but you still need to have/make that load balancer in the first place for even that to work.

That's the tip of iceberg when going from 1 per 4 minutes to 1,000 per minute though. How do you make and distribute enough batter for that system, and plating and serving that is going to take a pub/sub bus, err, conveyor belt to support the cooks' output. Again though, you still gotta make that kafka queue, err, conveyor belt, plus the maintenance for that is going to a team of people if you need the conveyor belt to operate 24/7/52. If your standards are so high that the system can never go down for more than 52.6 minutes per year or 13.15 minutes per quarter, then that team needs to consist of highly-trained and smart (read: expensive) people to call when the system breaks in the middle of the night.