Comment by gravypod | Hacker News Reader

Comment by gravypod | Hacker News Reader

Parent Full thread

There are many people who are commenting that this is so much work for 20m lines of code, they're reinventing the wheel, they should have just bought something off the shelf, etc.

Two things to consider:

1. It is sometimes hard to buy things that works well: https://danluu.com/nothing-works/

2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo.

It is very difficult to understand how big these two things play into these decisions. Having ~20 people manage build tooling, build caching, source control, CI (build+test), and building release artifacts (which it sounds like Pants did) is not a bad deal.

Consider:

1. GitLab will charge you $20/month/user.

2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.

3. You'll still need to hire a person or two to negotiate the purchasing of the GitLab.

Now remember that GitLab might not be prefect for your use case. GitLab doesn't increase the performance of builds on your laptop by sharing a cache with all engineers. There's also a limit to the mount of data you can store for source + build artifacts (50GB). The CI runners they host for you are very expensive for how slow the machines are ($10/1000 minutes).

Sure, you can do things like run your own GitLab runners but now you're back to running your own infrastructure which is an engineer's job. Sure, you could save money by using Gitea or something that if free but again hosting your own infra. As soon as you have someone managing that infrastructure they're bound to say "how else can I save time and money" and at a certain scale these things become reasonable or even required.

Also it's needed to remember that Twitter had 100 million users in 2012. A lot of "off the shelf" infra that we can use today just did not exist in 2012. They had to build for scale at that time, and a lot of those choices might not translate easily to something off the shelf now.

> A lot of "off the shelf" infra that we can use today just did not exist in 2012

Do you happen to have some examples? Very curious to hear about this.

First version of Docker was released in 2013 and was a curiosity for a few years.

Kubernetes was launched in 2014, and similarly, was mostly a curiosity for a few years. Twitter at that time made an (probably?) unlucky bet on Mesos, a competing but not that similar technology, that is now pretty much dead.

Google Cloud launched it's VM service in beta in 2012. A lot of base AWS services did exist, but was similarly a curiosity: AWS 2012 revenue was 2 billion. This year they are on track to 80 billion.

I mean, even like J̶e̶n̶k̶i̶n̶s̶ and Kafka were barely a year old in 2012. Hadoop wouldn’t be replaced by Spark for 2 more years. Mesos was in its infancy.

Other companies had similar problems to solve, and they also had to build it themselves for the most part.

Jenkins came out in 2005. 2011 was just when the project was renamed from Hudson due to Oracle trademarking the original name.

Bazel was released in 2015

kubernetes

> 2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo.

In theory, maybe. But can anyone external honestly look at what Twitter has shipped for the last few years and justify that it was at all likely their productivity problems in engineering had anything to do with the build system.

I've worked as an FTE in a platform team for a major public tech company before, and I don't know many smart ones that came out of that thinking it was a good trade off to have teams whose very survival necessitated potentially unnecessary platform tooling / changes -- unless you're literally Google or FB and can see measurable results from optimizing that tooling.

Twitter never got close to that point.

> But can anyone external honestly look at what Twitter has shipped for the last few years and justify that it was at all likely their productivity problems in engineering had anything to do with the build system.

You say you've worked at a "major public tech company before", so why would you assume Twitter's paralysis was the fault of engineering and not management/product?

> If you have 2000 SWEs you'll pay 40k/month just to have access to the website.

That doesn't sound like a bad deal if you have to hire such a big build team; let's say you could fire 10 people, 10 * 150.000 (rough estimate) / 12 = 125k/month. That's a 85k/month saving.

I highly doubt GitLab fits the need of such massive monorepos; my only point is that the financial argument is not very strong IF GitLab were a viable alternative to such a big build team.

2,000 SWEs @ $100K each, $200M/year. $40K doesn't sound so bad (my large enterprise employer licenses GitLab Premium and it's worth every cent we pay).

We already hire the people who negotiate the purchase- it's the same people who manage the infrastructure for our onsite instances, etc. IE, that cost is already priced in.

We run our own runners, on a nodegroup in k8s, which downsizes after the runners go idle for a while.

In enterprise, many attempts to save or squeeze a last penny out of something just ends up moving the costs elsewhere in the company, or reducing productivity, or driving away good employees.

> 1. GitLab will charge you $20/month/user.

> 2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.

So the cost of just 1 more engineer... 2001 instead of the 20 extra that this build team creates.

In 2012, GitLab was just a HN beta. https://news.ycombinator.com/item?id=4428278

In that year, Twitter had 100 million users and already functioning build system, that they opensourced soon later.

It must be tempting to cut the Gordian knot by requiring everything to be written in Go, and built from source in a monorepo.

There'd be exceptions, but you'd save a lot on build cost/time and dependency and artifact management.

Well said. The outcome of the Bazel/Pants changes along with the avoidance of the "just buy it" from the outset will pay dividends in efficiency and savings forever (current-events-doomsaying aside).

Reply on news.ycombinator.com