The internal tools at Meta are incredible tbh. There’s an ecosystem of well-designed internal tools that talk to each other. That was my favorite part of working there.
"your type of company" sod off. Meta is only like this because its got a massive advertising revenue stream.
the sheer amount of engineering time wasted because we don't document stuff is astounding.
For example, how many message queue systems do we have?
how many half arsed message queues have been created because they didn't know about FOQS?
Lots of nonserious companies that take those issues as enough of a reason to move slowly.
Many fewer serious ones where bad tooling is expected to be fixed, smoothed over, or replaced entirely in the interest of future dev time.
Yes, thats great.
> being able to land a diff to fix the issue is awesome imo.
yes, if its a one off. but for my last project that would involve spinning up many "XFNs" (multi-team chat fests) to argue that actually they don't want to have that change because of reason x,y and z.
At which point you just give up and make a stupid fucking hack.
So much is not about engineering excellence, its about trying to get people to accept change.
Except you're not really sure if there has been a murder, or sometimes you wonder if you're the murderer, because at every turn you're told that you've been a bad dev for trying x,y and z
Since Bento doesn't appear to be usable by the public, aparallel version of this that people can get a feel for cross-tool integration would be Google's Colaboratory / Colab notebooks (https://colab.research.google.com/) that have many baked-in integrations driven by actual internal use (i.e. dogfooding).
For any kind of general Python/C++ work, its a _massive_ pain.
The integrated debugger rarely works, and its a 30 minute recompile to figure that out. The documentation for actually being efficient in build/run/test is basically "ask the old guy in the corner". You'd best hope they know and are willing to share.
The code search is great! The downside is that nobody bothers to document stuff, so thats all you've got. (comments/docstrings are for weaklings apparently)
You want to use a common third party library? You'd best hope its already ingested, otherwise you're going to be spending the next few days trying to get that into the codebase. (yes there are auto tools, no they don't always work.) Also, you're now on the hook to do security upgrades.
I assume all of the big tech companies host internal mirrors of every single code dependency + tooling. Otherwise they could not guarantee that they can build all of their code.
That decision is also illustrative of why they end up forking most things - Facebook's usage patterns at the far extreme end for almost any tool, and things thats are non-issues with fewer engineers or a smaller codebase become complete blockers.
At Meta, N tends to be hundreds of billions to hundreds of trillions.
So your algorithm REALLY matters. And git has a Big-O that is worse than Mercurial, so we had to switch.
Nail in the coffin on this was a benchmark GitHub ran two years ago that got the results that FB should have: git status within seconds.
Facebook didn't use mercurial because of big O, they used it because of hubris and a bad disk config.
Anyways:
1. Github benchmark: https://github.blog/engineering/infrastructure/improve-git-m...
2. The original email thread: https://public-inbox.org/git/CB04005C.2C669%25joshua.redston...
3. There's another email thread that gets linked everywhere - but in light of the prior thread, the numbers don't track: https://public-inbox.org/git/CB5074CF.3AD7A%25joshua.redston...
I recall there being a message from someone either at AirBnB or Uber who mentioned that they have a similar monorepo but without the slow git status, but can't seem to find it now - it's likely on one of the other mailing list archives but didn't make it to this one.
Point being that painting this as "the community was hostile" or "git is too slow for FB" is just disingenuous. The FB engineer barely communicated with the git team (at least publicly) and when there was communication, it was pushing a single benchmark that was deeply flawed, and then ignoring feedback on how to both improve the performance of slow blame, commit by repacking checkpoint packfiles (a one-off effort) and also ignoring feedback that the benchmark numbers didn't make sense in absolute terms.
Half-remembering a blog post I read - the git maintainers also wouldn't give Facebook the time of day on code changes to accommodate FBs requirements. Mercurial was more amenable. This also disproves the "Facebook has a fork of evertyhing, because the attempted to upstream the changes they wanted)
Plenty of other customers with the same magnitude problems as Meta are using Git perfectly fine.
I mean there aren't. there are perhaps three places that have the same scale problem.
A monorepo for a place with about 50k developers, that has been operating at that scale for 5 years.
The current checkout if not sparse would be >80gigs
The commit rate is > 20 a second.
no amount of branching strategy is going to help on that.
I love git, I used it professionally since 2010, but git is not a good fit for something _massive_
The few things that don't have forks are usually the open source projects like React or PyTorch, but even those have some custom features added to make it work with FB internals.
Google also maintains a monorepo with "forks" of all software that they use. History diverges, but is occasionally synchronized for things like security updates etc.
[0]: http://www.boa.org/
zeus likewise.
my memory is hazy, but when i started i was called an sre. Then someone made up the term "app ops". then it was production engineering. then I called myself an SRE when i was interviewing to leave fb.
Oh, and i was called a DBA before being acquired by google.
fork open source, then demand resources to maintian this monster.
easiest promotion + job security.
its even called "Platform Engineering" these days
I interned thrice as phd student at FB. your friend isn't entirely wrong but also just doesn't have enough experience to judge. all enormous companies are like this. FB is far and away better than almost all such companies (probably only with the exception of Google/Netflix).
lol bruh read my response again - FB's and Google's and Amazon's tool are lightyears ahead of #ARBITRARY_F100_COMPANY. you haven't a clue what "bad" means if you've never worked in a place that has > 1000 engineers.
I'm not sure what your idea about PRs and features has to do with the above... he's not there to work on the internal infra framework. He's there for ML stuff. Unfortunately, the road to the later goes through the former, but he's not really a kind of programmer who'd deal with Facebook's infrastructure and plumbing.
The point is, it's inconvenient. Is it inconvenient because Facebook works on a five-year plan basis or whatever other reason they have for it doesn't really matter. It's just not good.
I also have no problems admitting that all big companies (two in total, one being Google) I worked for so far had bad internal tools. I don't imagine Facebook is anything special in this respect. I just don't feel like it's necessary to justify it in any way. It's just a fact of life: large companies have a tendency to produce bad internal tools (but small often have none whatsoever!) It's a water is wet kind of thing...
My idea is if he's not making the monorepo codebase changes himself, he's going to have to wait for an awfully long time for any non-trivial improvements he'd like because the responsible teams have different priorities sketched out for next calendar year. It's a function of organization size, unless you have the support of someone very high up on the org chart, ICs can't unilaterally adjust another teams priorities.