Hacker News Reader

Comment by tallowen | Hacker News Reader

I always find these comments about interesting, having worked at Facebook and Google, I never quite felt this way about Google's Monorepo. Facebook had many of the features you listed and quite performantly if not more so. Compared with working at Facebook where there are no owners owners files and no readability requirements, I found abstraction boundries to be much cleaner at FB. At google, I found there was a ton of cruft in Google's monorepos that were too challenging / too much work for any one person to address.

View on HN

For non-Googlers like me, here's some background about "readability":

https://www.moderndescartes.com/essays/readability/

OWNERS files rarely get in the way - you can always send a code change to an OWNER. They are also good for finding points of contact quickly, for files where the history is in the far past and changes haven't been made recently.

Readability really does help new engineers get up to speed on the style guide, and learn of common libraries they might not have known before. It can be annoying - hell, I'll have to get on the Go queue soon - but that's ok.

This isn't true at all for OWNERS files. If you try developing a small feature on google search, it will require plumbing data through at least four to five layers and there is a different set of OWNERS for each layer. You'll spend at least 3 days waiting for code reviews to go through for something as simple as adding a new field.

3 days for a new change on the biggest service on the planet? Not bad.

I agree that it could be worse! Facebook has significant (if not more) time spent and I found adding features to news feed a heck of a lot easier than adding features that interacted with google search. Generally a lot of this had to do with the number of people needed to be involved to ensure that the change was safe which always felt higher at Google.

I have the same background: I find the code quality at G to be quite a lot higher (and test pass-rate, and bug report-rate lower) than News Feed, which was a total shit-show of anything-goes. I still hold trauma from being oncall for Feed. 70 bugs added to my queue per day.

The flip side is of course that I could complete 4 rounds of QuickExperiment and Deltoid to get Product Market Fit, in the time it takes to get to dogfooding for any feature in Google.

I'm only an outside observer in this conversation but could it be that the review process (or the lack thereof) and the ease with which you can add new features has had an impact on the quality of the software?

The thing is, in my experience as a user Facebook (the product, not the former company) is absolutely riddled with bugs. I have largely stopped using it because I used to constantly run into severe UI/UX issues (text input no longer working, scrolling doing weird things, abysmal performance, …), loading errors (comments & posts disappearing and reappearing), etc. Looking at the overall application (and e.g. the quality of the news feed output), it's also quite clear that many people with many different ideas have worked on it over time.

In contrast, Google search still works reasonably well overall 25 years later.

There are pretty different uptime and stability requirements for a social product and web search (or other Google products like Gmail). When news feed is broken life moves on, when those products break many people can't get any work done at all.

One of Google's major cultural challenges is imposing the move slow and carefully culture on everything though.

It’s not considered ok for newsfeed to break. It would be a massive issue that would command the full attention of everyone.

And yet folks who are on call for it say things like this: https://news.ycombinator.com/item?id=40826497

I think I have heard similar things from other googlers and I think there might be two factors on why I think this:

- I worked on Google Assistant which was responsible for integrating many services. This meant I had to work with other peoples code way more regularly that many at google.

- I moved from FB to google - I'm not really sure how many people have had this experience. I think many of my colleagues at google found it surprising how many of the things they thought were unique to google actually also existed at FB.

At the end of the day, any of these processes have pros/cons but I think the cruft of having APIs that are a couple steps harder to evolve due to finding Readability/Owners for everything you touch just makes things slightly less cohesive and a trickier place to have a "good" codebase.

When I worked at FB, I would frequently rebase my code on Monday and find that, for example, the React framework authors or another smaller infra team had improved the API and had changed *every* callsite in the codebase to be improved. This type of iteration was possible in certain situations but was just much less common at google than at fb.

> I think many of my colleagues at google found it surprising how many of the things they thought were unique to google actually also existed at FB.

Google workers are groomed to believe Google is the best, and hence they are too. A corollary of that, then, is that nobody else has it that good, when in fact, others sometimes have it better.

I also made the move from FB to G and echo everything said above. Googlers have a massive superiority complex. In reality, it's naiveté.

My 2 cents: OWNERS is fairly useful, if only as a form of automating code reviewer selection. Readabilty is a massive drag on org-wide productivity. I have had diffs/CLs take MONTHS to be approved by every Tom Dick and Harry whose claws were added to my code and made me re-design whole project approaches, and they were only there because they're supposed to check if my new-lines are in the right spot for that language. I thought about quitting.

People really underestimate how much productivity drain there is in having a bad code review culture. One of the worst things about working at Amazon was that any feedback on a merge request, no matter how small, required you to request a re-review.

It's not culture (organic), it's systems (planned): if you don't, on day zero, agree what code reviews should cover (and what not), then code reviews are a pissing contest first, and a useful tool second.

I noticed a lot of people don't understand both the limitations of code reviews, and what issues they can and should solve. Writing good critique (of anything, not just code) is hard, we don't train people to do it, and usually don't even regard this as something that needs training and understanding.

+1.

Going from FB to $REDACTED to Oculus was a pretty wild ride, there were a lot of different cultures, though I think generally speaking the best qualities filtered through.

(also, howdy former teammate)

(trying to recall unixname to human name mapping.....)

Huh? Facebook has a lot of that infra because ex-Googlers built it there. It takes an insane amount of delusion to notice something common between a father and a son and say that the dad inherited it.

QED

No, we’re quite aware the world outside has been catching up. There’s even a famous doc by a senior director about it…

The system still grooms Googlers to think they're better than though. Until that root cause would be fixed (which would have to be at a huge cost to Google, so no surprise it'd never be), nothing would change.

In Google Cloud at least, we're quite aware we're not the market leader, so there's a pervasive humbleness. We're proud of certain technical achievements (ie: Spanner), but the world can catch up quickly (ie: CockroachDB, Foundation, CosmosDB, etc). This might be a departure from the feeling in older divisions of the company, haha.

Huh, also having worked at both I had exactly the opposite experience. Google’s tools looked ugly but just worked. At meta there were actually multiple repos you might have to touch and tools worked unreliably across them. Owners files made sure there was less abandoned code and parent owners woild be found by gwsqueue bots to sign off on big changes across large parts of the repo by just reading these files.

Same and another vote for meta. Meta made the language fit their use case. Go into bootcamp change the search bar text to ‘this is a search bar!’ press F5 and see the change (just don’t ship that change ;D). It’s incredibly smooth and easy.

Googles a mess. There’s always a migration to the latest microservices stack that have been taking years and will take many more years to come.

Like meta just changed the damn language they work in to fit their needs and moved on. Google rewrites everything to fit the language. The former method is better in a large codebase. Meta is way easier to get shit done to the point that google was left in the dust last time they competed with meta.

I think what you're saying is true for www, but not fbcode, and the later starts to look a lot like google3. I agree though, Meta's www codebase has the best developer experience in the industry.

> no owners owners files and no readability requirements

Move fast and break things, right?

I don’t think that owners files are the best way to ensure things don’t break.

Reply on news.ycombinator.com