Hacker News Reader

I think Facebook has very poor engineering culture. They built their iOS app with 18k classes, then claim that iOS and Git can't handle their 'scale'.

Let's avoid putting here obvious companies and companies who over-engineer, maybe?

http://quellish.tumblr.com/post/126712999812/how-on-earth-th...

https://www.reddit.com/r/programming/comments/3m5n2n/faceboo...

I've spoken to Facebook engineers who work on their internal source control systems, and it was pretty obvious from talking to them that Git wouldn't work for their purposes. There is plenty of writing about this topic that explains in some detail why Git is unsuitable, and why Mercurial was a better option for them.

As for the 18k classes in their iOS app, I don't see why this is a sign of bad engineering? I'd expect that a large number of these are auto-generated, for example from Thrift API definitions, and I'd also expect that what we see as users is the tip of the iceberg in terms of the code needed. There's obviously a lot of analytics code, and also I'd expect the resiliency and reliability needed requires much more code than we'd expect.

It's nothing more complex than 100,000s of other apps out there. Yet somehow they managed to create 18k classes?

Scale is handled at the back-end.

You are completely right.

As a small-company person who migrated a few years ago to one of the Big Four, I can confirm that almost every time someone tells you about "scale" they are bullshitting you. Most developers on those companies either do not deal with scale at all (anyone dealing with front end stuff) or do so by using abstractions which allow them to ignore scale (99% of developers). But scale sounds sexy so everybody pretends they have something to do with it. And it sounds better than "bloat" which is what they really mean.

The other variant of "scale" is "I have this single-computer application running on X hundred thousand servers" or some ambarrasingly parallel problem of that sort. Then they will claim boldly that something that has a 0,0001% chance of happening "happens multiple times a day here" which sounds impressive until you realize that that thing has no impact on the SLA and you get saved by the load balancer so it isn't particularly important anyway.

The 1% of people who work on real problems in the area don't need to brag on the Internet about "scale" because they are too busy working.

That's not quite true - frontend has to scale for features.

I can't immediately think of any apps that are just as complex as Facebook's, but maybe I'm just not thinking hard enough. Could you give me some examples (honest question)?

Unreal Engine.

Based on their documentation—which, albeit, I know has many classes omitted—Unreal Engine has ~6900 classes. That number probably gets greatly multiplied, though, if you count generated classes.

And it's managed in git.

I can't find the link, but I remember an Epic engineer saying that they use Perforce internally and do a 2 way sync with Github.

Correct, it is managed in Perforce and MIRRORED onto Github.

I'd think that the Google+, Google Photos, the Twitter apps are at at least 75% of the complexity of the Facebook app.

Auto-generated code shouldn't be checked in.

I believe the 15k number came from decompiling the binary, not from looking at source control.

But does the app make money? Do people use it?

The light FB app probably does very well by class count metrics, but it's intended for a totally different market. Stateside, much as we may bitch about it, it apparently doesn't matter for an app in FB's position.

I'll be very blunt: In situations like this, they are far far closer to the code than an armchair observer like you. I trust their engineers far more than I respect your pithy complaints.

Even worse, look at their Android app - They made an overly complex open source image loading library which is despised by most quality Android Engineers because it uses parts of memory it shouldn't be using, switching to and from it is terrible, has a big dex footprint and often results in OOM crashes. They use runtime hacks and then complain directly to Android developers when something breaks during updates, their SDK is terrible and pulls a ton of dependencies, they made a "2D physics engine" just to make a bounce animation. Met a dozen of their developers and I have noticed way more arrogance than brilliance. None of them are the kind of people I'd like to work with.

The most mind-boggling thing they did with their Android app (that I know of, at least) was hands down this: https://jaxenter.com/facebooks-completely-insane-dalvik-hack...

The first tumblr link at the bottom in the comments linked me to FizzBuzz Enterprise Edition

https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...

The problem of 'scale' they were addressing in the iOS app wasn't scale of users, but rather scale of devs. They have 18k classes because they came up with a rather clever way of allowing many separate developers to all work on different features of the same app, without stepping on each others toes. This might not seem like a hard problem if you've never worked on a non-trivial iOS app, but it is a pretty impressive feat.

Facebook wants the shortest possible time between product idea and real world users interacting with it, which is why they also have a solid CI/CD pipeline. You might not agree with the tradeoff, but the pragmatism is impressive

The architecture of all software projects eventually matches the structure of the organization writing the software.

> They have 18k classes because they came up with a rather clever way of allowing many separate developers to all work on different features of the same app, without stepping on each others toes.

What makes you believe this is "clever" as opposed to a poor engineering decision? I strongly suspect that FB might have delivered a higher quality product through a more traditional team-ownership development model. Throwing large numbers of developers at a single app is unlikely to lead to a well-architected holistic solution.

All these things also apply to Google, except:

* you decide on a team before you start, but after you know you passed the interview

* moving fast is not valued as much

I would also add that Facebook uses a lot of Open Source, meaning your experience will be applicable elsewhere.

Here are some potential advantages about working at Google:

* Being ethical is a core value of the company.

* Google/Alphabet is more diverse (Search, YouTube, Android, Hardware, DeepMind, Waymo ...) and has more offices worldwide.

* It has an advantage in Machine Learning and Research. Google even has an internal research conference.

* Google even has it's own incubator for entrepreneurial employees (Area 120).

* Google is trying hard to make it's hiring and promotion process objective and unbiased. That's why they happen by a commission of peers.

Being ethical? Maybe the engineers need to have some ethics but not all products do.

I am assuming you are talking about privacy issues. Google collects a lot of data, but is very thoughtful about how they use it, who has access to it and gives users options to view and delete it.

That is what I was referring to. They take measures like you said, but do non-technical people even know about it? It is not advertised as far as I know.

Nobody who's been a victim of ContentID scammers (basically, 80% of YouTube Creators) would never say Google is "ethical".

"Being ethical is a core value of the company."

How does it make them stand out? What does it translate to in practice?

It means that Google often passes on opportunities to make more money if that would hurt or mislead users, advertisers or publishers.

Google's search rankings are unbiased. The ads team is separated from the search team. It's business model benefits from and supports an open internet, unlike other companies which lock users into their ecosystem.

Google is very transparent internally for a company it's size.

Google releases large of amounts of open source software and publishes research. For instance Google published a paper that explains in detail how it's Machine Translation system works.

You should get the YouTube developers in on this, because their ContentID system is about the LEAST ethical piece of software in common use.

In my 2 dozen or so ContentID flaggings, maybe two of them have been by parties with a legitimate claim to the material. All the rest have been fraud.

My most recent "pleasant" experience with ContentID was some scam copyright company claiming they owned the soundtrack to Elder Scrolls: Morrowind. The beauty of this system is that not only are they fraudulently claiming they own that content, but to get my video published I have to swear that I'm not stealing the content they actually stole.

How do I report this scammer to YouTube? Well you can't. There's no automated way, and there's no support whatsoever. They can just scam in peace for years and years and nobody can do anything about it because nobody at Google cares.

"Ethical". Riiight.

https://support.google.com/youtube/answer/2797454?hl=en

Of course you can DISPUTE the claim. And in fact actual scammers will release the claim right away because they make all their scam-dollars off old videos where people don't check the ContentID status every week.

But there's no way to tell Google, "the guy making this claim doesn't own the content, he's a scammer". That's the feature I need.

I am 100% certain that the company claiming to have owned Morrowind's music: 1) Is not Bethesda (the developer of the game) 2) Is not affiliated with Bethesda in any way 3) Has no right to the music

But I have no way to communicate my research to Google. Because, guess what, those "ethical" Google employees make money off the scam, too! They take a percentage of all ad revenue, even if it's a ContentID scammer's ad revenue.

* you decide on a team before you start doesn't seem to be true.

Most of the compliant about Google hiring practice has been along the line of, and I paraphrase "I was promised to be with a team or a position and then I got placed in a totally and differently unrelated position/team"

Unlike most companies you interview for the entire company and not for a specific team at Google. That means you only know what team you are going to be on after you pass the interview. However, you will always know before you start. It mainly depends on what teams have headcount at that time and are a good fit. You can choose to take a pass on a team you are matched with and wait for another match.

>>> Google is trying hard to make it's hiring and promotion process objective and unbiased. That's why they happen by a commission of peers.

How on earth does this follow? I know that peer review is in fashion right now, but without some very strong context, this seems more like a recipe for a Keynesian beauty contest than a useful measure of merit.

It's committee of people from all over the company that don't know the person in question. The candidate prepares a document that shows their contributions with some quantifiable metrics.

There has been research that showed that this kind of decision process decreases bias.

> Being ethical is a core value of the company

> Google is trying hard to make it's hiring and promotion process objective and unbiased

Google tries very hard to appear to be ethical, unbiased, responsible, and generally virtuous. Sometimes this attitude results in good work. Other times, and I think more often, it amounts to empty posturing and counterproductive gestures. In the social sphere, this "be seen as good" attitude manifests itself as annoying virtue signaling, speech policing, and other behavior that you also see on American college campuses. This behavior is annoying, but very common in tech and easy to ignore.

The "be seen as good" attitude also manifests in technical culture, however. There's this weird attitude inside Google that what is painful must therefore be right, as if difficulty were its own reward. While it's true that the right thing is often painful, the converse does not hold, so Google makes some bad decisions.

Google's attitude toward developer friction is, "This process is slowing you down? Too bad. That's how responsible engineers work." This hairshirt-loving attitude leads to large teams, moving slow, and general risk aversion, and I think it leads to bad engineering-cultural choices.

Google: "This language feature causes problems? Let's ban it. We'll add headcount to make up for the productivity drop."

Facebook: "This language feature causes problems? Let's add tooling to warn developers about the dangerous usages and make sure we can quickly find bugs."

I've worked at a lot of tech companies. I don't think the people at Google are actually any more or less ethical than the people at other top-tier tech companies.

> moving fast is not valued as much

It's important not to underestimate how much moving fast makes you happy and how unhappy you become when moving to a slow environment having worked in one that moves fast.

Facebook's elevation of moving fast to a first-class goal goes a long way to slowing the kind of decay that Google has suffered.

"Being ethical is a core value of the company."

Google's business model is pervasive surveillance, so that's a pretty odd thing to brag about.

I agree about Facebook.

They must have a great culture. Look what FB did for the JS ecosystem just recently—yarn, React, React Native, Immutable. Really excellent stuff which you not only need superb people but also a good culture.

So, I'd say that excellent products, both the commercial ones but also open source, are a good indicator for 'culture'.

> With few exceptions, engineers aren't hired for a specific team; they go through bootcamp where they get exposure to all parts of the stack and tasks from different teams.

How is that good? You essentially don't know what you sign up for.

I'm assuming that after the bootcamp they would put you on a team based on your interests and aptitudes. For me, having a choice - or at least feeling like I have a choice - over teams and projects is empowering and exciting.

In other companies, you have that choice during recruitment. Here, you need to basically trust that they won't screw you over ("Look, it turns out that the only opening we have that matches your skills is maintaining this internal CRUD app for HR - but no worries, stick with it for a year and maybe something better comes along!").

But that would be evidence of a bad engineering culture. Which is United has said is not true. So we can probably assume they meant FB's bootcamp is more about placing their engineers on teams they would enjoy and are good at, most of the time.

Exactly! The people you work with are far more important than the tech.

I don't think I could work in an open plan office. I just can't concentrate well enough to do really good work in an environment like that.

Having worked in a bullpen style office (I now work from home), it wasn't the open floor plan that was a problem, it was the number of people management crammed into that open floor plan that became problematic.

At one point, we had completely outgrown our space and we were constantly tripping over one another. It was miserable and everyone was constantly sick. Eventually, we got more real estate and everyone was happy, more productive, and less ill.

As long as you have the real estate, open floor plans are nice and help you feel less imprisoned than cubicles imo.

Oh, cubicles suck too. I prefer an office with a door.

> and help you feel less imprisoned than cubicles imo.

That's not help. I don't feel imprisoned in a cubicle. I like walls, the taller the better.

I'm currently in a short-wall cubicle farm where we have enough space to not to all get sick, and even there the cross-talk from multiple conversations can be completely distracting.

While management/facilities ignorance can definitely make open plan offices worse, I think they're still not for everyone. Some people seem to not mind being distracted, other people are really sensitive to it.

> * With only very few exceptions, every engineer has access to the near-entirety of source code at FB (Oculus, Whatsapp, Instagram...).

Sorry, this could be slightly off topic for this thread. I'm curious if this kind of access also applies to all the data that users put up on the platforms? To put it in other words, how does FB prevent or work to prevent LOVINT [1] like cases? If it's the same kind of access, wouldn't working in the company make the employee uncomfortable that there are people who can look at all information?

[1]: https://www.washingtonpost.com/news/the-switch/wp/2013/08/24...

P.S./disclaimer: I personally dislike FB as a platform and as a company for privacy and other policy reasons I don't agree with (like "authentic names", for one). So my bias may be showing in my question, although it's intended to know more and not meant as an attack.

Why do you think that having access to source code implies having access to customer data?

I do not think that, but am curious to know since the GP seemed to know about how FB works internally.

It sounds like a great place to work at. Kinda sucks that, they only interview new grads from top schools unlike the other Big 4.

Not true. I've interned three times at FB and will join full time, and I'm from a Romanian university that is nowhere near "top".

Definitely wrong. My wife recently interviewed there, not from a top 50 university

I'm quite sure it's not true. There are at least 2 engineers from the React team who have dropped out of "normal" schools/colleges in their first years.

I've interviewed at Facebook 2 1/2 years into my career and have an open invitation to interview whenever I want without having gone to a tier 1 university for undergrad or grad (although I went to a prestigious program for grad math, UIUC) - my education turns out to be a poor indicator of what I can do.

A caveat is that I originally entered FB's pipeline through a referral from a friend who interned and works at FB right out of CMU computer engineering. Referrals are often enough at these big tech companies.

How is the work/life balance at Facebook?

> * With only very few exceptions, every engineer has access to the near-entirety of source code at FB (Oculus, Whatsapp, Instagram...).

What are the exceptions?

I think the author meant that the exceptions were Oculus, WhatsApp and Instagram. This would make sense given that they are acquisitions and appear to have separate offices, deployments, datacenters, etc in some cases.

Data point: I read it the other way, that all employees could look at all of those things, and that the exceptions were un-named.

I'd imagine the exceptions are some small black boxes that try to catch abuse/spam/etc in various ways & places, or are closed off to prevent a code leak from exploiting a loophole in said software. Defense in depth, and all...

Yeah, that's what I meant. In general, Whatsapp/Oculus/Instagram source code is accessible (at least read access).

For the most part, the exceptions are acquisitions (but those get integrated eventually, see above) or some third-party IP subject to obligations.

FB's culture is ... immature. That's the best way I can put it.