How Facebook Ships Code
framethink.wordpress.com
framethink.wordpress.com
Bull. Shit. This one line makes me think that this entire post is completely hear-say. A 500 person engineering team that writes bug-free code on a system as a large as facebook? Give me a break.
So yeah, lack of QA => lots of bugs.
The whole system is pretty interesting. They maintain a mapping of every token on the site to every language and only user translators verify portions.
Part of the reason I never made a real FB account was my general feeling in 2005 that securing a MySQL/PHP site against internal abuse takes incredible effort, and they wouldn't ever bother to do it. According to this article, that's likely true. And the "everybody can modify anything anytime" philosophy with little QA explains why, after 6 years, FB is still a gaping maw of security holes.
This is through a third-party, so I'm not shaking the finger at Facebook outright yet, but the wording of that — sharing private user data — is kind of frightening. What about merely accessing private user data? Like you mentioned, I know Google is super-paranoid about even simple data access.
And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go!
Former Facebook dev here. That's all I have to say in response.
I have a hard time believing many aren't abusing this, just out of human nature ("did she mention me to any of her friends?").
Given they have 2000+ employees this is just a process that is looking to be constantly abused. I surely don't expect people aren't talking about it, but I'm nearly as confident that it is happening in a way that most people aren't aware of. No evidence, but again, I've seen people do worse, with less access.
I'm gathering that this will never change.
Otherwise we might get to know each other, and become friends, and I'd never be sure that they weren't looking me up. My strategy for avoiding the prying eyes of facebook engineers is to be personally uninteresting to any facebook engineers.
This might not work so well if I were a chick.
* Read-only all the time. * Access through a special shell which logs your actions remotely. * Seemingly in keeping with the aggressive name & shame culture, if you join across all tables or forget a limit or something else which affects live operation, you get called out.
I'm fairly certain if you give me the schema, I can be very inconspicuous over a given amount of time.
On Reddit, someone mentioned that 'becoming someone', which I presume means making the software think you've logged in as that person, is unusual and must be justified and closely tracked when it happens. So I'm guessing that you can't just pull up records for some person from the database; you need to 'be' that person to get their data. The free access is probably just for stuff that is shared via the privacy controls.
What the hell? why not?
No, we don't. We have test instances of everything. There are a privileged few who have access to sensitive production data, and their access is carefully monitored.
How do you transfer the data? rsync through external network is too slow.
The most convenient way would be to mount physical drive on one of database server and copy things directly. But that would require cage access, which I assume not many employees have.
It would have to be code rarely seen by others, e.g. Apache's mod_rewrite.
"Thousands of random users' data has been downloaded from Facebook and published on Wikileaks" would be enough for a good media scare story.
I think he'll be losing some of his libertarian fans if he publishes information of people who simply want to be more free.
I understand that in some cases redacting names weakens the impact of the leak and its political statement. However, releasing the name of some random informant working on the ground in Iraq has no substantial effect on the impact of the whole leak, but can have an enormous effect on that person's life, especially if he's killed.
The article is has a few inaccuracies:
There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn't look at every change personally.
People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren't around to support them in case something goes wrong (and haven't found someone to cover for you).
Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.
We have automated testing, including "push-blocking" tests which must pass before the release goes out.
We absolutely do not believe "most engineers are capable of writing bug-free code", much less that this is a reasonable notion to base a business upon. If this is a real quote, it comes from a very junior engineer. It is empirically untrue.
The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.
But that still doesn't address wide open access to the live DB by brand-new devs, and the complete disregard for having a dedicated QA team.
Why QA? I would think FB engineers are more busy doing real work at work than fooling around on their own site (heh, the irony) and finding out that Junior Dev 39's change to the thingamabob broke chat for the umpteenth time in IE7. And as many people have pointed out here, they are a platform for many other businesses now--there's no dogfooding that--and the API stinks in terms of stability and documentation.
We also have very small QA team. They work to make sure that the buttons that make us money aren't screwed up.
The key to this sort of uber-agile development is that you have very, very talented engineers, a release process flexible enough to deal with errors, and a management that buys into moving so fast that mistakes will happen.
I have to say that it is simultaneously exhilarating, humbling and a little terrifying to work like this. Luckily, mostly the first two.
Couching my conclusion in the topic of this post, it would seem that Facebook gives extraordinary freedom to developers, but also hangs everything around the developers neck. Screwing up in this area sounds pretty fatal, and not an aspect of a company I would want to work for. I'm kinda old, though.
Couldn't the other corollary of this be that the company/development team has invested heavily in automated testing?
I'm still not sure having the product team responsible for hours and hours of black-box QA testing each week was an effective use of resources.
Scary, even for a developer like myself.
edit: Can I ask why this is being downvoted? Because I am disagreeing with the parent poster? I don't think I am raising poor points or somehow not contributing to the discussion. Facebook seems to not do a lot of "standard" things because they view them as counter to their goals of building their product. I am merely stating that perhaps there is a lesson in there about the value of what some of the convential wisdom in our industry dictates.
There's very little that they can't fix after deployed, and they have no SLA with consumers (advertisers may be different). In essence they have 500 million testers.
This is a bit harder to do with a database or compiler.
>I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and release software.
So I'd take whatever is written here with a grain of salt. My communication with friends working at Facebook yielded similar thoughts but nothing that comes to what's written that implies a callous recklessness. I know for a fact that they have some code-review tools and blocking tests.
Anyway, my point is that the author doesn't seem to be embedded too deeply in the engineering at Facebook and his notes are, while not outright false, definitely misleading.
(I don't speak for my employer, Facebook.)
- Jeff Skilling, former president of Enron
http://www.gladwell.com/2002/2002_07_22_a_talent.htm
Contrast that with the Facebook approach:
"Resourcing for projects is purely voluntary. -a PM lobbies group of engineers, tries to get them excited about their ideas.
-Engineers decide which ones sound interesting to work on.
-Engineer talks to their manager, says “I’d like to work on these 5 things this week.”
-Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
-Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves."
Good god. Does this strike anyone else as disturbing? Surely every single piece of work being taken on should have the USERS needs and concerns as top priority and not sexy stuff that can attract sufficient engineering interest?
What if there's important problems that are really bothering lots of users and a PM can't get anybody interested (or no-one decides to take on the problem?)
Here's a complaint from a guy about maintaining a personal page and fan page:
http://www.stevepavlina.com/blog/2011/01/leaving-facebook/
Some quotes from that piece:
"As a programmer myself, I can’t fathom that it would take much technical and design effort to address these issues, and Facebook is flooded with complaints from users begging them to fix these headaches. From my perspective as a Facebook user with a very active personal page and fan page, I can’t help but get the impression that Facebook deliberately wants to make some basic admin tasks (like blocking spammers) difficult or impossible in order to compel you to spend more time on the site. There doesn’t seem to be any other logical reason for these glaring design flaws that I can comprehend, other than pure incompetence, and based on their success in other areas, it seems more likely that these choices are deliberate."
"Surely someone on their team is aware of all the complaints and requests to fix the broken elements. So why do they seem to ignore what appear to be such glaring (and fixable) problems?"
"I thought that Facebook would be an interesting place to share inspirational messages and build more community around growth-oriented people. But the current implementation of Facebook can’t handle the way I’ve been trying to use it without creating more headaches than it’s worth, and their momentum appears to be headed in the wrong direction for me to expect that these problems would be fixed anytime soon."
"So I’ve crossed the threshold where Facebook’s value isn’t worth the hassle to use it. I concluded that the best choice was to simply drop the service altogether and invest my time elsewhere."
Who will take on these issues?
Again from the Malcom Gladwell article linked above:
"You might expect a C.E.O. to say that if a business unit can't attract customers very easily that's a good sign it's a business the company shouldn't be in. A company's business is supposed to be shaped in the direction that its managers find most profitable. But at Enron the needs of the customers and the shareholders were secondary to the needs of its stars."
Facebook should wake up in my opinion. They did really well to reach 600m users (or whatever the figure is now) but if they want to stay there they should get their priorities straight.
EDIT: Grammar
From what I know of Google, their internal staffing of projects is done much the same way: you're assigned to certain teams when you first join the company, but after that the various Product Owners try to recruit and advertise within the company for engineers to join their team.
When I was interviewing there I recall seeing flyers posted in common areas advertising different projects that were in need of engineers to come join them ("Interested in _____? Google ___ could use your help!").
Could you clarify why this is? I'm not trying to imply anything, certainly not that Facebook is Enron, just that (to me) they have similar attitudes to assigning staff to work duties. When I read the Facebook quote, I immediately thought of Skilling's quote.
It seems to me and many others here on HN that I've read over the last few months have problems with the way Facebook operate e.g. those who need to use their API to integrate with their own service, privacy concerns, spam, etc; and Facebook doesn't seem to address these issues at all. That was my ultimate point - how they address with these issues, and could the fact they are not dealt with in a timely manner actually be a symptom of letting engineers pick their own work?
Let's see how I can put this. What you've just done is the business equivalent of Godwin's law. It would be like comparing US customs to the Nazis because, you know, both want to look at your papers.
What makes Enron Enron isn't how they did staffing, it's how they cooked their books and defrauded people. Their name is synonymous with fraud and corruption, and so invoking them in conversation about similar staffing practices seems... misleading.
If facebook is starting to go down this same road they might arrive at the same end.
But obviously those users are a tiny minority.
I challenge your conclusion that they should get their priority straight: if their goal is to keep growing at current rate as you seem to suggest, then they probably shouldn't change a damn thing, because they are darn good at it.
This goes back to the well-known advice of "make a product you want to use"
Perhaps they just need to sustain this model long enough to get to an IPO.
re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.”
It does explain how facebook ships with bad documentation etc. -- well, now we have enough knowledge to know how to compete with it :)
Kasparov often remarked how a good process is more important than the actual participants. An average human and pretty good computer with a great system won the championship against great computers and against great grandmasters.
Google believes in this, and their products are very well engineered, with full documentation, videos etc. (although admittedly, many haven't taken off). Yahoo definitely understands this. But these are the same companies that are losing to facebook because of social.
If google and yahoo understood the dynamics of social, we would all be better off.
Hasn't everyone read, "Microsoft Secrets" ~ http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-Te...
...because what the author describes is pretty much the MS dev process (cf CH4 Defining products and development processes). Reading JOS, "How to be a program manager" also shows how MS PM's worked in conjunction with developers ~ http://www.joelonsoftware.com/items/2009/03/09.html MS sure knows/knew a thing or two about organising large groups of people to produce software & output product in a corporate setting.
"resourcing for projects is purely voluntary."
As someone who works for an online gambling company where we are not even allowed to use the product we are building (legal/trust issues i guess), it would _rock_ to be able to have this kind of impact on features
As someone who develops on top of the Facebook Platform, I'm not surprised. Huge, obvious bugs that affect many applications are released far too often.
This sounds like Mark Zuckerberg living his 'revenge of the nerds' dream in his Facebook nirvana! It certainly doesn't sound like good management practice!
Why not? I worked for silicon valley startups for 15 years and not once did I meet a "product manager" who did anything worthwhile.
Oh my other comment was just meant in jest. But thanks for downvoting it and killing all the worthless and meaningless HN karma points i had accrued!
I really like this part. Many new staffs prefer to come in and do the "sexy" enhancements instead of the "mundane" support. This leads to their lack of understanding of the system and poor design. I had always believe that new starters should do support work for a while to gain an understanding of the overall system.
Or it is labeled "unsexy" to work on it.