nbm

It's hard to answer this easily in a holistic fashion without getting really abstract. Is there a particular part you're wondering about?

That's not totally accurate - sometimes you're exchanging more than just the item in question, for example, putting yourself in danger.

In other words, if you can sell an item for $100k if you follow a set of laws (a government's, an organisation's, a community's, or your own), but you could sell the item without doing so for $500k, the item is still really only worth $100k (to you), but the (physical/societal/moral) danger involved in selling the item illegally that goes with it is not worth $400k to you.

Or, put another way, the item may be valued at $500k (to some potential buyer), but the legal buyer of the item is offering not to put you in danger in exchange for $400k.

The existing Facebook deployment system supports running deployments within an LXC (and setting up cgroups, &c.) and was written well before docker was available.

Some background:

* http://www.slideshare.net/dotCloud/tupperware-containerized-...

This isn't my area, so I don't know what we've previously disclosed or what agreements we might have in place with vendors about, say, not mentioning them, so I don't feel comfortable disclosing that.

If it was anything like experiences in other cases, my suspicion (as I said, not my area) is that switch vendors have relatively few customers like us (certainly few that discover bugs, change configuration, upgrade firmware nearly as often as we do, or who make use of the particular set of features we do at the same time), and so some things we really would want would not really be in their interest to work on relative to things that would be useful to most of their customers.

At some point it probably became worth trying something new like Wedge/FBOSS (which while technically hard at least can build on our experience building hardware and software for servers) in the hopes of improved turn-around time and getting the features we want further down the line.

I'll try remember to track down someone from the team to give a less vague answer after lunch.

I work in a somewhat related team (Traffic/CDN) at Facebook, and I'm very excited about what this is going to allow us to do in future.

Current switches just don't support the deployment, monitoring, and configuration power we have for servers. While we've done a lot (probably close to the most than can be) to bring them somewhat close to par, Wedge should not only leapfrog to equality, but also use the same infrastructure - and gain whenever the server processes improve.

The opportunities opened by being able to quickly canary some new features (without doing a firmware upgrade before turning on and again after turning it off), have detailed logging and monitoring and reusing our existing tools for correlation and comparison, and to do some things we currently are forced to do on separate machines now are fairly large.

Bryan is still here. He's actually kicking off a Haskell class within Facebook at the moment - https://twitter.com/bos31337/status/475335956556705792

The systems described here isn't directly involved with any PHP code.

Like all Facebook services, they are communicated with over the Thrift RPC system, and may have PHP (or any other language) clients, and may talk to other services using Thrift (or occasionally other protocols), some of which may use PHP.

If you're asking a general question about batching requests in PHP, http://docs.hhvm.com/manual/en/hack.async.php may be informative.

The blog post - https://code.facebook.com/posts/302060973291128/open-sourcin... - describes the initial usage - to assist the Sigma system in answering questions like "Is this content spam?".

Some background on that system, including the FXL component that Haxl replaces, is here: http://research.microsoft.com/en-us/projects/ldg/a10-stein.p...

The release blog post is here - https://code.facebook.com/posts/302060973291128/open-sourcin...

It contains a lot more information about the problem it was originally created to solve, and potential other use cases.

Is there a reason to ask people for their names? To just try it out, I first just ignored it and pressed "Start". When that failed, I typed "a" as my name and tried again, which also failed.

The Reddit front-end is a relatively simple application. It has a page with a list of stories. It has a page with one story and a bunch of comments. It has a page to add comments. It has an endpoint to vote up or down stories, and to vote up or down comments. And maybe a few other less-interesting things, like admin interfaces.

Facebook is surprisingly easy to underestimate even as someone using it a fair bit.

Just try find every single interface in the front-end. For your own profile. Feed is front-and-center, as is timeline. Then look at events, groups, &c.. Look at messaging.

Then look at the interfaces for managing your privacy and permissions. The apps that you're using and information about when last they requested information. Security like login approvals. Then think about the flows involved in reporting content as abusive or inappropriate. For verifying your identity if you've forgotten your password. For adding more security if you log in from a new computer or from a new location.

Then look at pages, including insights and scheduled posts and so forth. Then look at advertising - boosting individual posts, creating campaigns, &c.. Then look at the interfaces for developers. For translators. Interest lists.

The backends for most of this are in C++ and Java. There's a large amount of data processing happening to track hidden things like spam and scam prevention. But the front-end surface area is quite clearly an order of magnitude or maybe two larger than Reddit (at least where it was when this happened).

As much as I prefer not to engage trolls...

Which language would have been a good technical decision in 2002-2003? It needs to be fast enough in terms of iteration. It needs to not require more resources than PHP. It must be easy to onboard people who don't know it onto. It needs to be easy to operate, and not be costly to deploy on the tens, hundreds, and then thousands of servers necessary. (Spending time learning a new technology that others think is cool or which seems cool, while trying to also build a product, has sunk more than a few startups...)

What was bad about the decision to keep the reasonably well-performing and reasonably suited-to-purpose PHP code for front-end code, and peel off suitable tasks into services like the feed, typeahead, messages, and so forth into languages like C++, Java, and so forth. What was bad with the decision to let hundreds of software engineers continue to build the PHP code-base while a much smaller group of people work on projects to improve the efficiency of both the execution environment but also the tooling and developer efficiency on that code-base? Their contribution there is multiplied out by the improved efficiency of those hundreds of developers and the code they wrote.

Seemingly Facebook survived its growth phase fairly well and didn't need a tiger team from outside to handle it - and without the reliability problems others who chose to use languages other than PHP for their front-end and decided to rewrite their much smaller surface area in other languages.

As much as people may dislike PHP (and I'm one of them), it was definitely "good enough". Many languages may not have been, even if they are nicer languages in some objective way.

(Disclaimer: I actually have to write code in Facebook's PHP code-base every once in a while. But most of the code I work with is Python, Java, or C++.)

Facebook hasn't used plain old Apache/PHP in production for several years (HipHop for PHP was announced in Feb 2010), so it is hard to compare.

HHVM is its own web server (although it supports FastCGI for easier use with existing infrastructure like nginx), and that's how we use it in production. It's hard to compare memory usage, except at scale, where it benefits from not having a whole bunch of interpreters (in different processes) running at the same time and some other benefits by using more appropriate types to store values through type inference.

Facebook already has several decoupled services written in other languages. The PHP codebase is mostly the front-end code which, beyond some database work, dispatches heavy lifting to back-end services written in C++, Java, and other languages.

As you say, Thrift connects this all. You can look into our C++ core library, Folly, and other C++, C, Python, and Java applications and libraries like Rocksdb, Presto, watchman, Buck, flint, scribe, and so forth on github.

Pretty much only the front-end code is written in PHP (or Hack, I guess).

Services (things like the feed, the typeahead, graph search, messages, &c.) are written in C++ and other languages - ie, the things it is sensible to write them in...

I believe Python might have worked. It's grown well as a web development language since then by itself. It may have forced several rewrites before WSGI came about. It is a relatively easy language to learn, to hire for, and has few massive warts. (Ruby is roughly in this category too, in my opinion.)

I believe Perl probably would not have worked. It hasn't grown much as a web development language since then (although it has some great web frameworks). The language itself has languished in both development and community growth, and it fairly esoteric and certainly harder to learn than Python and PHP, and would have been harder to hire for. It seems unlikely that pre-2010 Facebook to meaningfully have changed the trajectory of the language (ie, Perl 6 being a success).

I don't think Java, C++ (or C), Cold Fusion, ASP, Haskell, Ocaml, and many other languages would have worked, for a bunch of reasons. Harder to hire for, harder to iterate with, more costly in terms of infrastructure, harder to teach/learn, and so forth.

PHP isn't a totally insignificant part of Facebook being as successful as it has been. A good-enough language choice was necessary, at least, and PHP it seems was a good-enough choice.

Can you rephrase your question about the ops perspective? Is there something in particular you want to know about?

Which mainstream language would you have chosen in 2003 that you think would have had similar success?

Come interview at Facebook for a Production Engineering position[1]. I bet you your interviewers will push you at least that far or I'll give you a free lunch[2]. Even if it's just for the interview experience.

[1] I work there and love it. My contact details on my user page. [2] I'll give you a free lunch either way. No purchase required. Void where prohibited.

Give me a shout when you're in bootcamp (my unixname is 'nbm'), especially if you're interested in working in infra and want help finding out about and choosing between teams.

I'm not sure if Wyatt, Qi, et al. looked at pre-caching at all. I've not looked at the numbers yet, but it's on my list of things to look into once the next few months of more obvious wins are worked through.

I work with the Facebook CDN team and (amongst other things) maintain the data pipelines that log requests, the tailers that fetch/annotate the requests, and populate the Hive tables that we used for this research and other improvements to serving content.

There is certainly a lot of infrastructure that this is built on that many small teams don't immediately have access to in other companies. Whether it's self-service hardware provisioning, Scribe logging infrastructure, tailer frameworks with checkpointing and retries (and job systems to schedule them), and large amounts of available space on Hive for experimentation. But most of the software parts are available as open source, so it doesn't need remain unavailable.

This is my first team at Facebook that I've been heavily involved in this scale of data capture and analysis, but it only took a few days to get up to speed through a combination of great tools and good documentation. Being able to drop a Python file in a code repo to ensure that some complex data warehousing task takes place every day after that is pretty powerful.

Yes, mostly due to new content. You have to miss on the first request for the new content, and each of the 350 million or so photos a day will contribute a miss. Also, this only ran over a particular time period, so one-off requests for older content (someone revisiting their old photo albums) would be misses too.

I'm not sure whether a "refresh" (ie, 304 not modified due to expiration) is counted as a miss in this data.

From http://www.foxnews.com/tech/2014/02/13/male-female-or-custom...

  Agender
  Androgyne
  Androgynous
  Bigender
  Cis
  Cis Female
  Cis Male
  Cis Man
  Cis Woman
  Cisgender
  Cisgender Female
  Cisgender Male
  Cisgender Man
  Cisgender Woman
  Female to Male
  FTM
  Gender Fluid
  Gender Nonconforming
  Gender Questioning
  Gender Variant
  Genderqueer
  Intersex
  Male to Female
  MTF
  Neither
  Neutrois
  Non-binary
  Other
  Pangender
  Trans
  Trans Female
  Trans Male
  Trans Man
  Trans Person
  Trans*Female
  Trans*Male
  Trans*Man
  Trans*Person
  Trans*Woman
  Transexual
  Transexual Female
  Transexual Male
  Transexual Man
  Transexual Person
  Transexual Woman
  Transgender Female
  Transgender Person
  Transmasculine
  Two-spirit

Have you tried doing that recently? Doesn't work on Firefox and Chrome, not for some time.

The other alternatives specifically for logging are Scribe (which we use here at Facebook) and Apache Flume.

As I understand the use case for this software, it would be talking UDP to a local syslog server, not over the network.

At Facebook, we use scribe, and generally use (in-memory, in-app) batched writes to local scribe, and then batched writes (memory+disk, scribe daemon) from that local scribe to an upstream scribe storage system.

First, don't do anything that will likely hurt your own situation. If you have a reasonable belief the HR team or others would share your identity along with the disclosure, and you don't feel confident you can quickly find a new job , don't do it.

First prize is speaking to them yourself. Let them know that you considered it inappropriate and why - that it might be hurtful to others (or that it was hurtful to you), may lead them to not enjoy working at the company, may lead to them having various troubles, &c. Probably no need to take it further if you believe they've taken your feedback on.

Otherwise, assuming you consider it safe, speak to someone in HR at the very least. Ask them what the policies are and what would likely happen, and feel comfortable you can share the identity of the person without them getting instantly fired, if that's bothering you.

If you can and feel comfortable, speak to their manager. That's what their manager is there to discuss with them. They hopefully have a good rapport and this can be brought up in their next discussion.

If you have a good relationship with your manager, talk to them - but realise that they're obligated to inform the HR department about it.

If your company has an "Employee Resource Group" for LGBT employees, speak to someone in it (if you're in a sufficiently "large tech company in the bay area", it will probably have one).

(If you're at Facebook, firstly, I think you should feel comfortable doing all the above without fear of any downside, but I'd be happy to talk to you about it in person or online and connect you with the right people.)

How did you learn and change?

I'm involved in several (20+?) conversations a year where Big-O is used. It's mostly used as a very efficient shorthand for explaining the source of a performance or memory problem (instead of an algorithm being "slow" or "big", it's that it is an algorithm whose performance is acceptable for certain set sizes), and why a particular change is a good fit for the solution.

(Then again, I also am involved in a few constant-time optimisation discussions a year, where cache coherency beats out complexity as useful in an algorithm.)

I don't expect people to know the notation, but I expect that they can explain the underlying values for a particular algorithm.

I'm surprised nobody has brought up that this was originally called GOAWAY in the spec, and that it was only renamed from GOAWAY to GTFO five days ago. It's not like it was always named GTFO or that people are remotely attached to it. In fact, people involved in the process (ie, the working group mailing list) only heard about it after it was renamed.

(I realise that's because this isn't a protocol-related issue, only a documentation one, but my argument is that there's no real impact in the change back.)