nbm

I'm a little confused by what you mean by "the review happens before you upload the code"? Can you explain what you thought should happen vs. what does happen?

Thanks!

There is a degree to which scale makes it necessary to develop these sorts of projects. Losing 1% of productivity in a engineering group of 10 people might not be a big thing, but at 1000 people that's 10 full-time people worth of productivity you're losing. Dedicating 1-10 people for a few months or a year (or even two years) to remove that 1% productivity loss is clearly worth it.

Documentation costs productivity to write, but when there are many people who would be made more productive from it, it makes sense to do it.

I think "slack" is the right way to think about it. It isn't a free-for-all - the business still needs to run - but there's enough space to explore, to rewrite, to document, to polish, and so forth. That's where the magic happens - the unplanned and the unexpected things around that which you thought you were going to do.

Different people find different things exciting/sexy to differing degrees. Some people like building stuff that others can see. Some people like building stuff that others will use. Some people like building stuff that makes other people build stuff faster. Some people like building stuff that is highly reliable, really fast, really scalable, or really efficient. Some people even find improving advertising really exciting - they expound on how ads can be win-win, and even that they can only find their full effectiveness when they are, or something like that.

I don't think anyone is "avoiding" any work - there's just tons of work that needs to get done, and people mostly choose to work on that stuff that interests them. One could say that product engineers "avoid" infrastructure work and infrastructure engineers "avoid" product, but I think that would be inaccurate in most cases.

That is a surprisingly hard question to answer.

I think the key is that these projects actually provide value - they make things faster, more reliable, more scalable - whether that's the code's execution or the people writing the code (or debugging issues, or whatever).

They generally aren't solutions seeking problems - they are responses to problems that exist.

Engineers generally don't build things like this on the weekend - unless they like to structure their time like that, I guess. It may or may not be a full-time job, but the job whatever it is isn't some search for abstract perfection, it is again to solve real problems encountered by others in the company. Often it is a part-time component built as part of trying to solve some more direct goal - like fighting spam, or serving bits, or whatever.

Often it is something the engineers just do - it makes sense to break things up into libraries, or services, or whatever, and they do that, and then that library or service is usable/useful elsewhere, and that's it. Other times they may suggest and motivate it as a goal-in-itself in a team goal setting situation.

I doubt that's a particularly useful answer, but maybe with further questions I can make it more useful to you?

As kmavm said, thanks for saying so on behalf of the people who have been working on open sourcing things (I haven't as yet).

I don't think there's been any policy changes. I think the policy has always been that we're open to open sourcing stuff, so long as it is useful to others (ie, not just a code drop that nobody can use) and someone signs up for the work and there isn't something important that's being dropped in the process.

The difference I think is people, energy, and momentum.

We've been able to find people (many already at the company) genuinely interested in some of the less glamorous parts of building a scalable program to open source things - things like sync processes and pull request management and UIs for ACLs and CLAs and so forth.

We've had people who, often due to the availability of these tools, have developed an internal energy to want to put in the extra effort to make their project ready to be open sourced (like making sure all dependencies are available already, or scoping out or stubbing out things that are Facebook-specific (our asset management flow, for example) while still keeping the software useful, and so forth.

The momentum has also made the idea of open sourcing code more top-of-mind to people, which helps to get people to rewrite their changes to accommodate a nascent open sourcing effort on a piece of code, or to get more discretionary time to investigate or work on making something open source. Or even just moral support from your team and colleagues.

More details on the official FB announcement - https://code.facebook.com/posts/360346274145943/introducing-...

The previous smallest unit was a "cluster" - imagine for sake of example that it is the same number of racks as 3 pods. Some time ago, clusters were somewhat arbitrarily limited in size by a few things - human understanding was definitely one, management software and visualization, network layout and port density issues, and so forth. However, each cluster had a bunch of overhead associated with it that outweigh the benefits, including the primary one - failure domain. If we only needed one more pod-worth of servers, we would have to add a cluster with 3 pods worth of racks.

I don't know the actual strategy (I work in a nearby team, but my focus is mostly on load balancing and CDN infrastructure), but one could imagine that in future it may be more normal to augment existing clusters/failure domains (say, add one pod) rather than building whole new ones.

haproxy only got TLS support in 2012, whereas work on proxygen began quite before that. In addition, haproxy does not support SPDY natively (although it can forward it to something that does).

Apache Traffic Server only got SPDY support in a release about 45 days ago according to their release notes.

There are many other relatively basic features besides those, and similar considerations exist for the other projects that existed back then (and even now).

The machines running Proxygen do the TLS termination.

There are some D advocates, users, and enthusiasts (I fall into the last category) at Facebook, and a slowly increasing amount of D code, but the vast majority of infrastructure projects are done in C++, and most people are still choosing it for new projects.

By expensive, not just capital costs, but costs around operating them - they weren't as reliably configurable, health-checkable, and instrumentable as we'd want, and Proxygen (and a later L4 load balancer) were.

Also the previous load balancers had constraints we weren't willing to accept - they required special connectivity to our networks, we could not use particular combinations of options, and we had to rely on vendors to solve problems that most of their customers were not encountering and/or able to detect.

A lot of that data is duplicated to allow for efficient querying or transformation. It often is too slow to process the data as it comes in, so an initial process will write the data in a raw form, and some other process might select a subset of the data to process, and then submit it in an "annotated" form (filling in, say, the AS number of the client IP). Another process will run later in a batched fashion and perhaps annotate the full set of information and summarize it into a bunch of easily-queried tables.

A lot of that data is also not tied to individuals either - for example the access logs for the CDN (which, being on a different domain by design, does not share cookies so is not attached to an account) even reasonably heavily sampled is probably tens of gigabytes a day, and is rolled up into efficient forms for queries in various ways. A lot of it isn't even about requests coming through the web site/API - it may just be internal inter-service request information, or inter-datacenter flow analysis, or per-machine service metrics ("Oh, look, process A on machines B through E went from 2GB resident to 24GB in 30 seconds a few seconds before the problem manifested").

(Not that it makes too much of a difference at this scale, but it is closer to 860M daily actives.)

There is no memcache - there is just TAO, and it talks to the database. TAO is a read/write-through cache, so the only way the data changes in the database is through TAO. TAO contains the indices necessary to answer queries like that efficiently (although there may be other systems for doing similar but more specialized or slightly different queries), as well as a cache of the data.

So, when you add a comment, TAO updates its internal structures with the new comment in the right place (after the DB is updated), and there are no "keys" that need to be updated beyond that.

There isn't one database, although there are a few major types.

The majority of core information (attributes of people and places and pages and so forth, as well as posts and comments) is stored in MySQL and queried through TAO.

Some data is primary stored in things like HBase, such as messages.

Non-primary-storage data (indexes and so forth) exist in various forms optimised for different workloads - so data in either MySQL or HBase might also exist in Hive for data warehouse queries, or in Unicorn for really fast search-style queries.

Other data (such as logs) might reside in one or more of the various data stores, such as Scuba, Hive, HBase, and accessible via Presto, for example.

TAO: https://www.facebook.com/publications/507347362668177/

Unicorn: https://www.facebook.com/publications/219621248185635

Hive: https://www.facebook.com/publications/374595109278618/

Scuba: https://www.facebook.com/publications/148418812023978/

Presto: http://facebook.github.io/presto/

There is a just-before-storage cache (through which the CDN edge requests) which reduces requests per object, but it does not affect the number of hot objects per disk. With Haystack each object is fully available with sane performance in degraded state. This means 1/3 I/O per disk, whereas F4 only has 1 easily available full copy, which means 100% load on one disk per object stored.

Hope that makes sense - let me know if not.

What would you have liked to know about CDN usage?

(I work on the team that builds and runs the Facebook CDN infrastructure.)

The compiler says:

    /d111/f857.d(9): Error: pure function 'f857.foo' cannot call impure function 'f857.modify_global'

Facebook has several formal and informal training opportunities available for engineers (and all employees, really).

There's Bootcamp, primarily for New Hires, but which anyone can go to if they like. I teach a class there about how our Traffic management works - load balancers, and so forth.

There's also classes about learning to program on iOS or Android (those are generally a week long), or on Haskell or C++ or D (these are generally a few hours a week for a few weeks), and smaller ad-hoc 1-4 hour classes on things like the data pipeline tools, debugging and performance tools, and so forth.

There are also ad-hoc "Tech Talks" which just discuss general topics without a specific "learning" focus.

And then there are non-technology but career-oriented courses about time management, technical leadership, managing your manager, how to give good feedback, how to have difficult conversations, and so forth.

It is a six-week program, although the last week or so is a bunch of teams pitching themselves to the new hires, and the new hires meeting with the teams they're choosing between.

Strangely, you're describing ageism. Replace older/more experienced with women, and salary demands with pregnancy and family responsibilities, and you're at sexism.

Older people tend to have more job experience. Not even starting the interview process on someone because they have a lot of experience, means not starting the interview process for older people, which is ageism.

Rather make the offer if they earn it, even if it isn't in the range the person might seem to want. It's up to them to decide to accept it or not.

Well, the examples at Facebook are: Engineer, Engineer, Engineer, Engineer.

I get "Staff Engineer" (some seniority indicator, which we don't have at Facebook on purpose), but "Architect" seems more like a change in role and in expectations rather than advancing on your existing path.

Facebook (and probably many other companies) maintain an IC (individual contributor) track that doesn't entail becoming a manager or becoming some sort of architecture astronaut.

There are a few flavours of seniority for ICs - deep domain expertise, deep skill set, high productivity, high flexibility, able to design and communicate and drive large-scale changes, mentorship, and non-technical organisational leadership like running a training programme, improving hiring, and so forth.

Saying "Seriously, ... you're doing something wrong" when talking about someone else's financial situation when you don't know about their situation (single income family, kids, supporting parent(s), paying partner or kids through university, medical situation) is not particularly productive.

Focusing mostly on #3:

Might be worth interviewing or just chatting around at a bunch of different places (big companies, startups) and see whether something grabs your interest. Passion is often a bit contagious.

I've been pretty happy changing teams every year or so at Facebook, so also consider whether your employer has any internal transfer options available.

Focusing on #4:

If you don't think you have people to learn from in your team or sufficiently close to you in your company, you should move - either within the company or without.

If your company doesn't offer an IC-only track, you probably should make a move elsewhere.

#1 - you'll only really find out about comp once you've got an offer (and then negotiated on it), so you'll need to get out there.

#2 - Tell your manager what is eating up your work time on things you don't enjoy. If they can't make a difference on that, look around. And also just opt out of things. Also might be useful to be more clear here on what is eating up your time.

What's great for me working at Facebook for the last three and a half years is that I can get that productivity reinvigoration by switching teams and solving new problems, without having to learn quite as much stuff as I'd need to at another company, and keeping my existing reputation and also my friends and work acquaintances.

I've worked on at least four teams since I joined (and a few smaller projects) - ranging from spam fighting systems, search infrastructure, internal infrastructure management systems, and now on the CDN team.

I can put you in contact with a friend who works with me at Facebook with a wife and (slightly older than toddler) kids if you would like to talk to someone. My contact details are on my user page.

Your recruiter can do the same thing, of course.

My personal advice:

You control how much time you spend at work and doing work at home. I've gone through periods of being very careful of not doing more than ~40 hours a week of work, and also taking advantage of the ability to work from home some of the time. I've also gone through periods of doing much more than that, because I chose to. I am confident I would not be looked upon differently not having done so.

When you choose your team after Bootcamp, make sure you are clear with your manager on how you want to work. (Feel free to reach out to me after you start - my unixname is nbm.)

I'm not sure about how Tumblr might do it, but one can use a combination of ECMP and ipvs (with a consistent hash) to do the lower-layer L4 load balancing. This means that even if one of the L4 load balancers go down and the connections originally going to that L4 load balancer by the switch/router get moved to another one, the consistent hash to the L7 load balancer handling the request means the connections will not be reset (except in some interesting and less-frequent cases).

This two-stage process also allows for good health checking from the much-simpler Linux ipvs L4 load balancer servers to the more-complicated L7 load balancer servers.

This was described in this Velocity 2013 talk - http://velocityconf.com/velocity2013/public/schedule/detail/...

Any chance of getting some transparency of what automated or manual processes have affected a particular story (ie, on the story comment page itself), possibly delayed by some period to prevent bad actors from realising their voting rings or whatever were discovered?

That is a bit simplistic and is missing a bunch of things.

First the numerous C++ and Java backend services that do most of the heavy lifting like producing the feed, doing searches (of various types, both from what people type in, and internal ones), sending and receiving messages, spam and fake account detection, &c. The PHP portion acts as a front-end to these services.

TAO is the graph caching system that has largely replaced memcache at Facebook.

Just on the traffic side, there is the global traffic management system (for dynamic and for static content), different types of load balancers, CDN-specific caching software, image and video transcoding/encoding, &c.

And that's just the stuff involved in the production path - there are all sorts of behind-the-scenes things that do async work of various types - either as soon as possible (video encoding, updating) or on a daily or whatever cadence to update infrequently-changing things.

I don't recall Beast ever having anything but a Page - in fact, I remember that Zuck mentioning in a Q&A that he was using Beast to get a better feeling about what it's like to run a Page.