http://malteschwarzkopf.de/research/assets/facebook-stack.pd...
…and it would be doubly interesting to see the "related work" arrows between the different companies' infrastructure platforms. We'd need a 3D visualization for that though.
This is interesting -- I tried to do my best to find out what's still in use and what is deprecated based on public information; happy to amend if anything is incorrect. (If you have publicly accessible written sources saying so, that'd be ideal!)
Note that owing to its origin (as part of my PhD thesis), this chart only mentions systems about which scientific, peer-reviewed papers have been published. That's why Scribe and Presto are missing; I couldn't find any papers about them. For Scribe, the Github repo I found explicitly says that it is no longer being maintained, although maybe it's still used internally.
Re Haystack: I'm surprised it's deprecated -- the f4 paper (2014) heavily implies that it is still used for "hot" blobs.
Re HipHop: ok, I wasn't sure if it was still current, since I had heard somewhere that it's been superseded. Couldn't find anything definite saying so, though. If you have a pointer, that'd be great.
BTW, one reason I posted these on Twitter was the hope to get exactly this kind of fact-checking going, so I'm pleased that feedback is coming :-)
This link has papers on pub/sub, HHVM, and so on: https://research.facebook.com/publications/
Re Haystack: possible I am misremembering, or the project to completely replace Haystack stalled since I left.
If you want to gather a more complete picture of infrastructure at these companies I suggest, well, not imposing the strange limitation of only reading peer-reviewed papers. Almost none of the stuff I worked on ended up in conference proceedings.
I also added Wormhole, which I think is the pub/sub system you're referring to (published in NSDI 2015: https://www.usenix.org/system/files/conference/nsdi15/nsdi15...).
Updated version at original URL: http://malteschwarzkopf.de/research/assets/facebook-stack.pd...
Regarding the focus on academic papers: I agree that this does not reflect the entirety of what goes on inside companies (or indeed how open they are; FB and Google also release lots of open-source software). Certainly, only reading the papers would be a bad idea. However, a peer-reviewed paper (unlike, say, a blog post) is a permanent, citable reference and is part of the scientific literature. This sets a quality bar (enforced through peer review, which deemed the description to be plausible and accurate), and allows the amount of information to remain manageable. The number of other sources of information makes them impractical to write up concisely, and it is hard to say what ought to be included and what should not when going beyond published papers.
I don't think anyone should base their perception of how Google's or Facebook's stack works on these charts and bibliographies -- not least because they will quickly be out of date. However, I personally find them useful as a quick reference to comprehensive, high-quality descriptions of systems that are regularly mentioned in discussions :-)
1. Ask the developers per email.
2. Fly out to SF, visit the campus, have lunch.
Will work for FB (I do it all the time), Google people won't tell you anything.
I misread "This section [...] illustrate[s] how replication and distributed transactions have been layered onto our Bigtable-based implementation." in the paper as meaning that Spanner is partly layered upon BigTable, but what it really means is that the implementation is based upon (as in, inspired by) BigTable.
Spanner actually has its own tablet implementation as part of the spanserver (stored in Colossus) and does not use BigTable. I've amended the diagram to reflect this.
gfs -> hdfs
bigtable -> hbase
google mapreduce -> hadoop
dremel -> presto
protocol buffers -> swift
Haystack still handles the initial writes and the burst of heavy read traffic when an object is fresh. It has a higher space overhead of 3.6x because it's optimized for throughput and latency versus cost savings. The Haystack tier's size is now dwarfed by the size of the F4 and Cold Storage tiers, but it still handles almost all of the reads for user blobs on Facebook due to the age/demand exponential curve.
After a haystack volume is locked and has cooled down enough, it moves to F4 to bump its replication factor way down for cheaper, longer term online storage. And then Cold Storage is used for the older stuff that gets barely any reads but that you still want to store online for perpetuity.
That is why the team that works on media storage is called Everstore; they take storing people's photos and videos seriously and view it as a promise to keep your memories available forever. It feels really good to see photos from 5+ years ago and have them still work, and someday there will be 30 year old photos and videos on Everstore as well.
Source: I built Haystack with a couple other people and founded the Everstore team. :-)