Facebook's software architecture | Hacker News Reader

Facebook's software architecture

muratbuffalo.blogspot.com

295 pointscolbyaley22 comments

I found this part interesting: "F4 currently stores over 65PB of logical data and saves over 53PB of storage."

It looks like most of the savings here is due to optimization of the replication level of storage, according to the usage patterns of the data.

Also found it interesting that it doesn't talk about CDN usage.

What kind of footprint would a system of this nature require I wonder? Geographically speaking.

You can use 1-3 petabyte per rack as a rough yardstick (before you start counting RAID overhead etc.)

It can be more/less than this depending on your rack configuration and how big drives you use, and how dense you pack them (e.g. "standard" off the shelf servers typically peak at 24 3.5" drives per 4U, while top-loaded specialised storage enclosures reach about 50 x 3.5" drives per 4U; full racks vary between 42U and around 47U etc.)

But you could if you wanted to get this much storage into about a 5m x 5m server room....

Of course you wouldn't, since you'd want multi-datacentre redundancy etc. Often there's also no point using the biggest drives available because you need higher throughput. Or no point packing them as tight as possible because you need more processing power per drive.

What would you have liked to know about CDN usage?

(I work on the team that builds and runs the Facebook CDN infrastructure.)

If the replication factor for blobs is calculated from the 'hotness' of the data (by looking at the age for example), I'm wondering how CDNs come into play.

Increasing the replication factor would allow for faster reads, but there usually wouldn't be a need for that if you offload reads of hot data to a CDN service (could be an internal CDN...).

There is a just-before-storage cache (through which the CDN edge requests) which reduces requests per object, but it does not affect the number of hot objects per disk. With Haystack each object is fully available with sane performance in degraded state. This means 1/3 I/O per disk, whereas F4 only has 1 easily available full copy, which means 100% load on one disk per object stored.

Hope that makes sense - let me know if not.

"An Analysis of Facebook Photo Caching" [1] discusses CDN usage and complements the original Haystack paper [2] [3] well.

[1] http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf

[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...

[3] http://muratbuffalo.blogspot.com/2010/12/finding-needle-in-h...

Excellent article. I like the architecture explanation. I also like the simplicity of the description, makes it really easy to understand the descision making process.

I think it is a very logical solution. Naturally when I think of facebook, its the freshness of the data that is important for me, as a user I like to know whats going on now, as opposed to older timeframes. I think it seems quite logical that it is much more optimal to manage their data this way.

On this topic, where do you guys learn more about architecture of large conglomerates of web services and the latest trends?

I read High Scalability and the occasional company blog. Is there anything else out there that might be even better? Blogs, books, forums, doesn't matter.

Get a job at one of them, cant quite beat that.

I did already. There are multiple issues with that approach: you get only flavor of how to do things, unless you keep jumping teams over and over again for years. It's extremely inefficient way of learning this.

You're also putting the burden on the folks in charge to know what's out there and pass it down to you, instead of actually doing your own research.

Also, sometimes those people don't know what they're doing and they're struggling just as much as you would be in their position, so now you're learning from people who are making it up as they go, which you could have done equally as well by yourself.

I've learned heaps from reading the papers that these companies publish. Google, Facebook, and Amazon all publish well written, readable, papers. If you're looking for somewhere to start, then the Dynamo paper is a classic: http://www.allthingsdistributed.com/files/amazon-dynamo-sosp...

just curious: "learned heaps" --> did it just improve your knowledgebase or it actually helped you practically to solve a problem which you were facing.? I am just curious, as I am a starter to reading these kind of papers. thanks.

A bit of both, some like the Dynamo paper helped me understand Cassandra better, while other like Spanner aren't directly applicable to my day to day work but introduced me to new ideas.

This is a talk about the evolution of Dropbox's architecture from 2012: http://www.youtube.com/watch?v=PE4gwstWhmc. It is incredibly detailed (down to the exact sql schema they use for file metadata etc).

+1 and I'm particularly interested in Amazon's architecture, I couldn't find much info about it (besides High Scalability). Can anyone from Amazon answer questions?

Great post. Thanks this is an awesome blog post, I love it when someone summarizes the good stuff :D! I wish someone would do the same for the AWS cloud papers.

Wow TIL what Tao is and what it can do.

This part is very sexy to me: Facebook's new architecture splits the media into two categories: 1) hot/recently-added media, which is still stored in Haystack, and 2) warm media (still not cold), which is now stored in F4 storage and not in Haystack.

Some AWS related links from that blog: http://muratbuffalo.blogspot.com/2010/11/dynamo-amazons-high...

http://muratbuffalo.blogspot.com/2012/10/building-fault-tole...

http://muratbuffalo.blogspot.com/2013/04/aws-summit-nyc-day-...

http://muratbuffalo.blogspot.com/2013/04/aws-summit-nyc-rest...

http://muratbuffalo.blogspot.com/2013/04/aws-summit-nyc-day-...

Ow my! This is Christmas for me right here! Thanks again, this is awesome!

the title of his blog"Facebook's software architecture" is misleading. actually these papers are talking about data storage and database. not "software architecture." Though, his blog is informative and consistent. Love it!

Digital data and information storage is not a component of a software architecture? Of course it is, and anyone working on a similar project would need to include these layers in their architectural specifications.

One could argue that software architecture is the inner structure of an application, i.e. the libraries, modules, classes, functions and, most importantly, their interfaces. The GOF design patterns are purely a matter of software architecture, for example.

System architecture on the other hand describes the different applications/services that compromise a system, including the ones that you write yourself, plus the ones that you just "use", e.g. database, storage, load balancer, etc.

I would argue that the article mostly talks about system architecture with focus on storage and I guess the parent you replied to would too.

Reply on news.ycombinator.com