How Instagram Feeds Work: Celery and RabbitMQ

They are overstated the naivety of the "naive" approach.

The article states:

'directly fetch all the photos that the user followed from a single, monolithic data store, sort them by creation time and then only display the latest 10'

That isn't how the query would work. It is implying that the query would return all the results, clearly it would only return 10. I haven't read all the details, I'm sure Instagram does something better then this basic SQL now but it is silly in a technical article to overstate the problem in such an obviously incorrect way.

In short, the feeds are heavily denormalized and are constructed when new photos are added by people you follow, rather than at request time.

They seem to favor using disk space and saving on processing time and memory.

It seems that these problems would be better solved at the database level. It is just that the open source databases lack good implementations of materialized views and data replication to slaves. Instead you have this more fragile solution involving the Postgres, RabbitMQ, and Redis to accomplish nearly the same thing.

This post is great.. but its from April 2013. I wonder how their architecture has changed since then?

Were they only using two RabbitMQ brokers? I have a habit of running one per machine, which allows every app to simply connect to localhost; but on virtualized clusters, RabbitMQ gets a network partition almost every other day, and having just two would obviously cut down on the number of partitions.

If they are using EC2 hosts, I wonder why they don't use AWS SQS/SNS for their queue systems and Dynamo for their materialized views.