Foursquare and MongoDB: What If? | Hacker News Reader

Foursquare and MongoDB: What If?

facility9.com

57 pointsroder35 comments

This is "how I would make Twitter scale" all over again. Unless you're an engineer working on that particular system, you can't credibly give advice on how to improve it. Doing so is an insult to the profession.

What a bull-shit statement. So Twitter engineers can't take any outside advice because those people aren't working on Twitter's particular system. No one else in the world could possibly know how to make large scale messaging systems work.

I am not saying any of the arm-chair analysts are right but it is pretty presumptuous to dismiss them outright.

While luigi's statement is a little extreme, I think the gist is right. Yes outsiders can offer advice on a system, but it's really difficult to offer sound advice without specific details of the system. Perhaps outsider advice shouldn't be dismissed outright, but it ought be taken with a grain of salt.

I very much agree that the devil is in the details, and the largest detail here is how Foursquare goes about calculating badges. So without knowing that the article can't fully evaluate alternatives. That has less to do with the author not being an engineer on Foursquare's team rather Foursquare not detailing the calculation. Maybe on Riak map/reduce is fast enough and Foursquare can shard by checkin, which would be much easier to balance.

Maybe they should just implement their own sharding based on user activity. Maybe a few SSD drives on their Postgres server would solve they issue. They through the issue out there, if nothing else works tell us why.

Some informative links, in case you haven't seen them already:

* http://www.quora.com/What-caused-Foursquares-downtime-on-Oct...

* http://blog.foursquare.com/2010/10/05/so-that-was-a-bummer/

* https://groups.google.com/group/mongodb-user/browse_thread/t...

Someone on this thread suggested that Foursquare's performance problem is related to the calculation of badges. I'm not sure where this idea came from. I haven't seen any mention of the issue being related to badges in first or secondhand sources.

That said, Mongo DB's map/reduce operation would not be a reasonable solution at this time. Mongo DB's map/reduce performance is, at present, somewhat lacking because it runs via the javascript engine which is currently single threaded. I know there are plans to improve performance of Mongo DB's javascript engine by switching to V8, but I don't know if V8 is multithreaded.

Often design decisions that look bad in hindsight get baked in early. By the time you realize that an alternate design would yield better performance, there may be too much data to migrate so you just have to live with it.

I am not sure if badge calculation is the bottleneck. Foursquare did say the calculation is why they have to keep everything in memory though. I just assumed that they sharded by user to keep all of the user's check-ins in the user document and that they are doing calculation with that set of data.

harryh mentioned it here: http://news.ycombinator.com/item?id=1769909 Computing the badges online (ie at each checkin time) requires having the whole db in ram for acceptable performance.

Maybe, but this article isn't really a look at Foursquare's problem; it's looking at these other databases, using Foursquare's problem as a lens. Not so useful for Foursquare, but useful for people who are considering which database to use.

Not so. They built a system that abused their datastore, and switching it out wouldn't solve anything.

Did you not read to the end for the Conclusion paragraph?

This is essentially a non-discussion depending on how they're using Mongo. I suspect that they, as we are, might be making use of Mongo's ability to query documents based on embedded objects, which means their only choices would have been a traditional RDBMS with JOINs or MongoDB.

That's just absolutely false. What makes you think that other database systems can't be made to perform these kinds of queries?

This is based on my own experience at our startup. We looked at a number of NoSQL databases for our backend storage. While others may "be made to", MongoDB was the only one that advertised it as a feature and was by far the easiest to use for that feature.

I'm not questioning the validity of the data point, but why would RDBMS have a cost of 10 times per unit of storage? I haven't heard this reasoning before. Anyone know where this comes from?

Assuming RDBMS is to be understood as ACID, this may be because of the need for more reliable hardware, and good IO hardware is comparatively expensive ?

If you use things like mongodb and co, you are giving up on single-node durability, for example, and you don't need to have as much efficient IO (e.g. reasonable fsync performances). Otherwise, I don't see why mysql would require more expensive hardware mongodb

Interesting writeup. The thing it seems to not address is that a different sharding model might have not worked for foursquare's data access needs. In this http://news.ycombinator.com/item?id=1769982 HN post harryh alludes to the fact that using user ID as the shard key was a design decision. Their current design requires that an entire user's checkin history be able to be loaded for calculating badges so sharding by user ID might have been necessary to guarantee not having the performance overhead of multi-server retrievals. From reading the various write ups it seems like the real problems (using the current load all data for badge calculations method) came from getting too close to the max memory and then the issue that foursquare's checkins weren't filling full pages so migrating data didn't actually give nodes 1 & 2 any place to write new data to (in memory) until after compaction could be performed.

Why not shard by check-in id and map/reduce to get the badge calculation? When sharding by user id, no solution is going to lead to a balanced cluster since some users will always be more active than others. If map/reduce takes to long, batch it and return the badge calculation later.

If the data was sharded by user ids and some ranges of user ids were more active than other user ids, why not shard by a hash of the user id. In that case all the data for a given user id would live in a single shard, but active ranges of users would be broken up across shards. If any single user is creating too much data for a single shard, you've got other problems.

Why should getting someone's badges require any "calculation" at all?

I find this sort of uninformed armchair analysis to be unfulfilling.

This begs the question “How we could handle the constant write load?”

Please stop doing that. http://begthequestion.info/

"While descriptivists and other such laissez-faire linguists are content to allow the misconception to fall into the vernacular, it cannot be denied that logic and philosophy stand to lose an important conceptual label should the meaning of BTQ become diluted to the point that we must constantly distinguish between the traditional usage and the erroneous 'modern' usage. This is why we fight."

That's much more eloquent than I'm typically capable of putting it; thanks.

Language is dynamic, get over it. Unless you want to stick to 'thee' and 'thy' (which clearly is not the first form of English). IMO "Begs the question" == "Raise the question" makes much more sense than the 'correct' interpretation.

I suppose I should of expected that reply.

I'm certainly not saying it to be a douche, it's just how language has worked...forever. Fighting it is pretty useless IMO.

Off-topic: IMHO, this page has a unique and beautiful design (reminds academic papers)

The font is entirely too small, but the layout is nice.

Amusingly, I make that exact statement every time I'm presented with a printed academic paper...

I disagree; it might seem pretty from a distance, but that's about the only good thing about the design.

The font is almost unreadably tiny, the lines are too long, and the links are in simple boldface, rather than underlined and/or in a different color. The headings are also represented by boldface text of the same size, which actually stands out less than the link text. There's very little visual distinction between the body of the article and the navigation junk surrounding it, making for a visually confusing layout.

This is a prime example of why the Readability bookmarklet is great.

Yeah, that page lasted about 2.5 seconds before I Readabilitified it.

It's a template designed by Khoi Vin, that's why its beautiful and unique. www.subtraction.com

"To be honest, Foursquare’s data load would be trivial in any relational database." Pretty insulting statement, imo.

A little off-topic here, but it kind of puzzles me why MongoDB gets so much attention and the other half of FourSquare's stack: Scala/Lift, that scales just fine, and has not given them problems at all gets hardly any attention.

I think you answered your own question...

Foursquare did add more MongoDB servers, the problem was that the 5% of records that were migrated off of the full server were small, so the removal didn't result in many memory pages being released, so it didn't help. This article does not address how any of the alternatives address small records and rebalancing with small records.

Reply on news.ycombinator.com