Hacker News Reader

Comment by rbranson | Hacker News Reader

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what actually brought the site down completely, hopefully they'll be able to trace it down and fill us in on that.

View on HN

Here's what I do in these situations (I'm an Oracle DBA, but this should apply to most loaded shards):

1) Use connection pooling at the application layer to prevent overloading the DB of any specific shard. This means that if a shard has 16 CPUs, having 16 connections sounds reasonable. Additional connections will not give you more performance. This means you need to queue and throttle requests at the application layer and with some thought you can probably figure out what to do with the waiting users - show partial results? show a nice whale? A "loading please wait" sign?

2) If you didn't do #1 and the DB is getting overloaded, my normal response is to start shooting down connections. Oracle has separate unix process per connection. MySQL has its own way of shooting connections down. Put up a small script that will kill the correct percentage of sessions to prevent overload on shared resources. This will generates lots of errors and will cause a percentage of the users to hate you, but you won't be down.

1) Application connection pooling won't scale. In a scenario like FourSquare, there is likely a 4:1 ratio of app server to DB shard server. Further, connections don't necessarily equal load.

2) This sounds like a great way to create data inconsistencies, unless you've got very tight constraints on your database, which is impossible in a sharded scenario.

I agree though, that ultimately they should have had some way to "fail whale" instead of getting overloaded.

The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.

MongoDB is designed to help ease the pain of scaling, but what you're asking for is magic. If all the sudden a large amount of requests start coming through that overload a shard (think very popular users, like if several very popular users ended up colocated on a shard), how is MongoDB going to anticipate this? Any kind of shard scaling will only work well in scenarios where you have a reasonable increase in load, not a crippling surge.

In addition, it's very, VERY difficult to scale writes against a single object, such as Justin Bieber's profile data, say, if you've got a view counter on it. You can either serialize writes on read like Cassandra does, which has it's own drawbacks (the more writers an object has, the more expensive reads become), or you can have single-master-for-an-object sharding like MongoDB employs and most other production sites (Facebook, Flickr, etc) use.

There was no sudden crippling surge in this case as far as I can tell. There was no mass-updating of a single object either. It failed even though it fits your ideal situation pretty much perfectly.

How is this primarily an issue of overlaod? One particular shard got overlaoded, yes, but the reall issue is how that brought the whole system down.

Reply on news.ycombinator.com