Hacker News Reader

Comment by swombat | Hacker News Reader

1000 new servers a week is a ridiculous number for this sort of company. Their code must be really, really bad. Anyone who works there care to chime in? Is it all written in ColdFusion or something?

View on HN

Downvoted for not thinking through the numbers.

When Zynga launches a new game, they can get over 1M new DAILY players on that game within the first week. [1]

For their games, it seems that 7% of daily players are online at any given time. [2]

So within the first week of launch, you have, about 100,000 people playing, simultaneously.

Let's say the client is sending 5 requests per second to the game server. And let's say that this week, all 1000 servers went to the new game.

Now, of course that's just an AVERAGE across the day. In reality, there will be a cresting factor, like any distributed system, and sometimes the load will spike, say, 8x.

Is it ridiculous for a single server to process 500 requests per second on average and 4000 requests per second peak? That seems pretty standard by my book.

[1] http://techcrunch.com/2010/06/14/pincus-frontierville/ [2] http://techcrunch.com/2010/08/17/zynga-launches-first-locali...

An average of 5 requests per second from a casual game's client is a sign of a poor design. Realistically you very close to 1 user action = 1 request. Now, a top tier starcraft micromanager may average pull 300 actions per (ed: minute), but an average user should be somewhere around 1 action per second to 6 actions a minute.

PS: The key to pulling this off is running the same simulation on the client using the same random seed. User clicks attack blob with fork. > Client says: attacked blob at (timestamp, with fork) > Server says: blob took X damage at (timestamp) random seed at (value) > Client shows: A long string of actions that adds up to that value and or requests a new game state if something does not add up.

> 300 actions per second

You mean per minute, right? Either that or I'm way worse at SCII than I thought.

Ops, fixed.

You're right. I had the Starcraft 300 action/minute metric in my head, and I've definitely seen Farmville players at that level, but the average is probably much closer to 60 actions/minute.

My original point was less about the actual numbers, and more about the exercise of estimating server load, instead of just blindly declaring that they must have crappy code.

100'000 people playing simultaneously adds up to say about 10'000 requests per second. Farmville-like games are not exactly rocket science - most of those requests should be processed sub-50ms total cpu-actual-usage time. that's 200 requests/s per server. Which adds up to 50 servers. Let's double that to account for db load too, and we're still an order of magnitude away from 1000 servers.

So each player generates a request every 10 seconds? Sorry, but that's way off. Every time they click, at least, generates a request. And then there's the famous Zynga tracking. There's no way it averages to less than a request a second.

And you really think those 100'000 are actively clicking around all the time? There are no doubt spurts of activity where they do one, even two clicks per second (most of which should not result in backend queries... surely Zynga has already implemented some kind of queuing mechanism to send those usage data in batches, rather than one by one). Most of the time, though, the users are most likely sitting there watching their crops grow - or even have another window open so they can pretend to work.

Note that it's not 1000 servers every week, it's up to 1000 servers in a given week. This was likely done to support the launch of a new game, and may factor in server turnover (they're on EC2).

That being said, launching 1000 nodes for non-batch jobs is still pretty remarkable.

Zynga can have absolutely crazy growth on launch. FarmVille was growing few million users a week.

Also keep in mind interactive multiplayer games with persistent state are different beasts compared to usual web applications. Games are very write heavy and do not cache well.

"as many as 1,000 servers every week"

Really this probably means that one week they added 1,000 but the actual average number is much lower.

Makes for a good talking point worded that way!

Exactly. 1000 servers was probably a peak, probably a week when launching a new title - Farmville 3.0 - Return of the Wheat or whatever.

Is it all written in ColdFusion or something?

http://gigaom.com/2010/06/08/how-zynga-survived-farmville/

Zynga uses Apache PHP on the front end, memcached for active user play and MySQL on the back end. It uses memcached to store key value pairs to deal with active user play during sessions and then later writes it to disk.

I'm obviously asking the wrong person, but if they use memcached as a temporary datastore until another process can come along later and write the data to disk, what do they do if a memcache node fails?

Or would a node fail rare enough for the data loss to not matter much?

I imagine they have other memcached nodes for failover. Have no idea how it's all managed though.

Zynga has acquired a bunch of different game studios. While I'm sure they've tried to retrofit the acquired games to better architectures, they probably weren't designed with scalability or efficiency in mind from the beginning.

At this scale the performance advantages of a language like Scala start to seem pretty significant.

Reply on news.ycombinator.com