Shout Out to the Server Teams
stevestreza.com
stevestreza.com
If your load-testing does not prepare you for the worst then your load-testing plan is garbage.
They already knew how many copies had been pre-ordered and could make a pretty good guess how many copies would be sold and activated on the first night. Take that worst case estimate, now double it and test for that.
"But the cost of supporting worst-case scenarios!!" some may cry. This is where rented servers / cloud setups are useful for elastic scale without breaking the bank.
There are companies that can simulate load from users across the globe, I have no doubt that blizzard would have the connections / influence /cash to set-up a kick-arse load-test system.
I'd think there are some discussion-worthy points here. For instance, I'm not proud of it but at a former employer I talked them out of load testing an app, which then had serious scaling issues. I learned my lesson the hard way.
However in this case what can the users do that can be different to their plan? The most basic example would be logging in, there are not that many execution paths I can think of for that one.
As for the integration / reporting during gameplay, this is all done via a strictly defined API which is called by the game client in relatively predictable ways (game started, player does X, authentication ping every x seconds, achievement unlocked), unlike a web-app where users can do whatever they want the API usage flow is basically controlled by Blizzard.
That is why I am not impressed with blizzard on this one; they control basically the entire use of this API apart from one thing, the number of users trying to use it at any one time, which is there the load testing plan should have worked.
1) Diablo was developed from the fairly unusual position of knowing it would face a launch to millions of users before development ever started. I'd have imagined having this "scale it to infinity" mentality from the starting line would have helped a lot.
2) The whole game is conceptually VERY easy to shard. Unlike WoW, there is very little interaction between players that are not in a party together, and the maximum party size is four.
I wonder whether the failures have anything to do with the achivement tracking/broadcasting. It's the only component I can think of that breaks out of these obvious sharding boundaries, and I can kind of imagine how large friend lists might cause problems. Additionally, it seems achievement progress was lost from the time leading up to one of the downtimes.
I know it's easy to speculate from here, and there are probably very legitimate reasons for all of this. As an outsider, it seems like these particular failures are things that, in general, just happen. Still, I would have expected Blizzard in particular to be better prepared for this. It's a bit surprising.
I'm looking forward to reading (hoping they release) a post-mortem on this.
Distributed services under very heavy load are susceptible to all the same small bugs due to all the normal mistakes every developer makes, except it's much easier for those bugs to cause catastrophic failures!
And yet… I feel like Blizzard could have made an effort to make its single player game run offline. The multiplayer is fantastic, but give us something to fallback on.
It's essentially the Ubisoft Route (Assassins Creed 2 et al) that everybody was incredibly critical of, and they actually removed it by now.
But when golden boy Blizzard pulls shit like that, suddenly everybody is like "Awwww, yeah Diablo!" and I see only few criticisms about the always-online system. (compared to Ubisoft)
One of the reasons I heard for the missing offline single player was "Cheating vs. Real Money Auctionhouse". This seems fallacious to me. Just provide an offline mode that is completely separate from the online part and also separate from the Auctionhouse.
shrug
It's a pretty sad case of putting business concerns over user experience. If it wasn't an anti-piracy thing they would have happily made a offline mode because it would drastically reduce the server load and all the support and development costs that massive multiplayer games have.
If it's not surprising in the slightest, then they should have planned to scale.
Years ago, before the days of the cloud and well-understood fail over mechanisms, a very enterprise-y product happened to share datacenter space with Blizzard. One fine day, Blizzard shipped an update to WoW and from what I hear, it took down networking across the DC and left everyone scrambling.
Try explaining to your customers that your business critical service just went down because Azeroth got a new continent.
Consider: they have access to all of the sales data so they know how many copies of the game could potentially be played at launch.
They ran an open beta so they should have a good idea of how everything scales relative to total simultaneous user count.
They have extensive experience with all aspects of patching, scaling, and server operations through World of Warcraft and Starcraft II.
They intentionally decided to go with a "single player" experience that required connectivity and incurred server load.
Given all of that, there really is no excuse to fail as hard as they did on launch day. It is 2012, the standards are pretty high for getting things right with digital distribution and with online games. More so, if you make a bold decision to force connectivity for a single player game you damned sure better get it right or you are going to destroy your credibility.
Blizzard is enormously lucky that they have a very strong history of compelling games, these sorts of issues could easily cause an upstart game studio to go out of business.
It was a stupid decision and I'm sure there were many angry fans, but if I've learned anything from spending too much time on Reddit for the past couple of years it's that people will forget this pretty quickly.
Take Football Manager 2009 as an example. In order to halt piracy Sports Interactive added an online authenticator for all installs, and to cut a long story short for the best part of three days the authentication server was down due to load, and the only people playing the game happily were the growing number of people who pirated it. After a month or two, it was completely forgotten.
Yes, if a poor developer makes mistakes they'll suffer, but Blizzard is loved by all and all will be forgotten when the fans get their hands on sweet, sweet Diablo action...
That's expensive.
Compared to that servers are cheap.
WoW also had launch issues. Players complained. Money hats were made.
But yes, if anyone realistically thinks these server issues significantly alter sales, they must have also forgotten the SC2, WoW, DiabloII launch.
They have massive amounts of experience in this field, I'm sure they had the capability to make launch day run much smoother, so why didn't it? Perhaps they thought 'unprecedented demand for new game forces it temporarily offline' sounds like a nice headline in the paper.
It's just one day after all, and all the people who play on launch day have already spent their money, and probably aren't really the type to get a refund.
I was at the London launch event Monday night (working not buying), the first person in the queue had been camped there since Saturday lunchtime. He said that by the time he got his copy signed at 11pm he had time to get a train home, get to work with ten minutes to spare, work a twelve hour shift, then go home and spend the night playing the game.
A lot of people care a lot about games like this, and that includes how soon they can play it.
For WoW they were stoned for a while and considered a real name policy in their battle.net system..
Now Diablo comes along. Blizzard isn't special for me anymore and always online games get -100 points by default. For now I guess I'll pass (with SC I don't regret it one bit) and go back to Warcraft III again.
Blizzard of today is to me what the 'new' Star Wars episodes were: Bringing joy to lots of people, but disappointing for me.
However, as far as the Diablo 3 launch goes, some thoughts:
+ Blizzard have been doing this for years
+ They know how many people have pre-ordered and pre-installed the game
+ The game is singleplayer, yet they decided upon this online requirement (no offline play, thanks Blizzard)
All in all it's pretty annoying to purchase something and not be able to play it because they simply haven't upgraded their infrastructure for the load they should have expected.
Is it a common name for devops in other companies?