Hacker News Reader

Comment by fatbird | Hacker News Reader

The question is why did they assume 10k concurrent users, or plan for that to be the cap handled under the current architecture? Any competent project manager should have said "let's pretend we're massively successful... how will it work then?" and have some contingency plan in place so that they're not trying "experimental" fixes.

The flaw here isn't the heroic efforts, it's that heroic efforts were needed. In the days of cloud based architectures, it's a lot easier to be ready for massive horizontal scaling than it used to be, so much so that being unready is malpractice.

View on HN

Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of money (and more importantly time) designing for load they never needed to handle before release.

In about a month my company will be releasing an online game. We think we will get about 15-20k concurrent users in the first few days, but we will be testing the realm for as high as 50k concurrent users using load testing bots.

What will happen if we suddenly get 100k or 200k concurrent users? Even throwing more hardware at the problem might not solve it immediately if there is some new bottleneck we haven't found before.

Testing the realm for more than 50k concurrents and then spending engineering effort required to make it work will consume time and effort that we can't afford to spend. All for an outcome that is unlikely to occur.

As long as you've put in enough consideration to avoid painting yourself into a corner with your system architecture so that you know how you'd scale the system should the need arise, you don't need to invest the engineering time up-front to actually build it. However, if you don't understand how your application is going to scale then you might be totally screwed by bad architectural decisions.

For example, you might assume that the data for all concurrent users fits within RAM on a single server. It's one thing to plan for an 8GB server for 50K concurrent load and then have to scale the RAM to 16GB, 32GB, 64GB, 128GB, and so on. But if your RAM usage is O(n^2) for n concurrent users and both your concurrent user count and the average data stored per user increases such that you need hundreds of terabytes of RAM on a single server in order to continue to function, maybe putting in a modicum of up-front work would have paid off.

I'm not in any way saying that you shouldn't plan for the future. We certainly have designed everything wherever possible not to get in the way of future scaling. It's just the problems you can't predict are what really get you.

For example, in theory the first "designed in" limitation that our architecture has should kick in around 1.5 million concurrents. There is a service in the backend that, due to the way it is designed, could never possibly handle more concurrents than that without redesigning some pretty core internals.

In reality though, when we get to 2 orders of magnitude less concurrents then that the realm starts to develop issues that we are currently resolving.

So yes, don't design things that can't scale, but don't expect your designs to actually scale without serious engineering effort.

Assuming massive success is stupid, but testing for it seems prudent. Given how easy it is to spin up EC2 instances generate bot-load, even just to see what'll happen or to find the collapse point, seems so straightforward as to be routine. And as the commenter below observes, the Curiosity interaction model requires high server load for all interaction.

The problem with your theory is that there are lies, damned lies, and load tests.

Any sensible app is going to have caching layers. A simple load test is going to just be a really good exercise of the caching layers. By contrast a complex load test with lots of calls to do really random stuff is going to stress you out in entirely unreasonable ways. And finding a realistic balance of traffic is difficult unless you've simply sampled real traffic - which you can't do until after you've generated some.

This!

I cant agree with this more, unless you approach me with an experiment that would put most science to shame I do not value your use of Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses in it, and you are not engineering a nuclear reactor I probably wont even start the conversation.

.... At $DAYJOB, when we were putting together a new search engine we spent some time upfront putting in a lot of tooling to worry about and test performance every step of the way. Some of this tooling was built around jmeter and focused on generating realistic test plans (which we could derive from the already operational existing search engine). It was a neat infrastructure it tracked everything by commit, captured all manner of metrics and used huge datasets, mined from the logs, built out with markov models. Everything we reported into an install of the software that runs http://speed.pypy.org, we thought we had it conquered.

Yet it was so flawed! Data would get stale, models would break, caches invalidated datasets and much much more.

It was funny at the same time I wrote a tool that we now know as darklaunch, this tool is stupid simple, it captures incoming traffic to the production system with pcap, and then (after a little mussing with it) sends it through to a always on test cluster. Yet its this tool that produces all the load, has all the weird user behavior, never gets out of sync and can run load, soak, endurance and impact tests.

Guess which one we use to test out new software, or new features, or to tune a garbage collector in as close to production as possible ?

I have not started Jmeter now in over a year, its been no great loss; sure we have microbenchmarks where they matter but for everything else you often hear the words "darklaunch it"

Now for sure we have the luxury of having real traffic to use and abuse, but if you don't have real traffic then realise you are using a scrying device anyhow and I will bet you good money your assumptions are wrong.

I totally agree. There's nothing like reality and customers doing absurd things to break stuff you'd never imagined that could break.

We will throw bots at the servers until they fall over, but at any point after 50k concurrents we will not devote engineering time to fixing it.

Test for the big numbers. With server virtualization you don't have to buy the hardware ahead of time so there's no reason not to do so. Even if it doesn't work at more than 50k currents, you may as well find out whre the failure points are ahead of time rather than when your getting squeezed up against them.

Because people make mistakes? Aren't people allowed to make mistakes anymore?

Most software developers love solving problems. When presented with a tough problem they don't mind skipping out on a few dinners at home or not watching the latest sports ball game. It is just being a developer, not some kind of malpractice, or slave driving, or not having read enough mythical books about herding cats.

Shit goes wrong sometimes, it is in some peoples nature to want to fix the problems.

10,000 concurrent users within a day of release is massive success in today's application market. Engineering a system to handle an order of magnitude more traffic than you expect is sensible; engineering it to handle three orders of magnitude more than you expect would, virtually always, be ridiculously wasteful.

Also, Curiosity is not your average horizontally scalable web app, because all the users are continuously interacting with the same data set. Scaling this up to 2 million concurrent users would be a challenge even with time on your hands.

Very off topic, but what are your thoughts on this;

I built something alone for both the client (iOS with Cocos2d) and server (Ruby, MongoDb, Redis) that managed 3k users on release for a full day. The app was only for a day (Event app for an ad agency)

This took three weeks to develop. The client had 3d graphics and a custom map coded and drawn from scratch.

As you observe, Curiosity's interaction model implies continuous server load. 10K users actually seems low to me, given that.

Reply on news.ycombinator.com