Cloud / VPS Apache Performance Comparison | Hacker News Reader

Cloud / VPS Apache Performance Comparison

chadkeck.com

14 pointsnicklongo15 comments

Full disclosure: Judging by this man's twitter (@ckeck), he seems to be employed by Rackspace. That may or may not have something to do with the tests he ran and the settings he chose. (There are an awful lot of "Test failed" entries under competitor columns, hmm...)

I'm starting to become suspect as well. My own m1.large EC2 apachebench (-n 100000 -c 128) yields:

Requests per second: 5162.92 [#/sec] (mean)

Time per request: 24.792 [ms] (mean)

Time per request: 0.194 [ms] (mean, across all concurrent requests)

100% 60 (longest request)

I think he's comparing an EC2 small (we can tell because he says the memory is between 1GB and 2GB); it looks like the headings on the table are wrong.

I think he's comparing this to a Rackspace 1GB, which is actually cheaper than EC2's small instance; the more direct comparison is to the Rackspace 2GB.

You m1 large is $0.34 per hour, vs $0.06 per hour on the Rackspace machine in the blog (I believe). You only got twice as many request per second out of it, so on the requests per second per dollar metric (the real metric, I think), ec2 is lagging.

Yes, this was a typo on my part and has been corrected. In the "Medium Instances" section it was an Amazon EC2 Small instance that was used which is in line with the 1024MB Rackspace Cloud Server and the Linode 1080 server instance.

With microbenches sucking in general, this test is especially CPU bound. I ran it on a c1.medium instance ($0.17/hr) with -c 160 -n 100000 and got 8602.92 [#/sec] (mean). That's more than 3x the performance at <3x the cost. EC2 gives you all that CPU continuously, not just in bursts.

I make no attempt to mask that I am a Rackspace employee and I even updated the post with a large disclaimer at the top just in case.

The fact of the matter is that the small instances from Linode and Media Temple could not perform the tests as conducted with their out of box configurations. I made no enhancements to any platform. Some helpful readers have pointed out some settings I can change that will allow the benchmarks to complete successfully so I am going to do a follow-up post with all of these "tweaks" implemented across all the providers. I am being as objective as possible here, don't let the fact that I work for Rackspace fog the air.

These tests are also very easy to duplicate on your own if you really want to test the data set and results.

-Chad

don't let the fact that I work for Rackspace fog the air.

If you don't want us to recognize this as the cheap PR-piece it is then perhaps next time refrain from conclusions like the following (verbatim quote):

The tests show some significant performance benefits when running on the Rackspace Cloud Servers platform most likely due to (in my opinion) a more robust network infrastructure, lower hardware utilization, and better host server hardware.

And from unverified claims like (quote):

Even if different web server software is used, it is unlikely that the results and trends would differ.

Otherwise, if that truly was supposed to be a honest benchmark then please hand in your engineer-badge at the door...

I'm about done with people comparing benchmarks on burstable CPU environments like Rackspace Cloud or Slicehost with a fixed CPU environment like EC2. Over the ridiculously short period of your benchmark, the burstable environment will almost always be faster. Over the long term, not so much. Please compare apples to apples.

Didn't EC2 sell the cloud on the idea of being able to burst? Isn't it better to be able to do that for short periods without having to fire up a new server, configure it, and pay for a whole hour?

Your point is very valid from the point of view of benchmarking, but in the real world, I see bursting as a massive advantage.

It feels like we need a better benchmark suite; like I/O test suites that can measure the burst rate and the sustained rate.

EC2 scales across instances, it does not burst CPU resources in a single instance. You can't depend on the "burstable CPU" to be available when you need it. It's a highly unreliable form of compute time. You just have to pray and hope no one else is using the same resources on the same physical machine, or you're SOL.

Right, I agree that scaling out is great. However, bursting is also a great feature; most loads are bursty (you're caching data and have a cache miss or timeout, you request an admin page which does a lot of computation.) All time-sharing involves statistical reasoning, even within your own application: all requests might be served from cache, or ten people might issue different cache-miss requests near-simultaneously. You just reason that under normal circumstances e.g. 80% of your requests will be served from cache, and that determines your typical load. Short-term bursting means that maybe you can size your machine closer to the mean load, rather than the 2 or 3 sigma load.

So, I want _both_ bursting and scale-out :-)

The benchmark is fair for measuring CPU bursting performance, but we do need a more comprehensive benchmark suite.

Short-term bursting means that maybe you can size your machine closer to the mean load, rather than the 2 or 3 sigma load.

No, it doesn't mean that. Because you can't rely on the resources to be available when you need them. The potential headroom may catch your first two spikes - and then crumble on the third.

Would you buy a second-hand airbag for your car that only "usually works" but is a bit cheaper? Thought so.

Saying "I didn't want to customize the environment" is disingenuous -- it means you're just benchmarking the default configurations, not the performance of the host itself. Any benchmark with that many "test failed" results isn't very valid, or helpful.

I think the idea is to get away from the messy situation you see in TPC database benchmarks, where the environments are highly tweaked by the vendor just for that TPC test, and this bears no relation to the performance you actually get as a real world customer.

I think he's re-running the tests anyway where the vendor suggested a tweak; I would take a more hard-line approach (if it isn't what you ship, it's not what I'll test). If the vendor's customer support is responsive and suggests a reasonable tweak, I can see the need for a more nuanced approach!

Correct, I did not want to have any influence over the results of the "out of box" testing as many users on these platforms won't even know how to tweak the OS and Apache on a low level.

But as mentioned, I will be making some of the suggested tweaks to ALL of the platforms and rerunning the tests and posting a follow-up on that as well. I will make sure everything is as uniform as possible.

If there is anything specific I should be aware of please let me know, thanks!

-Chad

Reply on news.ycombinator.com