Rackspace vs. EC2: Performance Analysis | Hacker News Reader

Rackspace vs. EC2: Performance Analysis

thebitsource.com

53 pointstimf18 comments

I was wondering why the text read a great deal like a VNR (video new release).

The analysis may be "independent", but the fine print at the bottom of the article indicates that it was paid for by Rackspace.

This, compared with the way the "conclusions" are worded makes me put this in the "astroturf" category.

Fine print: "The Bitsource conducted an independent, third-party performance analysis of Rackspace CloudServers and Amazon EC2 on behalf of Rackspace Inc. The results were not influenced in any way by Rackspace Inc. as presented using the methods in this analysis."

The results were not influenced in any way by Rackspace Inc.

Yea. Right.

Rackspace just loves to pay for random pseudo-benchmarks. It's their way to give something back to the community. Or something.

Strangely enough, I knew what the outcome would be before clicking the link.

Compiling the linux kernel seems like a poor choice of test, since it is not a pure CPU activity and will perform a significant amount of I/O.

In addition, it doesn't seem like compiling the linux kernel requires very much RAM, so you get the most bang for your buck using a CS VM with only 256MB and the cost only goes up from there as you add more RAM that the test cannot benefit from.

(Edit: I would love whoever downmodded me to explain why I am wrong.)

its a decent enough test, mostly because most geeks out there have done it themselves, so you can get a good 'feel' for a machine by knowing how long it takes to compile the kernel.

its not perfect, and yeah, its kinda a geeky-benchmark, but it doesn't make it an invalid benchmark, as long as it was conducted correctly.

I consider myself a geek (and most of my friends would agree) but it's been 15 years since I last compiled the Linux kernel :)

That said, I wouldn't rule it out as a test because of that, but I do think that if you are going to compare 4 different RAM sizes of CS instances, and compare the costs of doing a job using those different sizes then you need to pick a job that is going to make use of the RAM. You can see from the results that there is a small (time) benefit moving from 256 to 512MB (but not enough to offset the increased costs) and virtually none above that.

Crank curmudgeon review: Loses big points for including illegibly down-filtered graphs in the page. Either make them readable or make them links. Using fancy animations isn't an excuse for including 600x266 pixel unreadable blotches in your document.

All the graphs have links to full size versions and the size in the article itself is dictated by the site (since the side bar is clearly mandatory) So I don't quite no what your problem is.

I can't read them. What's the point to putting them there if I can't read them? Just include a link saying "Graph...". Why format a graph to be 1200 pixels wide, with small fonts even at that size, and then shrink it by a factor of two?

I mean, the inline images might as well be pictures of puppies for all the value they provide to the page.

Edit: I should add that if the size of the images is dictated by the site, then some attention might have been provided to including inline images designed to be used at that size. It's not like it's impossible to display this information in an 600-pixel-wide format...

It seems from the article that they only tested local disks on EC2, not EBS. It would be nice to get a comparison with EBS included.

Agreed. EBS is really where the rubber meets the road in scaling web apps for EC2--esp. a de-normalized datastore.

Amen.

Rackspace knows that very well, wonder why they didn't tell the author to use EBS for the comparison. Oh wait, I might have an idea...

Anyways, as usual, don't trust any benchmark that you haven't faked yourself. This one, just like all other rackspace/ec2 comparisons that you see cropping up these days, was paid for by Rackspace and obviously tweaked to produce their desired outcome.

Ironically they even point out the flaw in their setup by showing how the CPUs on the EC2 instances are not fully utilized.

Guess what that means? It means the test was I/O bound.

I'd be curious to see the test repeated, but this time with a RAID10 over a couple EBS volumes. Not sure Rackspace will pay for that one, though...

In my own informal experience, running identical CPU-bound tasks on Rackspace and EC2, it's just a lot easier to get good performance on Rackspace.

Instance size on Rackspace does affect performance (as their FAQ explains), but it affects minimum guaranteed performance; there's bursting up to the capacity of the entire machine you have a slice of. I know there's something unsettling about relying on excess capacity, but in practice any Rackspace instance will tend to beat any but the most expensive EC2 instance in terms of raw CPU.

The sweet spot of EC2 really is offline processing where hundreds or thousands of machines are spun up by software. If you want to manually fire up a small but flexible number of fast Linux machines, I would recommend Rackspace.

Who makes a living compiling a Linux kernel? Perhaps web comparison benchmarks weren't selected because they weren't favorable to Rackspace?

Do they feel their cloud offerings are inferior to Amazon's?

This is probably a little better comparison of CPU performance between the difference platforms and a couple others: http://journal.uggedal.com/vps-performance-comparison

It's nice to see an analysis that looks at more than just CPU. Disk I/O can be very important in some cases.

It's also nice to see the tests being done over a period of time, rather than a single day.

Reply on news.ycombinator.com