Comparing Cloud Compute Services
blog.cloudharmony.com
blog.cloudharmony.com
1) Comparing between same number of cores. The core count selected for each testing level is completely arbitrary. With both web and database servers, which scale well to increasing core count, single-threaded performance is generally less of a concern and should not be a point of measure aside from average page load time. Some server configurations are optimized for higher numbers of slower cores, while others are optimized for fewer but faster cores. By comparing like core counts, this testing is highly skewed to the latter.
Comparing packages at the same price point, or just how the package fits into the product offering lineup (smallest, median, largest instance) would be much more fair to compare. If 4 cores at one provider costs the same as 1 core at another, it should be fair to compare the two at different core counts.
2) Server configurations. For both web and database servers, the best performance optimization that can be done is to cache to RAM as much as possible. With increased caching, the need for disk I/O also goes down significantly, and can easily be by as much as an order of magnitude. Serving static content uses minimal resources and is mostly dependent on network performance. Dynamic content is more CPU intensive, and most of the time you can and should be caching the compiled opcode/bytecode. Most website database usage is read heavy, and many of the queries can be cached as well. The one drawback to a heavy emphasis on caching is that if the server restarts, there may not be enough resources to service all requests while warming up the cache. However, given that dynamic loads is precisely what cloud offerings are supposed to excel at, you can spin up additional instances at these times, or just take a horizontally scaled approach to begin with so that a single instance failing will not have a major impact on your aggregate load.
3) Synthetic benchmarks, by their very nature, do a poor job of emulating real world performance. The best way to benchmark both web server and database is to take a real site, log all the requests, and replay the logs. What you want to measure for is the maximum number of requests or queries that can be served, average time and standard deviation at different requests/query rates, etc.
4) Network speed tests. The biggest mistake that most tests make is that they measure performance from content network to content network, rather than from content network to eyeball network. Especially with the current peering issues going on between carriers and eyeballs, this is more important than ever. This is a very difficult problem to solve however, as it's not easy to do throughput tests from a large number of different eyeball networks. You would have to take a very large number of client generated results, and compare differences for all the different providers in all their different locations, which would be nearly impossible. The next best thing, while still a lot of work but more feasible, is to collect up IP's for eyeball networks for as many different locations as possible, but perhaps just the top X number of cities by population, and run continuous pings/traceroutes over an extended period of time. You can then just use average latency, standard deviation, and packet loss % as the metrics.
1a) Core selections were not arbitrary - they covered most compute instance sizes offered by each service: 2, 4, 8 and 16 cores
1b) Testing was not skewed by faster cores. CPU performance analysis was based on SPEC CPU 2006 using base rate runtime. SPEC CPU is a multi-core benchmark with metrics that scale linearly on number of CPU cores.
1c) The value analysis (based on CPU performance and price) highlights differences in pricing between services. I believe matching CPU cores and deriving value is preferable to comparing compute instances based on price.
2) The intent of the post was to provide and compare multiple relevant performance characteristics for these types of workloads: CPU, memory, (non-cached) storage IO, network. Each of these characteristics is analyzed separately using relevant benchmarks and runtime settings. If your workload relies more heavily on cached IO, then more emphasis could be placed on the memory performance analysis.
3) There are hundreds of cloud services. High level analysis like that provided in this post can at least get users pointed in the right direction.
4) Web server network performance analysis is based on "eyeball" or last mile testing from our cloud speedtest http://cloudharmony.com/speedtest. As stated in the post, this is the most complex performance characteristic to measure. Internal network performance is also included in the DB server analysis.
I wasn't suggesting that the testing was skewed by faster cores, but that your criteria favour hardware configurations optimized for fewer, faster cores because of your comparisons between packages of the same core count.
> 1c) The value analysis (based on CPU performance and price) highlights differences in pricing between services. I believe matching CPU cores and deriving value is preferable to comparing compute instances based on price.
We'll have to agree to disagree on this point. It's been my experience that in real-world situations when comparing options, nobody is going to reject higher core counts when at the same price. Generally, a decision maker looks for the lowest price that meets all requirements, or the best option that still fits within budget. They're not going to simply say that since one provider offers a package with this particular core count, that they'll only compare against other providers' options with the exact same core count regardless of price. That's what I mean by the core selections being arbitrary.
For example, an E5 2643v2 6x3.5GHz CPU costs about the same as an E5 2670v2 10x2.5GHz CPU. The 2670v2 offers approximately 25% more aggregate performance, and is clearly the better option except for the unusual cases where single-threaded performance is more important than aggregate performance. However, given how you choose to compare different packages with the same core counts, infrastructure using the 2643v2 would be favoured in your testing. The decision to compare a single 2643v2 core vs a single 2670v2 core, would be completely arbitrary.
> 2) The intent of the post was to provide and compare multiple relevant performance characteristics for these types of workloads: CPU, memory, (non-cached) storage IO, network. Each of these characteristics is analyzed separately using relevant benchmarks and runtime settings. If your workload relies more heavily on cached IO, then more emphasis could be placed on the memory performance analysis.
My point is, the vast majority of web and database servers either rely heavily on cached IO, or should be. SSD's should not be necessary for web servers; you will be able to serve a higher number of requests with the same budget using a larger amount of RAM instead. Likewise, you should fit your entire database into RAM if you can, and just resort to SSD's when you can't. And this isn't limited to I/O, you can reduce the amount of CPU resources required by caching to RAM also. As such, doing a comparison of packages based on the amount of RAM included is going to be more meaningful than doing a comparison based on the number of cores.
Not just inappropriate, but plain incorrect as well. The charts as they stand imply that the user can pick an offering between "small server" & "medium server" and you'll get a performance along the interpolated line ?! Use scatterplots.
It sounds trivial, but with 20+ charts that I had to stick my face up to the screen for, I gave up after about 5.
Regarding "Due to SPEC rules governing test repeatability (which cannot be guaranteed in virtualized, multi-tenant cloud environments), the results below should be taken as estimates.". I'd like to have seen some attempt to migate this with, say, some kind of averaging of x tests over different instances. Although, I understand this would have extended an already long test process.
"This post is essentially a summary of a report we've published in tandem. The report covers the same topics, but in more detail. You can download the report for free."
If reports like this become regular (say, a monthly occurrence), would it be possible or feasible for the cloud providers to try to game (or optimize for) certain qualities?
I've performed, and also debugged, a lot of these cloud comparison benchmarks, and it's very, very easy to have bogus results due to some misconfiguration.
What is the total file size for the fio runs? (And, what is the intended working set size?) Was fio configured to bypass the file system cache and perform I/O directly to disk? (And if so, what is the rationale for bypassing it?[1]) Was iostat or other tools run during the benchmark to confirm that fio was configured and operating correctly, and that the results could be trusted? Was the same version of fio used, and built with the same compiler (same binary?).
The 118 page report does not include the actual fio commands used.
[1] Disk I/O latency can be a serious issue in cloud environments, and one that vendors can address by incorporating additional levels of storage I/O cache (eg, in the hypervisor). Picking benchmarks that bypass discourages vendors from doing this, which is not ultimately good for our industry.
Excellent work, thanks for doing a quality job on this.
I know what I'm suggesting is not the current norm for cloud evaluations. And I believe the current norm is wrong.
The more important question is how the benchmarks were analyzed -- what other tools were run to confirm that they measured what they were supposed to?
The benchmarks binaries, configurations and runtime settings were generally consistent for instance types of the same size across services, but we didn't verify efficacy of the benchmarks as they ran.