Unfair comparisons | Hacker News Reader

Unfair comparisons

tiamat.tsotech.com

43 pointsbhauer36 comments

Benchmark Nazis will always complain about any benchmark, no matter what. Don't let them dissuade you from continuing work on this amazing project.

Personally, this project has prompted me to change my career path from C# to something else, due to the abysmal performance of ASP.NET. I'm still waiting to see more tests, especially the asp-stripped one, and hopefully some OWIN/Katana in the near future. But for now, I'm seriously considering JS/Node.js or Scala/JVM as my main development platform. So thanks.

Is there any date for the next round?

Thanks, Sker. I really appreciate your kind words. We're not dissuaded at all! I figured I wanted to catalog my thoughts in response to questions about unfair comparisons into a single place. To be clear: I don't slight people for having strong opinions about right and wrong in web development. I just ask them to realize there are varied points of view, including many who have enjoyed looking at the totality of data.

We've been really busy of late, so we still don't have an ETA for Round 7, but believe me that we are very eager to get that completed. One thing we're hoping to include in Round 7 is Windows on physical hardware.

Also, just another aside, some folks might hate the background animation. You can turn it off at the bottom right. Consider it my trolling you if you must. :)

This should be formalized into some kind of internet geekery law.

> X Nazis will complain about any X, no matter what. Don't let them dissuade you from continuing work on X.

I love contrarianism and geekery, just don't get lost in the infinite correctness maze.

I worked on improving the aspnet (on Windows) results and I committed aspnet-stripped (based on @pdonald's aspnet tests) and HttpListener.

As that committer, I have to ask: Why would you change your career path due to supposed 'abysmal performance' reported on a website?

Keep in mind that the 'abysmal performance' of many of the frameworks can still handle millions of requests per day. Plus, many of the 'slower' frameworks have other development benefits like ease of development, community ecosystem, static type checking, etc.

The reason I replied is because I'm considering jumping from aspnet to a 'slower' framework like django and I'm not basing my decisions solely on the results of this collaborative benchmark where I don't even know if experts have tuned the results, and where I don't know why the results are 'poor'.

> I'm considering jumping from aspnet to a 'slower' framework like django and I'm not basing my decisions solely on the results of this collaborative benchmark

I'm not basing my decision solely on these benchmarks either, this is just the straw that broke the camel's back. Microsoft doesn't inspire me too much confidence on the future of .NET and neither does Mono. The .NET open source community has never taken off. And changing my main development language is not such a big deal, I already work with five languages on a regular basis.

I hope to see better results on future tests. I hope OWIN improves the situation for ASP.NET. And I hope future versions of Mono focus more on performance. But that's a lot to hope for and I'm not holding my breath.

You can actually know that experts often haven't tuned the results because many PRs are being ignored ever since the beginning of this project. There is a pretty clear agenda.

Say what? Which of these [1] is old and being ignored?

[1] https://github.com/TechEmpower/FrameworkBenchmarks/pulls

You can write perfectly performant apps in most languages. The problem is often the communities that surround them. Node.js has some nice advantages but I wouldn't compare by performance next to C# and Java. I'd compare on community, language philosophy and typical code quality. Java and C# and the median developer level are awful.

The area that I would be the most critical of the benchmarks is that it's unclear whether frameworks that use process based concurrency are being set up well. With frameworks that use process based concurrency, generally one wants to run as many processes as can fit in RAM with a nice amount of headroom. That way, when one app process is waiting on I/O, another will be handling another request (and the operating system handles the context switching).

Right now, the setup.py for Django specifies that it will create 3 * NCPU worker processes. On the i7, that would probably be 24 processes and possibly using an eighth of the RAM in the box. Similarly, for Rails, the Unicorn configuration specifies 8 worker processes (one process per GB of RAM) regardless of whether it's on EC2 or the i7 (while Django goes from 6 to 24 processes depending on the box it's on since it's 3 * NCPU). For all we know, the CPU usage might be around 25% and the RAM usage even less. As such, it's hard to determine if the speed achieved is because of the limitations in the framework or whether the box is simply being under-utilized due to sub-optimal configuration.

If you're going to do another around, more than any other changes, I think a view of the CPU and RAM usage would really help - if for no other reason than to show that the test is accurate. Right now, I have questions about whether the process concurrency frameworks are well set-up. Why would Django benefit from getting 4x the number of worker processes on the i7, but not Rails? Is 24 processes utilizing the i7 box with Django? CPU and RAM numbers would help provide answers.

To the degree feasible given the breadth of submissions we've received, we have worked to ensure that frameworks that use process-level concurrency are configured to in fact use multiple processes. See question #5 from our questions section [1].

That said, across the board, inclusive of both process-concurrent and thread-concurrent frameworks, we never rule out the possibility of a configuration glitch. We've asked that any subject-matter experts for any framework perceived as underperforming review the code and configuration. To date, we've received a great deal of pull requests to deal with many such matters. On the other hand, very few such tweaks have had a significant impact on performance data.

We do intend to capture CPU, network, and IO utilization in the future. For the time being, we invite fans/SMEs to review the code, run the tests on their own multi-core machine and/or EC2 instances and determine if there is room for improvement.

[1] http://www.techempower.com/benchmarks/#section=motivation

Thanks for compiling the Web Framework Benchmarks! http://www.techempower.com/benchmarks/#section=data-r6

I discovered some interesting and speedy platforms like OpenResty. And PHP performance for multiple queries isn't too bad either - top five.

For round #7, I suggest an addition of the HipHop-PHP (hhvm) Facebook's speedy open source PHP implementation. https://github.com/facebook/hiphop-php/wiki

Agreed! I'd like to see hhvm added. I just created the following issue:

https://github.com/TechEmpower/FrameworkBenchmarks/issues/41...

AFAIK hhvm doesn't have namespace support yet. So Symfony2, laravel, ZF2 can not be run in it. But basic PHP performance would skyrocket I assume.

HHVM is moving fast - Namespace support arrived. I heard reflection compatibility with PHP.net needs some work. They are fixing bugs that prevent top PHP project from working.

Also as per your information I just checked performance of Symfony2 and laravel against rails and django in multiple query section. Though those two PHP frameworks were at the bottom in single query tab, in multiple queries tab the situation just reversed http://www.techempower.com/benchmarks/#section=data-r6&hw=i7... . Interesting indeed.

I think it would be interesting if respective technology vendors could be motivated to tune the configuration, and be bound by some rules. eWeek did a DB shootout like this 10 years ago, and (as the rumor goes) led to the creation of the MySQL query cache to specifically beat the benchmark.

Getting the various vendors involved would require a lot of clout though :(

Here is some commentary on the MySQL configuration: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

* innodb_log_flush_at_trx_commit is not specified, so it's going to default to 1 (durable). This is a little unfair, since other databases won't do this.

* innodb_buffer_pool_size isn't specified, so it's going to default to 128M. A database that uses mmap files will have an atvantage here - since it can automatically use free memory and won't require this level of tuning.

* I thought it was fair to disable query cache though :)

Actually, mostly in an informal matter, we have in fact received many test implementations from maintainers of frameworks directly. If you go through the pull requests, you may recognize some folks. I don't want to name drop though.

I had in passing asked a few if they would be willing to apply an "official" indicator next to their rows. But the response had been uniformly "not interested." Maybe later I'll ask again. :)

Thanks for the notes on the MySQL configuration. There are a lot of gray areas in setting up the configuration, but we do seek recommendations such as these. Aside from concessions we've made to the matter of running a benchmark, the overriding goal for configuration is that "production-class" configuration is preferred.

I'm not an expert at InnoDB tuning, so you might tell me that "innodb_log_flush_at_trx_commit = 1" is not typical for production environments, in which case I'd want to make a change. But if it is typical for production, I'd like to keep it as-is.

If the point is that other databases do not have something comparable, that speaks to the point of the linked blog entry. These things are not directly comparable, feature-by-feature. We would like Postgres and MongoDB to be deployed in a typical "production-class" configuration, but that may or may not be feature-identical with MySQL.

In many production environments where a few lost writes in the event of OS crash aren't entirely the end of the world, innodb_log_flush_at_trx_commit is set to 2. In some it will even be set to 0 (in that instance, data is lost in the event of MySQL crash). If you care deeply about every write definitely hitting disk in a recoverable manner, you set it to 1, but at that point you may have to do other stuff (is the OS lying to you? Is your RAID controller lying to you? Is your disk lying to you?)

> If the point is that other databases do not have something comparable, that speaks to the point of the linked blog entry.

Other databases do have something comparable. It looks like MongoDB actually doesn't, but that's highly unusual. Postgres does (I believe it's on by default), Riak does, Voldemort does (one could make an argument that it's safer to leave it off in the latter two cases than elsewhere).

If I am parsing you right, it sounds like "innodb_log_flush_at_trx_commit = 2" is what you would recommend for a majority of production environments.

Really depends on the scenario, but for most web stuff, I'd tend to recommend a setting of 2, if the data durability caveats were acceptable.

Durability will typically be very expensive to implement unless you have a fast SSD or RAID controller with NVRAM, so it will skew the test considerably if you make MySQL offer it, but say MongoDB does not have to.

But I do recommend it for the majority of people. Shameless plug: http://www.tocker.ca/2013/06/19/deciding-whether-or-not-to-m...

Also: http://dom.as/2013/08/09/on-durability/ (Domas works on MySQL@Facebook).

Thanks!

Just for the record, on our i7 tests, starting with Round 6, we have been using a Samsung 840 Pro SSD for the database server.

Incidentally, on the EBS durability thing, it's addressed here: https://forums.aws.amazon.com/thread.jspa?threadID=27590 - An fsync isn't a guarantee, or wasn't in 2013, but they estimate the risk of loss after fsync to be very low.

Can't find anything on whether their newer SSD-providing instances guarantee durability, though.

That's really cool. Thanks!

It worked wonders with specially "optimized" drivers for 3d mark back in the day.

I think it is the exact opposite - a benchmark is wonderful in showing a product weakness but you should be very careful of http://en.wikipedia.org/wiki/Goodharts_law this.

And old joke back in the day with 3d mark enthusisasts was "I tried this 3d mark thing but after 8 hours still could not find the second level"

A moving background makes it easy to read! Thumbs up!

Indeed! Glad you like it.

Jokes aside, if you hate it, you can turn it off at the bottom right. Maybe some day I'll disable it by default. But for now, I kind of enjoy these responses. Edit: very well, it's disabled by default. :)

I'm sorry but when I can't scroll without horrible lag because someone decided to make a distracting background, it's a serious usability problem.

;) I did not see the "normal background" link on my first visit. I already instapaper'd the article. Let's see if I can add a more serious comment after I read it.

It does a fantastic job of pegging one CPU!

Yeah, I've noticed that Chrome is especially bad since it still does very little to GPU-accelerate SVG. Frankly, one reason I had left it enabled for so long was to make a point about Chrome's bad GPU+SVG support. Why should it be necessary to use the CPU to move a semi-circle?

But in any case, I'd rather not distract from the points I'm trying to make in the blog entry, so the background is now off by default.

Where does it actually run well? It doesn't peg a core on Safari, but scrolling still feels laggy and unpleasant. I initially assumed you'd implemented your own inertial scrolling, as with old Google mobile sites.

It runs well on all my PCs, which admittedly are all Windows. It's just an SVG+SMIL animation. My opinion is that in 2013, it should run entirely on the GPU. But I acknowledge that's not the case as much as I think it should be.

On my box, it uses ~2.1% CPU and ~16% GPU in Aurora.

Ah, yep; it actually works quite acceptably in Firefox. Very unpleasant in Chrome and Safari, though.

Because benchmarks are usually just useless. It is like what has better acceleration on free fall? Race car or truck?

Since we're throwing fruit onto the stand..

I'd love to see Perl Mason on here (the benchmarks)

Make the DB something CPAN like tie::Storable and load it in RAM. Lets see how that stands up XD

Reply on news.ycombinator.com