Web Framework Benchmarks Round 2
techempower.com
techempower.com
Thanks especially to those who have contributed! We hope this is useful information.
You have no idea how valuable this is to everyone! I know it takes a lot of effort to consolidate all comments, requests, fixes suggestions, etc. Personally, I've even seen you respond on the Play! framework Google groups.
Thank you for being such a down to earth person and helping out the community. You guys rock :)
Thanks!
Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC licenses, which I didn't find):
http://zedshaw.com/essays/programmer_stats.html
For instance, you state that:
> In this week's tests, we have added latency tab (available using the rightmost tab at the top of this panel). On i7, we see that several frameworks are able to provide a response in under 10 milliseconds. Only Cake PHP requires more than 100 milliseconds.
Also, I like the "sportsmanlike benchmarking game between different communities" vibe I'm getting from all this.
Would be nice if the community helps turn this into the de facto example of how to benchmark correctly.
Now I'll just have to wait and see how Go 1.1 compares ;).
Thanks for the input and constructive criticism, vanderZwan. It has been very helpful to get feedback from yourself and everyone else. I too am particularly happy with the sportsmanlike competition vibe. You have no idea how fulfilling that is to us.
As pfalls said, we're definitely open to different ways to present the data. More sorting would be cool.
Thanks for the great work!
See php.ini, php-fpm.conf.
I'll add in apc.php to help make sure other things are tuned properly though. different settings could be appropriate for different servers...
What happens now is 256 workers running and 256 simultaneous requests occur. So php-fpm sees 256 workers busy, 0 idle. The minimum idle is 256, so it attempts to start 256 additional processes.
I don't think they have anything against C#, probably no one just has committed C# version to be benched.
(edit: whops, linked fork first)
I'd think that it would probably land somewhere close to Java Servlets, but a bit slower. The framework stack for web requests in .Net is probably a bit more than it is in the servlet server in question. I would also think that Mono would be a bit slower than IIS, only because IIS does very well at pooling resources/threads for multiple requests.
There's also the question of async .Net handling vs. blocking. Most .Net code I've seen is blocking, but there are async options, and as of the 4.x releases are much easier to use.
I'm very surprised to see Play-Scala is not in the same playing field as other JVM based frameworks, had a lot of hope for it, I hope TypeSafe will take that into consideration...
Node.js is the biggest surprise for me on the good side, I think it changes my plans a little as for what to learn next...
Thanks for a very valuable study, this is great
In the first test it sees how fast frameworks can serialize json and in the second test it sees how fast frameworks can access the database AND serialize json.
An inefficient json library and a framework is doomed across all tests. There are a lot of things frameworks do beyond serving json. These benchmarks are presented as representative of broad use when really they are only representative of the frameworks if used as a json api.
A raw HTML hello world, and removing the json encoding step in the db test would go a long way towards getting results that can mean what the authors seem to want them to mean.
There's nothing wrong with a json API benchmark, in fact its quite valuable in a lot of cases. But if that's what this is intended to be, the authors should say so.
My first impression is that this data shows about 3 levels of web frameworks. At the bottom (slowest) we have Django's and Rails and many other introductory app server frameworks. Lets say after I was able to build an initial product successfully, would I then consider re-building the product in a higher performance framework as Go, Node, etc?
The third level, netty and gemini and servlet etc, I am not familiar with. Googling "netty" I get --"Netty is an asynchronous event-driven network application framework"-- I thought that is what Node is (in js) and Go does with gophers.
What are the use cases for these faster frameworks and do they follow an evolution of performance options that an app might go through?
I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone.
I find its good "food for thought" as it provides me exposure to frameworks that perhaps I can use one day but also to learn more about.
The most interesting aspect is all the code for each framework to do the simple page output, the queries, etc. can be viewed for each framework. I find this an incredible learning tool as it shows the personality of each framework/language.
This is exactly right, performance is one of many items to consider when building your own website, and while we're aiming to have a very comprehensive and useful set of performance data, we would agree that other factors need to be taken into account.
Use case example - Twitter was using Rails on the front-end for a while but it got to the point where using it made little sense. Part of Rails' performance is dependent on caching - which is really hard, if not impossible, to do well when you are dealing with live data.
FYI, Gemini is our framework that we've built and maintain internally. You probably won't find reference to it (except for out blog).
edit This is the link to Twitter's blog about Rails http://engineering.twitter.com/2011/04/twitter-search-is-now...
You add new servers and they get crushed. You scrutinize every line of code and fix algorithms and cache whatever is possible and you still can't keep up. You look at your full stack configuration and tune settings. You are afraid of growth because it will bring the site down. THEN you know it's time to consider switching frameworks. And even then I would try to find the core of the problem and rewrite that one piece.
Rewriting an app in a new framework can kill a company. Be leery of starting over. I personally know of one company who started over some 4+ years ago because the old app was too hard to maintain and only started rolling out the new app last year to extremely poor reception (even with about 1/2 the features of the old app). The reception was so bad that they had to stop rolling it out until it was fixed. They could have easily spent a year improving the old app and would be miles ahead.
That's not to say that all problems are due to the framework either. I've had web servers go unresponsive for 20+ minutes because apache went into the swap of death because KeepAlive was set to 15s and MaxClients was set to a value that would exceed available RAM. The quickest solution was to cycle the box. This was 10 years ago though and I think I had a total of 1GB of ram to work with.
You have to trade off developer productivity for performance when it comes to choosing frameworks. The slower the framework the more it does for you.
Note that Gemini is an OSGi framework built on top of Spring DM. For those who don't know, OSGi is Java modularity "done right" - basically a very loosely-coupled module system (which Java lacks) on steroids.
>>You have to trade off developer productivity for performance when it comes to choosing frameworks. The slower the framework the more it does for you.
And in that respect, the more the framework does for a developer, the better it is probably suited a starting point for new developers. It is a badge of success that Django and Rails can be able to serve a whole spectrum of needs.
When it comes to "learning" frameworks, I don't think the highest level of abstraction is necessarily the best. Some are better able to cope with black-box behavior than others. For those who feel the need to understand their tools as they use them, I'm of the view that the Sinatras of the world are a better starting point.
https://github.com/ServiceStack/ServiceStack/wiki/Real-world...
Is GitHub issues the best way to ask for guidance?
- Unnecessary parsing.
- Writing strings instead of buffers (they're copied, they aren't sent as-is).
- Using async. It does a lot of really nasty things, most of which break V8 optimization best practices. This is a perf benchmark, not a comparison of how concise your code can be.
- Having the main request handler in a gigantic function that will never be properly optimized by V8.
- Not getting helper functions that are clearly monomorphic warm before accepting requests.
While the code itself is what I would consider fine, when benchmarking against strongly typed compiled languages, performance concerns become important, even if you have to write ugly code.
On the other hand, shouldn't real code be tested? How accurate would a ugly-optimized-code be when you SHOULD write reusable code in your real life projects?
But ideally, yes. We'd be better informed to see how real world code performs rather than some idyllic perf-oriented creature.
That seems to defeat the purpose of benchmarking to begin with, or at least makes it less useful for real-world comparison.
That goes against the notion of benchmarking.
It's about measuring as well as you can how idiomatic code will perform.
It IS very interesting and is definitely something we have discussed methods for measuring for the sake of these benchmarks. Other areas that we discussed were cpu utilization, disk utilization, and network saturation.
But yes, the performance of a reasonable web app stack on Clojure is very good.
It's just important for people to be aware that "Compojure" actually represents "one possible production-ready stack that uses Compojure as the routing component", as do several other items on the list. I don't mean that as a criticism - just clarification.
Wish there was a way to benchmark those points.
Just use the language and framework you are most efficient in and enjoy using. Worry about framework performance when you notice server load increasing over time.
Matters most? There are 100x differences here. Take two companies writing the same application, one in Spring the other in Rails. Both have easy access to knowledgable people, both are industry standard. However the Spring application would be several orders of magnitude more scalable[1]. I'd consider that to be a more important consideration over enjoying the language.
[1] All benchmarks are suspect until proven otherwise. You can write slow software in Java/C/C++, you can write fast software in Python/Ruby etc etc etc
You can throw money at Rails and it will handle the load, but it would cost a lot more.
Edit: clarity.
98% of the websites out there could run on python and I doubt we'd see a big performance difference.
However, I really don't think there exists a 10x difference developing in Spring. Hell, I don't think there is a 10x difference in using Struts vs Rails.
Spring development is considered slow as it brings images of enterprise shops taking several years to go through the bureaucratic red tape necessary to incorporate a new feature.
I've worked with plenty of small companies using Spring and their development/release cycle is maybe 1/2 or 1/4 the speed of comparable 'easier-to-build' technologies, but it definitely isn't 1/10th the speed to develop.
I worked at a start-up a few years ago. I worked on a PHP-powered website that spoke to a Java-powered (Spring/Hibernate) service layer. The PHP team consisted of one: me. The Java team consisted of over 10 engineers. I outpaced them easily and consistently.
In other projects with similar divisions, I've had similar experiences, though not always so dramatic.
Last few years however I decided to stick around with one of my clients for a while and I realized that quite a few of the employees I considered sub-par before were actually pretty decent programmers doing their best to do quality work in a dysfunctional environment.
I've since then learned to really appreciate this quote, which my gut tells applies to your situation as well:
"Never attribute to incompetence that which is adequately explained by bad management."
While undoubtedly less expressive I think most of the bad rep Spring gets is because it's used in "Enterprise", which means programmers that are, if not of lower level, constantly interrupted and yanked around and told what not to do.
I wouldn't choose Spring anymore because there are better alternatives, but I'm pretty sure the 10x is in the programmer and not the framework. The framework might be 2x, if that, but not much more I think.
P.S. If you include Spring Roo then I'm pretty sure the 10x no longer holds, but I never got up to speed on Roo myself so I can't say anything authoritative on that.
The problem is that the Java world is full of over-engineered solutions which are really targeted at extremely large enterprise applications surrounded by a community of folks who like to menturbate about them. As a startup, you don't need to focus on those solutions. You'll not need 99% of them.
However, you can choose bit and pieces. I'm using Vaadin, JPA and Guice and finding it incredibly productive and am able to deliver functionality in far less time than it took me in Rails. I'm not using 90% of the Java EE stack, but it's available to me if/when I need it.
Rails was a great thing for the software development world. It drove convention over configuration and caused a lot of the competing technologies to pause and question why things were the way they were. But IMHO, managing the whole Rails stack (rails + coffeescript + css in our case) across a group of people became a chore. YMMV, but the statically-typed nature of Java along with the very capable technology stacks you can assemble have eased our daily jobs tremendously.
If you are developing business-facing applications and not just consumer web sites, as we are, I think Vaadin makes a lot of sense. It might even make sense for consumer sites, but that's not our current target.
I use Spring every day as part of my day job, and I agree that a lot of the development time slowness is due to culture/enterprise. But that is a serious problem that can't be readily dismissed.
The benchmarks, although good, are a best case scenario and of limited aspects of those frameworks. Real applications don't behave like this. What if on the average case such huge differences in performance becomes eclipsed by other factors?
If someone from my team suggested to use Cake PHP I would say no without even looking at it once. Not because I would never trade performance for productivity, but because I simply do not believe even for a second that productivity gains are the reason for that kind of horrific performance.
In particular, Lua-CJSON [1]
[1] http://www.kyne.com.au/~mark/software/lua-json-performance.h...
(I know, everyone wishes their favorite language was represented and not all can be tested. I'm just posting this in case the OP does a "Round 3")
EDIT: fix typo
BTW, LuaRocks is supported [2]
In particular, this is awesome because it's introduced me to some new frameworks that I hadn't even heard of (ie vert.x) that seem extremely interesting.
Keep it up and many thanks for this contribution!
https://github.com/TechEmpower/FrameworkBenchmarks/pull/43
We hope to get more as well.
Compojure's syntax and design seem really attractive... is anyone using that in production? Would it be a reasonable choice for a "serious" web application?
Note that Compojure is more of a routing library than a framework. It's similar to Flask or Sinatra. Clojure's philosophy includes aggressive separation of concerns such that full-stack frameworks aren't really "the Clojure way".
I also see that Snap got pulled in a couple days ago, so it will probably get tested in the next round. Should be interesting to see how it fares against Yesod.
It uses Warp.
Actually, the performances of Yesod are not so bad, as Play's are. Top performing frameworks (netty, vertx, go) are not web frameworks but async programming frameworks. They don't carry a lot of things around -- just push the serialized data on the raw socket output stream. Yesod/Play/Django/Rails/... on the other hand must manage routes, content types, browser headers, etc etc....
What I - again - find most interesting, is the large discrepancy in the 20-query test. Common wisdom seems to be that language performance is not important b/c every language hangs in the same IO/DB.
I'd like to see Lift in there if you guys do another round.
https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
Compare this to the Java Servlet, which just serializes directly in the Servlet callback.
https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
The former seems like overkill.
From our first benchmarks post: http://www.techempower.com/blog/2013/03/28/framework-benchma... "Why include this Gemini framework I've never heard of?" We have included our in-house Java web framework, Gemini, in our tests. We've done so because it's of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we're proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.
If not, can anyone explain in layman's terms why that should be the case?
PHP was never designed to be fast. As far as execution speed goes, PHP will be in the same ballpark as Perl?Python/Ruby. PHP comes up as faster in these tests because of all the extra things the frameworks are doing. See the_mitsuhiko comment above regarding flask for an example.
With such a trivial request handler like the one in benchmark, only a small percentage of the execution is actually taken up in the view, so all of the time you're seeing in the benchmark is pure overhead of the framework. A longer running view across all of these frameworks would cause things to even out a bit. All that being said, however, it's not an unfair comparison. After all, a benchmark of this nature should be a measure of overhead because that's really all there is to measure.
I don't know about Sinatra or anything else but I can give you the reason why Flask out of the box performs like it does. Unlike PHP Flask is threadsafe and uses locks and thread or greenlet local storage. This level of indirection adds a basic performance penalty when executed on a single threaded server like gunicorn. There are ways to change that if you think that becomes a performance problem by replacing the stack object with one that does not do dispatching and by removing the locks.
This in theory is easy to do, it's just generally not a real issue since most Flask apps will actually wait on database more than they waste CPU time.
Additionally Flask has a huge request handling overhead which will be primarily noticable in dummy benchmarks that don't do much in view functions. If you have an application that actually barely executes anything in the views you can subclass the Flask object to remove some of the functionality you normally always pay for (request processors, session initialization, message flashing etc.)
Lastly Flask has some sanity checks when Python is not run with `-O` that simplify debugging. Depending on which code paths you hit you might be paying for some of that.
In the database test the primary slowness is most likely not Flask but the way the database is configured.
Generally if you ever encounter Flask to be an actual performance problem I am more than happy to point you to ways to get more speed out of it. Generally speaking though there is not much point micro-optimizing until you notice it is a problem.
They are written against the core database layer or the framework (in case of Django the Django ORM) and in case of Flask: Flask-SQLAlchemy. The default config of both of these things is to not pool connections. That is not true for all other frameworks.
Also I'm not blaming anything here, just pointing out that there are differences in the setup.
1. More sorting options please :) I really want to sort by standard deviation.
2. Also in case of rails and other single threaded frameworks are you spinning up multiple processes?
3. Why is Go not being tested in the Database access tests?
4. Can you test rails with unicorn too?
2. Pat (pfalls) should know the answer to that more definitively, but we are definitely trying to use the CPU cores as fully as possible. I think you'll see some comments by Pat elsewhere in this thread suggesting some things coming in round 3.
3. Good question! Mostly because we haven't had the time to add it. I would really like to see it added, though. If you know Go and can write the code, might I entice you to submit a pull request?
4. I believe that is the plan, yes.
I cant't wait for the next round, when the updates of the last two days are included (elli, grizzly, play-java, play-scala, snap etc).
ps: i agree that raw jdbc would enhance the performance.
Kind of curious why jruby doesn't provide more of a boost (especially given compojure doing relatively well, too) and would like to see Django+Jython in version 3 (and +1 to the .NET/Mono stuff)
I was also surprised at the difference between the Java and Scala play test, since I thought they were supposed to be similar. But it looks like the approaches are quite different. The Java DB test uses the ebean ORM while Scala does not. The JSON code also looks a bit different, with the Java version surfacing more Jackson internals. I don't know if these are necessarily "wrong", but perhaps it's not as apples-to-apples a comparison as it could be.
If the contributions we've seen so far are any indicator, we're going to need to get more clever with how to show and hide rows in our results tables! :) But I'd like to see a few more rows added to cover the various permutations of the ORM and JSON options, as you point out.
EDIT: @json performance difference:
Seems to me like they tested these two versions:
Java - https://github.com/TechEmpower/FrameworkBenchmarks/blob/5203...
Scala (the version with the not working db) - https://github.com/TechEmpower/FrameworkBenchmarks/blob/b89f...
The scala version uses val, which ist like final in java. final enables the JVM to cache this object an run optimizations. So I suspect the object creation only happens once in the scala case, whereas in the java case on every request. I think this is happening here which results in much better performance for the scala version.
But maybe I am wrong and the difference stems only from implementation differences in the controller etc. I'm a php guy and don't know the jvm or scala very well. ;o)
We do in fact have 5 and 10-query runs available in the data. Switch the views to "Data table" or "All samples (line chart)" and you should see the data for 1, 5, 10, 15, and 20 queries per request.
Edit: Just saw this pull request come in, so expect some Go 1.1 love in the next round. https://github.com/TechEmpower/FrameworkBenchmarks/pull/64
cd go/src
hg pull
hg update tip
./all.bash--- edit: found it. https://github.com/TechEmpower/FrameworkBenchmarks
Seems to be using cluster. I'm not surprised that Go and servlets were faster... just surprised to see NodeJS in the middle of the pack position. Though performance isn't the only reason I really like NodeJS
We'll be including this in the next round.
It's worth noting that Rails' default JSON solution is the compatible-everywhere pure-Ruby JSON generator. Using a drop-in replacement like the oj gem will drastically improve throughput there.
I didn't get to the pull request for this round, but tweaking the GC parameters for the Ruby install should dramatically improve the general performance of the non-JRuby Ruby tests, as well. I'll see if I can get a PR in. :)
Obviously there's a price you have to pay for all the things Grails built on top of Spring/Hibernate (in the benchmark it looks like Grails' performance = Spring / 2), but in general, it's still faster than just writing everything in Groovy (Ruby, Python, ...).
One of the nice things about Grails is that you can easily drop down to the Spring and/or JVM level when needed for performance.
And I would love to see node-mongodb tests with the native Mongo driver and not with Mongoose (which is slower).
Also, just to reiterate something I've said elsewhere. We've done some spot testing with Go 1.1 and its performance is super. We really look forward to showing Round 3 numbers next week!
It wouldn't really be a fair test if we put all these frameworks up for testing, suggested Gemini was better than sliced bread with auto-applying jam, and didn't back it up in a testable way ;-)