Framework Benchmarks Round 19
techempower.com
techempower.com
Funny to see rails, django, laravel and other most popular frameworks from dynamic languages at the bottom of the list with such bad performance. Even the Java heavy weight Spring framework is much closer to the top.
Not to mention that JVM can support all kinds of concurrent, parallelism and scaling models.
For example, https://github.com/an-tao/drogon (benchmark winner) or https://oci-pronghorn.gitbook.io/greenlightning/ just aren't pleasant to work with. For example, compare Green Lightning with http://sparkjava.com/ and look at how much slower Spark is while clearly being nicer to use.
I worked with Netty at work for two years and it was a dream to use something like Express or Flask on the side, not having to worry sharing/disposing ByteBufs, not dealing with the weird in-channel/out-channel abstraction (vs simple middleware), not investigating how async-/thread-safe a given library was, etc.
It just all goes back to trade-offs. You're going to have different considerations if you're building a Caddy competitor vs a complex game server. Another consideration is how horizontally scalable you can be which is going to be different for a forum vs a websocket server.
Spring WebFlux is also built on top of Netty, and missing some benchmarks, but I have no reason to think it would perform poorly. Its very easy to use for anyone with Spring experience.
The fastest frameworks are all nonblocking, like Express. They use streaming and reactive patterns, if it wasn't for Java's types everywhere some people wouldn't be able to tell the difference between Vert.X using RxJava and Express. The main design difference is that JS does non-blocking on a single core vs Java which does thread-per-core. Basically, in Java you can scale out non-blocking across all cores and still share memory when you want to.
If you don't want to deal with thread safety and non-blocking DropWizard performs quite well using traditional threads.
Just saying, performance and easy of use are not always exclusive. Maybe Vert.X didn't exist back when you were using Netty but these days it would be unthinkable to use Netty itself, the abstraction Vert.X provides is far easier to use and you don't really lose any performance.
Java's bad performance reputation is 90% due to old versions of Spring. Spring Boot has come a huge way, with performance maybe 100X better than Spring 4. Spring WebFlux takes this even farther with a Vert.X-like api, probably reaching near-Netty performance
Look at Composite Scores in Techempower benchmarks. Jooby, Micronaut, Ktor, Dropwizard, Http4k.. are all nice to work with and are all pluggable with different network IO frameworks like Netty for example. And they are not barebones http layers only optimised for speed, they come with a bit of batteries included but not that much that you battle the framework. They are less magic and more composable with other libraries where you can more easily know what is happening. Similar to Express and Flask. But even a heavyweight Spring framework is at the top. Flask is stuck at the bottom. Even express has bad results.
And, this is a personal preference, but having built production apps in both dynamically typed and statically typed languages, I prefer the latter for various reasons. Using Kotlin to write backend apps is really enjoyable for me.
Greenlighting on the other hand would definitely be confusing for your average java developer. You have to write code very differently, if your goal is to not create any garbage. The style will be familiar to the HFT folk, but it's interesting to see some of those same approaches used for http services.
What makes you say that Drogon isn’t pleasant to work with? Is it some specific framework APIs that can be improved (if yes, how?), or is it a general disdain against C++ and its syntax?
Still questionable as to what is allowed in "realistic" though. Aspnet core using a "rawdb" adapter doesn't seem like what I would interperet as "realistic".
Perhaps there needs to be a "marketed" category, for use as the docs describe. For aspnet core, this would include using entity framework.
Not much point in splitting the categories if it isn't a uniformly split thing.
It'd be nice to see benchmarks where: • Everything was implemented as documented in the framework's tutorials. • Realistic things were included end-to-end (payload validation, csrf (if applicable), etc).
Still seems like a whole basket of fruit comparisons here.
Why not? I don't think using raw SQL instead of an ORM is particularly uncommon.
The "platform" benchmarks (aspcore-rhtx-pg, aspcore-ado-pg) that appear to skip the ASP.NET Core framework entirely and write raw HTTP responses are definitely not realistic though. I'm going to assume that these are mislabeled because this filter is new and not all the benchmarks have been updated yet.
additionally, I'm not quite sure that this is the recommended way to return database results: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
What I believe is more "realistic" is to:
• Use the framework as described: https://docs.microsoft.com/en-us/aspnet/core/tutorials/first...
To be fair, I'm picking on aspnet core because I like it so much (via F#). The framework is great, but this gaming of public benchmarks (techempower, benchmarksgame) by "breaking the same rules as everyone else" rather than investing time into making these results more useful is disappointing.
I would have liked to see some other C++ ones on there, like Crow[0]
Looks like it's dead?
Given a newcomer is at the top again that seems to ring true? Does seem odd older frameworks wouldn't keep up with that game, but perhaps once they get the publicity and user base they can worry more about features than having the highest benchmark?
e.g at one point they used Askama as their templating engine (may have switched), which isn't really fair as using Askama means you force a recompile of your project for each template change - and I can tell you from experience that your compile times will shoot through the roof as well.
Nobody is going to bother doing that.
I say all of this as an actix-web user. It's my go-to. I just view these benchmarks as "if I ever needed to, I know I can wrangle performance out of it... but I won't assume it by default".
As a side note, Ramshorn is even faster than Askama and does not need compiled templates: https://github.com/maciejhirsz/ramhorns
Thanks for the heads up re: Ramhorns. I'd seen it before but Mustache templates aren't particularly my thing - Django/Python guy in a previous life, so I've got love for the templating models from there. Willing to sacrifice a small amount of speed for developer ergonomics, things almost universally get rewritten once big enough scale comes into play anyway.
For a language like Rust, compilation times and tooling support mean that the technique is perhaps not ready for prime time, but it’s fundamentally a perfectly sound technique, and can produce wondrous developer ergonomics (again I cite Razor).
There’s something quite incongruous about using dynamically-typed and basically uncheckable template language when you’re doing the rest of the code in Rust.
An actix-web + Askama user.
Let me be very clear: I like Askama and have submitted pull requests to the project in the past. For a new project where I need templates, I value the speed of reflected changes over compiling more stuff into the binary.
There are also certain classes of problems with Askama that you can avoid with something like Tera - e.g, with Askama, if your global layout relies on a value passed in to the rendering context, you'll need to shove that value on every Template struct.
Furthermore, most people will hit bottlenecks in other areas of their application long before templating becomes the issue (i.e, this was the same in Django-land too - templating via cached compiled Django or Jinja2 templates is almost never an issue). Askama is overkill until you need it.
My recommendation, having built projects with both, is thus: build with Tera initially, and swap out hot paths to Askama down the road as they both use the same syntax (roughly) so it's not too difficult to convert one to the other.
In a perfect world the projects would be merged, and you can use the Tera aspect in dev and get Askama in production for free... but that's easier said than done.
I’m not sure I understand this, i. e. TRANTOR (which is the underlying network library for Drogon) recently merged in a patch that optimizes memory handling in a low level component of its event loop where object construction was minimized (first tests show another performance increase of about 5–10% on top of the current benchmark results).
I fail to understand how such improvements have something to do with “all the latest methods and libraries”?
What's up with that?
Its amazing to see python based stuff so low in the benchmarks, yet so high in general popularity.
In cloud based servers, where requests per second directly translates to inverse USD, its imperative costs are incorporated into the selection process for the software stack.
I use actix-web on another project. IMHO Actix is conceptually sophisticated in a good way, has great async capabilties, great speed, and runs on Rust stable which is necessary for that project. However, actix-web had a continuity problem when the leader stepped away - see HN if you want to know more about this.
So we decided to try Rocket for this new project. This project involves more junior developers, has no special speed needs, and is fine on Rust nightly for now. The Rocket guides are excellent, especially for onboarding. The concepts feel simpler, which is fine for this project. We're eager for Rocket to launch its async version and its Rust stable version.
Ultimately the bulk of the code is similar, because both projects are web apps that use Diesel, Serde, reqwest, and other typical crates. We're looking to the future to extract some of the code into framework-independent crates, such as our API structs, business logic, and the like.
I write this specifically because I'm never right and I hope that my writing this publicly will lead to Rocket's immediate release on Rust stable just to prove me wrong.
And here's a cheer for those doing the work to make it async.
https://github.com/SergioBenitez/Rocket/issues/19#issuecomme...
However, in terms of relative performance the async version was 10x to 20x better. Bear in mind it was a simple hello world benchmark. I also disabled keep-alive. Reusing a connection gets you even better performance.
Rocket is a well designed framework, so I’d expect it to perform similarly to other rust frameworks once the async version hits master.
> TPR-3 is a composite hardware environment score for a three-machine configuration, derived from all test types for TPR-tagged <T> frameworks. (More about TPR frameworks and TPR-1 versus TPR-3)[1]
[1] https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Tech...
PHP 8 is due for release in Dec 2020 and will feature a JIT compiler which is expected to improve performance even further.
Regardless of what people think of PHP, it's nice to see the PHP community tackle performace and resource usage as a major priority. A contrast to parts of the development community who seem to care very little for performance and resource usage.
The vast, vast majority of PHP code you will find in the wild is running a more traditional stack, and those results are much lower down the list.
The results are still impressive though. PHP is very good at gluing fast C code together without slowing it down much. Especially Workerman is impressive, since it appears to still run a significant amount of PHP code.
[1]: https://github.com/walkor/Workerman [2]: https://www.swoole.co.uk/ [3]: https://github.com/rryqszq4/ngx_php7
I use swoole w/ laravel, still trying to figure out best ways to milk it for more performance, and also grok the way it handles async and passes variables around using swoole_Table.
But usually a JIT is made in a way that you can then move forward with your optimizations in small tested steps. So it makes sense for JITv1 to have a very similar performance as the non-JITed version. It probably generates highly familiar code. First make it do the same, then optimize.
If you change the stack with event driven, a pool and persistent application, you get that fast results.
Virtually every single item on my list is now completely irrelevant.
Although I'm not a daily developer using .NET anymore, I am astounded that such a turnaround was even possible. My hat is off to you and the teams you work with. Also, for the record C# is still a phenomenal language in my book.