TechEmpower Framework Benchmarks: Round 20

I suspect the examples would be few at the top of the list and many near the bottom.

I don't care much about performance either, especially compared to documentation, but that does not mean these benchmarks are useless. For instance, comparing similar frameworks can show big differences which illustrate different software architectures. Among the Nginx/phpfpm/Mysql fullstack-with-ORM frameworks, Laravel has roughly 8% of the raw stack capacity, while the other mature frameworks can do 2× or 3× more. I suspect this reflects Laravel's abuse of magic (calls to non-existent methods are redirected at runtime to other objects created on-the-fly, and so on). This may not be the real cause, but such a poor performance makes me suspicious about the complexity or the quality.

I wouldn't use TechEmpower as a reliable source for checking framework performance.

And it's also nice to point at TFB when arguing with your architects that 100 rps is not "a lot".

There is some value but it’s not a lot. At the end of the day it all comes down to how much time and motivation the person that implements the benchmark has, you can get pretty far in just about any language, after all they can all call into a native implementation so even an interpreted language can be fast. Then you can spend lots of time micro-optimising for the specific hardware the benchmarks run on.

An example scenario could be if a company has a choice between Java (Spring) and .NET (ASP.NET Core). Both are quite popular, have documentation, a big ecosystem around them, etc as you mentioned. If you might need to do low level/advanced web code at any stage the plaintext benchmarks may sway them to use .NET over Java; all else being equal. In that benchmark ASP.NET Core trumps Spring by a very large factor (Spring's 2% vs AspCore's 100%); you then obviously look into the benchmark to compare and contrast as you shouldn't trust them at face value. Although having said that if the .NET one is better tailored than the Spring one it does send a signal that the .NET one has a more performance orientated community willing to maintain the benchmark which IMO is a positive still.

We upgraded an app from Spring 4 to Spring Boot a while back and it divided our hosting cost by 5.

I started caring about speed when I worked on an app that became much bigger than anticipated. If you start with a really slow framework your only option is a rewrite

I read once on HN that a lot of modern day deployment strategies and tooling seems like it was created because of very slow apps. When you're running a rails app, you need to scale horizontally, and so you need to orchestrate that, and so you have autoscaling instances, kubernetes, and so on.

But if a single, beefy instance can handle all the traffic your app can reasonably expect to see (think, StackOverflow on just a couple instances), then a lot of operational complexity just dissolves away.

Managing 10 servers is perfectly ok, having to handle 100 servers would take a lot more work. A 1000 servers would require organizations to hire a lot more people and work a lot harder on automating server management, deploys, etc.

I agree so far as that just using some standard language like C# or Java is probably good enough even though there might be another language and framework that is faster. I will personally stay away from php, python and ruby for larger systems.

Great, let's see those metrics tested and compared in an easy to read table, then we'll have a matrix of qualities on which to base a choice of stack.

I also got a bit suprised over this other benchmark showing how fast C# can be: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I think what really hits home are the stories about people upgrading elements at the foundation of their stack—their platform or framework—after those foundations have seen optimization efforts. It is rewarding to see application developers realize dramatic performance boosts within their applications with so little effort; we feel we have contributed to this delight in a small way. For example, check out this post by the Azure Active Directory team [1] where they saw massive performance improvements by upgrading from .NET Framework to .NET Core 3.1 (which isn't even the latest version).

Some people focus too much on specific rank ordering and the optimization efforts made by those who jockey for top spots. It's better to consume the data in rough tiers of performance, however you choose to define those tiers. We tend to encourage people to consider performance as one part of the puzzle when selecting infrastructure software. Several high performance platforms and frameworks (in Java, Rust, C#, Go, Python, and more) are also ergonomic and easy to work with. And using high performance infrastructure software means you can avoid premature optimization in your application code and avoid premature architectual complexity. You can enjoy the trifecta of architectural simplicity, low-latency, and good scale headroom.

Obviously we know this project will always receive diverse and critical opinions. That's fine; hackers are a very opinionated people. For those who value the data, ourselves included, we are happy to keep putting the effort in.

[1] https://devblogs.microsoft.com/dotnet/azure-active-directory...

  - Even if it's JS, just-js make almost no use of dynamic memory allocations.
  - The authors rewrote himself a postgresql driver that support batching request.
  - it wraps high performance c++ libraries

That accurately represents how I've been forced to run applications in production too, when under huge stress and we're trying to squeeze everything out of the existing hardware.

Welcome to _Hacker_ News ;)

Edit: And absolutely yes, check out the code behind it. Every benchmark has their flaws and biases so it's important to verify before making conclusions.

In many languages, a bunch of web server code might end up being implemented in C and not the language itself. When you're creating a minimal endpoint for a benchmark, you might be exercising that C code and not the language itself. For example, the PHP implementation uses `htmlspecialchars` to encode the information which is implemented in C. That doesn't tell you much about the performance of PHP, but just that it has an optimized HTML escaper. The `asort` function used to sort the results is implemented in C and has more limited functionality compared to more general sorting functions that might be able to take lambdas in other languages. The PHP implementation even takes advantage of `PDO::FETCH_KEY_PAIR` which will only work if there are only two columns.

Likewise, the PHP implementation doesn't use templates, but manually builds strings. The fastest Python implementation actually renders a Jinja2 template: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast.... That's much more realistic in terms of what you'd do in the real world, but you're going to be carrying the overhead of a real-world template system like Jinja2. Part of Python's failure here is that no one wanted to implement an optimized Python version that would just build a string instead of rendering a template.

Changing the test constraints a bit would ruin a lot of the advantages that PHP used there. Let's say that you had to retrieve three columns: `id, sort_key, fortune_text` and sort on the `sort_key`. Now you need to read more information back rather than just being able to make it an associative array (hash map). You need to be able to sort based on that sort_key which means probably giving a sort call a lambda.

This isn't limited to PHP. A bunch of Go implementations do things like allocating a pool of structs and then re-using the same structs to avoid the garbage collector. A lot of implementations create result arrays sized so that they won't need to be re-sized (creating additional allocations and additional GC work). The rules say this isn't allowed, but they do it anyway.

So, before comparing tests, I'd look at the implementations to make sure that they're comparable. A Django implementation that actually returns objects and is rendering templates and looks like a canonical Django implementation is very different from an optimized PHP version trying to avoid running any PHP as much as possible. When we start looking at the popular PHP frameworks which will be executing a bit of PHP like CodeIgniter or Laravel, we start seeing performance similar to Python frameworks as the PHP code is doing similar things like rendering templates. It just happens that no one implemented a Python version that didn't use a fully-fledged template renderer.

And this is the weird thing: the benchmarks changed your perception of PHP while comparing things that weren't similar. I think PHP is often faster than Python and Ruby, but probably not to the extent that your perception might be given these benchmarks.

I actually find it fascinating to look at the implementations and see which communities care about realistic implementations vs. leveraging all sorts of tricks to win the benchmark.

Most of them check the Nth letter of the HTTP method to do routing, hard code the content-length, etc. This benchmark shows which language is the fastest to throw the hardcoded data into the socket.

Meantime, Django[1] is the simplest and a real-world application.

1: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

I only see a single "unsafe" in the Rust implementation[0]? And that's only in the "raw" Actix instance, not the pg. Or are you talking about Actix framework being mostly unsafe?

> only the Go benchmark is written like “standard Go” such that most people writing a service in Go will write

Maybe it's just me but the Actix apis seem pretty ordinary.

[0] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

That seems like a clear cheating case, over-optimised to the circumstances of the benchmark.

With absolutely no snark intended: what alternative do you see (or imagine) to this? Applying quite a bit of mechanical sympathy to tune a program to the system and visa versa seems essential and inescapable.

Maybe so, but we've been running https://pernos.co on Actix for a couple of years now with no issues. We didn't have to use "unsafe" either, and I understand that Actix's use of "unsafe" internally has also been significantly reduced.

Tokio's "warp" is much less mature than Actix AFAICT. Version numbers can be misleading.

It’s worth looking at low level HTTP, socket and concurrency management if you want faster performance, but that’s not really a language-exclusive feature at that point. And the more realistic you make the benchmark — the more communication between microservices, for example - the more your application architecture, deployment hardware, kernel and network tuning, and so on can play a role.

I am reminded of http://rachelbythebay.com/w/2020/10/14/lag/ for example, as something really low level you probably don’t need to worry about… until you do. http://rachelbythebay.com/w/2020/05/07/serv/ also. The same is true of most of the performance optimization at the top. Really fast? Yes. Useful? Depends on the rest of your code. Are you writing haproxy? How much does your app really need to do? Read the benchmark source code and get inspired, maybe.

My conclusions for the moment: Switch to Go if low memory usage and high performance is as critical to you as post-1.0 stability. Go is generally stable these days ;-) Otherwise if instability can be tolerated but you want high performance, use one of these newer web server techniques and write a web server in unmanaged C++ that has very minimal functionality with language bindings. Just-js can serve as an example. Heck, the PHP benchmarks show that if you use PHP to write your own HTTP server you can still achieve high performance. That tells me the advantage here goes to web servers that literally “do less” rather than picking a language as faster over another language… especially when most (all?) languages can interface with C or C++ to do the HTTP layer at high performance while writing your code in whatever language you choose…

Techempower classifies each one. Actix is what they call a "platform"

>a platform may include a bare-bones HTTP server implementation with rudimentary request routing and virtually none of the higher-order functionality of frameworks such as form validation, input sanitization, templating, JSON serialization, and database connectivity

Even if you use the top ones in the list, you will have to build abstractions in code to handle software complexities. Those abstractions will reduce performance.

I do wonder, if we could use AI to dynamically remove those abstractions, to the extent possible and transform abstracted, multi-layer code to simple functions / compile targets, improving performance. Sort of like looking at the project, at this instant, and folding all database / processing code into higher performance code that reduces function calls or object instantiations.

Yes they offer everything we need for real development at enterprise scale.

Not to say they don’t have value — they do — but keep in mind your code may look very different from benchmark code. I do think it’s worth looking at the code to examine why some approaches scale better than others, though!

Rarely is it the case that more code = faster performance, but sometimes features you need are sacrificed for extra performance even when a benchmark is classified as “realistic”. Viewing source code is how you can determine which frameworks actually best fit your needs, rather than just which ones performed well in this test…

For example, it’s easy to think that raw Go performance is terrible compared to Rust or C++, but it’s worth pointing out that the raw Go example uses commonly available standard libraries baked in to the language to accomplish almost everything it does. Most other examples at some level or another have to deal with epoll and concurrency manually themselves or via the web server replacements they use. Based on this, the overhead or raw implementation of socket management is often what is being benchmarked. Implementations like just-js use C++ and epoll under the hood to implement their server, like most of the other high performance examples at the top of the benchmarks.

The question often then becomes, do you want to write your own HTTP server or take advantage of an API someone else has written that might have more features or middleware? And remember that configuration matters, I forget off the top of my head, but the Go examples with a new web server had more performance optimization than the raw-Go benchmark had. Obviously I should submit a PR to fix this, but my ultimate goal here is to say that reading the source code will both help you become a better developer but also catch the shortcuts and techniques used by high performance implementations…

The master branch of rocket does have async (and stable rust, not nightly!) support.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/f340...