Web Framework Benchmarks

Do you have any plans to add memory usage to the benchmarks? It adds an extra dimension and sometimes even shows if there's a problem with a specific implementation (e.g. in the benchmark game it's often a hint for a program that can be optimized if the ratio compared to other languages is off).

> Anything other than C or Java really

Rust, C#, and Go also seem to fit the bill, according to these benchmarks. C# particularly has rocketed up since the last time I looked in depth at these. Pretty impressive work from the .NET Core folks, I'd say.

I'm not sure how many companies use C, C++ for Web development. In my opinion C/C++ wont be good idea to use for most. It is about choosing the best ration b/w tool speed and developer speed(productivity).

There are very few companies which requires million per second request. For me Java, C#, Go, Node provide the best compromise b/w speed and productivity.

Something that I learned after being part of a startup back when doing web servers in Tcl was cool (AOLServer and such), and Zope was hip, was that I never wanted to work again in software stacks that lacked a JIT or AOT compiler for production code.

I find it very frustrating that Scala, a language that is entirely capable of running just as fast as Java, is full of frameworks that are relatively slow compared to the java frameworks. It has to be some sort of fixation on reinventing everything from scratch in order to fit into some idealized idiomatic scala way (ignoring the fact that there are a million different versions of "idiomatic" in the scala community). Even the Play Framework, which used to be based on Netty and was pretty damn fast, ended up spending entire release cycles focused on migrating to an akka-based backend and they ended up with shittier performance and now they have to maintain two backends because nobody wants anything to do with their new akka backend.

Sometimes mutability and imperative code are okay, even in scala.

Is it? Concurrent connection handling at scale is a major problem, and plenty of Fortune 1000 companies would jump at the chance to have even 5% fewer instances deployed.

366 rps in a test scenario doesn't scratch the surface of what I need, and I'm not doing anything I'd consider crazy scale. When you need to support hundreds of thousands of rps, small improvements can be very noticeable.

I agree with the overall sentiment. Plus almost all cases I run into performance issues with e.g. Python in general web development, that main problems were how the code was written, not what it was written in and being bottle-necked by the database.

That said I disagree a bit with saying it is a a solved problem unless you are Google-scale. In developing countries if you are in a small company without big investors, you need to do quite a bit more juggling with infrastructure budgets once you are past MVP compared to for example where I'm from Switzerland.

Until your app gets on HN/Slashdot/Reddit etc and experiences the hug of death.

Jokes aside, I do agree that from a developer's standpoint, the optimal way to act (optimising for business success) is as you say, ignore performance at first. But from an engineering/craftsmanship standpoint, it should really bother you to have an inefficient system.

Just think of all that extra CPU load, and all the CO2, all the coal burnt to sustain it ...

It's not so much about immediate performance, but about headroom.

The blog post https://www.techempower.com/blog/ puts it better than I:

> I argue that if you raise the framework's performance ceiling, application developers get the headroom—which is a type of luxury—to develop their application more freely (rapidly, brute-force, carefully, carelessly, or somewhere in between). In large part, they can defer the mental burden of worrying about performance, and in some cases can defer that concern forever. Developers on slower platforms often have so thoroughly internalized the limitations of their platform that they don't even recognize the resulting pathologies: Slow platforms yield premature architectural complexity as the weapons of “high-scale” such as message queues, caches, job queues, worker clusters, and beyond are introduced at load levels that simply should not warrant the complexity.

I agree with your general point, but there are many more companies and products that do require high performance beyond Google.

Yes, exactly. I'd love to see some examples of a CRUD based API built on top of h2o. At least this appears to be possible with one of the higher performers (cppcms).

I think that it allows you to evaluate the tradeoff between DIY services on raw http servers and using a framework that reduces development time but is much slower in production.

We use the word "framework" as a term of convenience covering the full spectrum from platforms, micro-frameworks, to full-stack frameworks. We are also liberal with accepting contributions from the community, which means we do indeed include several frameworks that are well outside the mainstream.

That said, in tests such as Fortunes, you will see many full-stack frameworks demonstrating their capability to deliver myriad web application fundamentals (such as request routing, database connection pooling, ORM, XSS countermeasures, character encoding, data structures, and server-side templates) at very high performance levels.

This project establishes a high-water mark of performance. Very few web applications process anything remotely close to the requests per second we're measuring. But using Fortunes as a proxy and applying a coefficient (e.g., 0.005) to adjust for real-world application sizing can give you a very rough but nevertheless potentially useful approximation of real-world expectations. For example, one can guess their application is 200 times more complex than our Fortunes test. Rough back-of-the-envelope math will then estimate a framework from the ultra-high performance tier (~200,000 fortunes/sec) might yield a generous ~1,000 real-world app requests per second on a modern Xeon server. Meanwhile, a framework in the middle tier (~10,000 fortunes/sec) would yield a more constraining ~50 real-world app requests per second. Again, this math is hand-waving and you can interpret the results in whatever way you prefer; you can dismiss the validity of approximating things so coarsely. But my experience is that real-world applications based on frameworks I've used—which are sprinkled among all tiers of our results—do align with this hand-waving approximation.

Edit: I invite you to read the last section of the blog entry about this round [2] where I argue the same point in another way and share one of the tweets from an application developer whose real world application benefited from the performance improvements made to his favorite framework.

[1] https://www.techempower.com/benchmarks/#section=motivation

[2] https://www.techempower.com/blog/2018/06/06/framework-benchm...

This is probably due to the fact that PostgreSQL (libpq) has an "async" interfere built in and MySQL/MariaDB don't. These benchmarks are running very basic queries that perform well on either MySQL or PostgreSQL.

> ( Or was that a joke I am not getting )

Yes, that was a bit of a joke in the blog entry. We suspected for years Docker would be a good fit for this project. Only within the past months did we find the time necessary to convert the hundreds of test permutations to Docker. It wasn't a quick thing, but we did get into a rhythm. The "joke," for whatever it's worth, is simply that we didn't sufficiently appreciate how useful it would be for this project and how insignificant the overhead would be. Perhaps we would have prioritized it higher in the past had we known. But it's done now!

In brief, the Docker conversion effort has increased the stability and reliability of the results.

There are a huge number of ways to consume the results and as many opinions. We welcome the diverse points of view and hope you find the data useful!

It's interesting to look at the source code. There's definitely some heavy differences in how each implementation is handling underlying aspects of the tests.

Just looking at the database related ones to see about some of the differences in interesting.

It also doesn't look like the Elixir code has really been updated in about 2 years (aside from version bumps). It's still using a JSON encoder (Poison) that's 4x slower than the primary one (Jason). For multiple queries, Ecto hands back the connection after every query for concurrency sake. Looks like the Plug logger is still setup on the endpoint.

I'd be really interested to see the Discord folks look at that as a pet project. :-)

What kind of traffic do you see in production? Rails gets hammered in these benchmarks, but does just fine for most companies' requirements. It's pretty rare that you'd actually need to eke out the raw numbers that the top contenders get. I'm of the opinion that-- within reason-- developer ergonomics, and ability to quickly solve business problems are more important than raw performance, so long as performance is good enough by some agreed upon metric.

Yep I feel the same about Elixir and Phoenix. It's way down but again, the "way down" is 20k requests per second. Not too shabby. The biggest project I've worked on had 50 requests per second, and that was an E-Commerce site that brought in 9 figures a year in revenue.

I'm sure IoT workflows are mainly where you start to see these more insane RPS numbers. But still, this just tells me there's so much great choice out there for pretty much any platform you want to stick with.

I'd love to see another metric that normalizes between Cloud and Physical, like $ per request. We don't know how much the cloud server cost vs the physical, do we? I mean it's a 10x difference in performance, is it a 10x difference in price?

Update: Azure D2v3 instances (used for the cloud benchmark) are about $55 a month. ($660 a year).

Just the Xeon in the physical costs $1,500. The full server probably costs $5k? So assuming a server has a life of 3 years, you're looking at the Azure instance being $1,980, the Dell being maybe $5k? So it's about 10x performance for 3x the price.

Someone please fix my math and fill in the gaps :)

I wonder how a Spring WebFlux [1] variant of the spring benchmark would perform in comparison. Also the spring benchmark has been updated to use a recent spring boot release about a month ago [2]. Before that it was using spring boot 1.3.5.

[1] https://docs.spring.io/spring/docs/current/spring-framework-... [2] https://github.com/TechEmpower/FrameworkBenchmarks/commits/m...

I think it would have been even faster if they'd test with SQL Server because you can do things with SQL Server that you simply cannot do in PG such as stored procedures that return multiple resultsets, which saves a ton of round-trips.

I'm confused by this as well. Though I've seen odd things locally like 7.1 being significantly slower than 7.0 when doing local AB tests.

theres a lot of weird things that dont make sense. like how a minimalistic framework sinatra/padrino is slower than rails doesnt seem to make sense at all.

i feel these benchmarks are misleading to say the least.

+1, and the test seems to use php 5.5, this is definitely not OK

Also since we are at php, how Laravel is much faster than Symfony? and why Symfony has so many error?...

The best resource to answer your questions is the Requirements for the various test types [1]. But in brief:

* Yes, Fortunes is named after the Unix tool of the same name. In our case, it's a test that executes a query of all rows in a table, adds an additional item, sorts the values in the application code, escapes the values as a XSS countermeasure, and then renders them using a server-side templating library.

* Yes, the single-query test is always executing a single-query per HTTP request and is measured at various concurrency levels. The multi-query test is measured with consistent concurrency and varies the number of queries executed per HTTP request.

[1] https://www.techempower.com/benchmarks/#section=code

The bottleneck on plaintext appears to potentially be physical network bandwidth.

https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...

I think TechEmpower are doing a great job on these generally, but the plaintext benchmark is problematic because they allow pipelining. (Which isn’t enabled in browsers, isn’t typically used in request libraries and tools, and has generally been sidelined) Frameworks which are tailored with that in place can get a big boost in the ratings there.

If you want a measure of plain request/response I’d use the JSON benchmarks as the baseline.

Nope, they run a warm up stage.

> How different are the various frameworks here?

Quite different. We have included test implementations for frameworks that span 26 computer languages. There are an infinite variety of opinions about how to do things in computer programming in general, and our project is no exception.

And on the other hand, fairly similar. Tests should stick to the requirements [1], which are designed to be permissive but sufficiently clear on the expected work load of each test type. The principal goal is that test implementations should be realistic, and we will mark those the community believes are not realistic as "stripped" implementations.

> Are they all running with SSL?

No, not yet. But planned future test types would include SSL/TLS.

> Do they keep track of session cookies?

Generally no. These are intended to exercise anonymous requests. But a future test type could include session management.

> Do they have Content Security Policy headers?

No, that is not a requirement of our tests. We have specified which headers we expect. Others are optional.

> There's probably a million different feature configuration differences between each framework.

Yes. And it can be challenging at times to get all of these opinions to fit into the same box. For example, we've to-date kept SQLite implementations out of the project since those would not incur network costs. And other times the box has to be reshaped a bit to make room for new consensus opinions about what is suitable for "production." We recently made a decision to allow for innovative features in the Postgres protocol that are—in a manner of speaking—analogous to automatic pipelining. That conversation is still ongoing on our discussion forum.

[1] https://www.techempower.com/benchmarks/#section=code

Yep. It took me about six months to write a replacement of ActiveRecord with an identical API (for what was needed) that could scale to arbitrarily large number of writes.

Shit, pre-A16Z raise 500px ran on Rails and they were a social network around photography. You think Twitter is bad? Try timelines where you have half a million photographers liking hundreds of photos an hour and every like gets pushed into every feed of the people following them. Guys syncing their entire photo library. It was 5 application servers, 4 MongoDB servers for the timeline with some crazy data structures, one or two MySQL DBs.

I think most web projects these days should be in either Pheonix, Rails, or similar. If you need something really fast here or there just fork off the request in Nginx or compile something in Rust or C and extend it into Ruby (or whatever). Or have a compiled worker that communicates through the DB or Redis. There is this long tail of UI you need to make for every web project and nobody uses it 99.999% of the time. It should be in whatever secure language / framework that brings it to market fastest.

Im with you, although we will see if animations can hide loading times.

Btw, you will always have people that disagree with you. I get crap about the loading time of my website, but since I changed to a massive beautiful change, my conversion doubled overnight.

I think a human can decide for themselves if something takes too long. I ran into this issue and optimized, but I wouldnt be chasing benchmarks.