Hacker News Reader

Comment by hkarthik | Hacker News Reader

I see a lot less value in these benchmarks than I did 6 years ago or so when they first started appearing. Maybe I'm showing my age but the speed argument just isn't that important to me any more.

These days when I'm evaluating a framework, I look for strong documentation, integration with tools for observability and deployment, and compatibility with a well understood runtime that makes operations easier.

These things are way more important for building software that works, that scales, and is operationally efficient for a team of engineers to work with for several years.

Everything else is just vanity metrics.

View on HN

I think the biggest beneficiary of these benchmarks is Postgres. Its drivers are often the bottleneck and everyone who uses Postgres benefits from more work on them.

And it's also nice to point at TFB when arguing with your architects that 100 rps is not "a lot".

That said, the top C/C++ benchmarks are still using an unofficial fork of libpq client that supports batching. Rust and Java was on the top of the benchmark for a while just because they their own pipeline supported drivers.

> an unofficial fork of libpq client that supports batching

I thought they were just hand optimizing their code to the point they submissions were totally unidiomatic. But now I learn even dependencies get this kind of attention.

Years ago I submitted a ticket asking for inclusion of memory usage stats (interesting in small environments) and start-up time stats (interesting in serverless environments). But nothing ever happened on that front; I guess it's because TechEmpower is a Java shop and that are not typical metrics where the JVM shines.

I think the biggest benefit of these benchmarks has been to trigger improvements to all the low-hanging fruit in various frameworks. The differences at the high end don't really matter for many people, but I find it quite comforting if a framework is fast enough that I won't run into any performance issues I didn't cause myself. Because those I can fix, but if I hit a bottleneck in the framework it gets so much harder to figure out a way around it.

Same for me. Also, I like to see a race once in a while :)

Seeing Rust climb, PHP/Ruby/Python/etc fail. Also what Node.js is capable of. It's fun!

6 years ago every time you started a project you'd have to argue with idiots saying "you should do it in C because it's fast". Doing some genuine benchmarks of a scenario that's at least a little reflective of the real world is a huge step up from where we were.

Most of these benchmarks are insanely optimized. The code is rarely idiomatic. There's also a few frameworks in there that just wraps C libraries to get good scores.

I wouldn't use TechEmpower as a reliable source for checking framework performance.

Very true. I dug into the code a bunch last year and found several things that even seem to violate their own rules from a DB optimization standpoint.

I’d love to see stupid simple benchmarks against each framework’s build-a-blog tutorial.

I see more value. For some frameworks Techempower rankings have encouraged massive optimizations that save a lot of money in real world usage.

We upgraded an app from Spring 4 to Spring Boot a while back and it divided our hosting cost by 5.

I started caring about speed when I worked on an app that became much bigger than anticipated. If you start with a really slow framework your only option is a rewrite

What version of Spring Boot did you migrate to?

The newest as of last year. I think 1.7 . Sorry I'm not on that team

One value of fast frameworks is simplicity.

I read once on HN that a lot of modern day deployment strategies and tooling seems like it was created because of very slow apps. When you're running a rails app, you need to scale horizontally, and so you need to orchestrate that, and so you have autoscaling instances, kubernetes, and so on.

But if a single, beefy instance can handle all the traffic your app can reasonably expect to see (think, StackOverflow on just a couple instances), then a lot of operational complexity just dissolves away.

I feel the same way, and not just for these benchmarks but all.

There is some value but it’s not a lot. At the end of the day it all comes down to how much time and motivation the person that implements the benchmark has, you can get pretty far in just about any language, after all they can all call into a native implementation so even an interpreted language can be fast. Then you can spend lots of time micro-optimising for the specific hardware the benchmarks run on.

Agree that you want those things too. But when you are choosing between two different frameworks of similar maturity and you have to choose a tech stack it can tip the decision somewhat, especially if you don't have the time to POC a stack and you need to make a decision.

An example scenario could be if a company has a choice between Java (Spring) and .NET (ASP.NET Core). Both are quite popular, have documentation, a big ecosystem around them, etc as you mentioned. If you might need to do low level/advanced web code at any stage the plaintext benchmarks may sway them to use .NET over Java; all else being equal. In that benchmark ASP.NET Core trumps Spring by a very large factor (Spring's 2% vs AspCore's 100%); you then obviously look into the benchmark to compare and contrast as you shouldn't trust them at face value. Although having said that if the .NET one is better tailored than the Spring one it does send a signal that the .NET one has a more performance orientated community willing to maintain the benchmark which IMO is a positive still.

Obviously, performance generally isn't the main criteria when choosing a framework. I would argue that your "software that scales" isn't important either in most cases.

I don't care much about performance either, especially compared to documentation, but that does not mean these benchmarks are useless. For instance, comparing similar frameworks can show big differences which illustrate different software architectures. Among the Nginx/phpfpm/Mysql fullstack-with-ORM frameworks, Laravel has roughly 8% of the raw stack capacity, while the other mature frameworks can do 2× or 3× more. I suspect this reflects Laravel's abuse of magic (calls to non-existent methods are redirected at runtime to other objects created on-the-fly, and so on). This may not be the real cause, but such a poor performance makes me suspicious about the complexity or the quality.

I think there are some really big differences when you walk down the list (at least in the last TechEmpower benchmarks I've looked at). It could be a factor of 10x or 100x. It might even be 1000x between some slow solutions and the fastests.

Managing 10 servers is perfectly ok, having to handle 100 servers would take a lot more work. A 1000 servers would require organizations to hire a lot more people and work a lot harder on automating server management, deploys, etc.

I agree so far as that just using some standard language like C# or Java is probably good enough even though there might be another language and framework that is faster. I will personally stay away from php, python and ruby for larger systems.

To me the interesting part is when you can read the benchmark source code and learn about a high performance feature for the framework or language you’re already using. That’s happened a few times for me with Go, for example. I hope more high performance enhancements are added even for frameworks that would appear near the bottom. That’s what keeps this relevant to me. :) Well, that and knowing roughly what the performance ceiling is for raw socket HTTP in various languages ;-)

It would be very interesting to see an additional column of "prime examples" which link to the biggest or most heavily used web apps/apis built on a given stack.

I suspect the examples would be few at the top of the list and many near the bottom.

> I look for strong documentation, integration with tools for observability and deployment, and compatibility with a well understood runtime that makes operations easier.

Great, let's see those metrics tested and compared in an easy to read table, then we'll have a matrix of qualities on which to base a choice of stack.

Performance doesn't matter until it does. One thing that can be nice is that an efficient backend can handle an enormous amount of load on just 1 server, which can save you a ton a complexity for various reasons (deployment, integrations with other services, and just less cost). Then on the extreme end where you have a very high traffic server, efficiency might let you scale back from ~12vms to ~3, which can be a significant cost savings, especially if done with no impact in productivity, which I believe is often that case just by moving from interpreted/dynamic langs to statically typed compiled/jitted ones.

Honestly? Who cares about cost savings?

Computers are stupidly cheap slave workers and are easy to spin up compared to dev time. Does it actually matter in almost any profitable business?

The speed argument is a much better argument imo, but we have to think if 100-300ms is worth the tradeoff of a great framework with a strong community. (I've not seen anything come near the beauty of Rails in Java, SpringBoot is not it. Last I checked there was no good SideKiq alternatives which feels kinda vital for most apps)

If a Rails app can handle only 300rqs that's over 1 million requests served an hour. A SpringBoot app handling say 2000rqs? Thats 7.2million a hour definitely a shit ton more but you know what, I'll probably just set the EC2 autoscale to 7 more instances and pay the extra $1000 a month because its still a magnitude of 10x cheaper than developer time. I can see your point in keeping server count low for simplicity being an advantage but not enough to persuade.

I say all this but I do like the speed argument, I think its important that we keep web requests fast for a better user experience. I mean Stripe, Shopify, GitHub all get by on the slowness that is Ruby but I will say that lately GitHub has been super slow for each page request but GitLab has been very fast which is also Rails.

There is another view point if you are operating on a slim margin business that keeping infrastructure costs tiny is important and that's another case for a faster backend for sure.

Now the interpreted lang vs statically type lang argument I can agree with, I hope for a language that sits somewhere between Java and Ruby. It seems Crystal fits that bill at the moment but has no backer and will be years off a great ecosystem if it takes off.

Companies who are feeding 2k+ machines to run their web servers care about cost savings. Hundreds of billions of requests/day, not millions. The database systems to handle that load, networking, power, bandwidth and all the people who maintain it.

Not to mention Fb developed their own bare metal for their specific use case because in the long run it would save them money

> Computers are stupidly cheap slave workers and are easy to spin up compared to dev time. Does it actually matter in almost any profitable business?

The complexities mentioned generally fall on the developers and sysadmins who have to be aware of them and work around them, those people are definitely not cheap slave workers. Screw ups handling these complexities can bring the whole system down.

>There is another view point if you are operating on a slim margin business that keeping infrastructure costs tiny is important and that's another case for a faster backend for sure.

Which is why Rails is only suited for SaaS where you get paying user early in the development. If you require traffics for ads revenue it simply doesn't scale. And majority of the Web is still based on ad revenue model.

Postponing (often indefinitely) the need for scaling out your application can benefit you tremendously, not so much because of the instance costs, but primarily in terms of reduced complexity in all aspects of development and operations.

Also, for a SaaS, keeping the cost per served transaction low can mean the difference between making a profit or a loss in a market that has a converged on a price-point for a type of service.

> Honestly? Who cares about cost savings?

Honestly? Sane people.

I worked at 80 person company which broke and closed because of this. We almost escaped this demise by going from PHP to HHVM which unfortunately was very alpha at the time, but we simply ran short of money and time. If it had worked, we would have scaled down 15x our AWS costs and would have saved at least 40 jobs.

Reply on news.ycombinator.com