Performance Comparison of JavaScript Frameworks | Hacker News Reader

Performance Comparison of JavaScript Frameworks

stefankrause.net

112 pointsetimberg54 comments

Every time I see JS performance data like this, I'm always mystified to see lots of error ranges of a couple percent. Working on a JS game has made me dive pretty deeply into JS performance (on Chrome at least), and I constantly find that the hardest part of performance tuning is getting any kind of consistent results to tune.

For a random example, I tried reproducing the "Swap rows" test for Vanilla.js. The author found this took 7.83 +/- 0.18 ms, with five warmup iterations. On my machine, running the same command six times consistently gives results like these:

    swapRows took 10.944999999999709
    swapRows took 1.3250000000007276
    swapRows took 1.3200000000006185
    swapRows took 1.269999999999527
    swapRows took 11.875
    swapRows took 1.2750000000005457

Also, and orthogonal to the above, I've always found that even when performance is consistent across repeated trials, it varies widely between page reloads. (This isn't at all surprising, since v8's optimizing compiler isn't necessarily going to make the same choices every time - but it still makes things very hard to tune).

As an example of this, I tried repeating the "append to large table" test. The author's results were 225.3 +/- 1.89ms. On my (similar) machine, trying this five times (reloading the page between) varied between 225 and 329ms.

Am I missing something here? I've looked pretty hard into v8 perf tuning and various benchmarking libraries, and I've just never seen the kind of consistent results one always finds in comparisons like these.

You're basically right. Thus we've decided to perform 10 runs for each benchmark and drop the worst 4. This eliminates cases where the gc or a background system process causes a slow run. In consequence this also strongly reduces variance, but is blind on the gc eye. It would be interesting to include gc, but I have no idea how to do it in a real good way.

The results are comparable (but nowhere equal) between runs and the difference between the frameworks is in most cases large enough that the ranking stays consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01 better result in the average slow down, would you? For ember, mithril and cycle.js v6 performance might be something to consider depending on your use case.).

If you look at the console output this is just an approximation, the real measurement is performed on data from chrome's timeline using selenium. The console measurement uses a timer to guess when rendering was performed. Of course this is more inaccurate than using the timeline.

I think an equally interesting result would be to keep only the worst 1-4 benchmark results and compare those. In my experience some frameworks put much more long-term memory pressure on the GC, and by discarding those, your benchmark turns a blind eye on that.

In another words, I'd be interested more on the worst possible performance of a given framework, than their best (assuming the app is written in a reasonably good way otherwise).

Also, +1 to the sibling post that asks for longer runs, that would solve the GC question by including it altogether!

Do the benchmarks take a long time? Why not as many runs as necessary to give you a tight confidence interval?

Running all the benchmarks takes a few hours.

I'll keep that in mind for the next round. Let's see if that works nicely. Would you still drop outliers?

Thanks for the explanation! That sounds like as good a methodology as any to me.

It's hard to get consistent performance when you have any form of memory management, caching, or a JIT behind the scenes. I feel like this is just something you have to accept when writing code in a high level language.

Sure, and in tuning my game I've tried to work out ways of accounting for this. My point is basically that JS performance data with really small error ranges makes me suspicious about whether the data is meaningful or not.

Link to the blog post explaining the table: http://www.stefankrause.net/wp/?p=301

I've worked on a few apps that feature large tables like this benchmark does. With Angular for example, it's a really bad idea for performance to create DOM elements for each table row when most aren't even visible which I think is done in the benchmark: https://github.com/krausest/js-framework-benchmark/blob/mast...

There's lots of grid/table libraries available that use what's called virtualisation to stop lots of rows slowing the app down; rows that aren't on the screen aren't computed or rendered.

My feeling is if you avoid rendering data excessively in this way the performance differences between most frameworks aren't that important. If you're required to display 1000s of dynamic values at a time your UI probably needs work.

I found this chart unusable without being able to hide columns, so I created this quick script to enable toggling by clicking the headers:

https://gist.github.com/rattrayalex/8450e2378465e85ed846837a...

copy/paste into chrome console

Just for reference, the fastest framework can be found here: [Vannilajs](http://vanilla-js.com/) .

VanillaJs is notable for how long it has been maintained, like 20+ years now!

I wish the implementers would go ahead and do a full rewrite, but there are probably too many entrenched users who would have a fit if they did...

Thanks for pointing this out, this made me laugh. Javascript frameworks require a dose of humor to stay sane at this point.

This might be a better link, which includes a brief write-up of the work: http://www.stefankrause.net/wp/?p=301

Agreed. Hopefully one of the mods can change it

I usually come away from things like this with the distinct impression that they're pretty much all "fast enough" for my needs. Admittedly, I'm not asking a lot of JavaScript, and my skill level with JavaScript tends to lead me to take the path of least resistance, or at least the path of best documentation and most examples.

That said, this kind of third-party benchmarking is probably useful, because so many frameworks have benchmark charts on their own site which indicate they're all faster than everything else. (Which may not even be inaccurate...for the things they benchmark, they may be the fastest. Which again maybe indicates they're all "fast enough" and unless your use case exactly matches that of the specific benchmarks in question, the benchmarks might be misleading you about which one will be fastest for your app.)

Also, I had no idea there were so many JavaScript frameworks that I hadn't heard of (and there's at least a few that I have heard of that aren't on the chart). Every time I turn around, there's two or three more JavaScript frameworks. Surely, this will calm down at some point, and a couple of leaders will emerge and folks will kinda settle into moving those leaders forward rather than reinventing everything?

> so many frameworks have benchmark charts on their own site which indicate they're all faster than everything else

Whenever I see something like that, I usually lose interest. I don't want to see benchmarks, I want to see what it brings to the table.

Bad:

- ours is X times smaller compared to Y

- ours has the full package compared to X

- you don't have to worry about doing X. You would, if you were using Y

- ours is faster than X (some graph where they're winning)

- rant on X

Good:

- we took this approach

- we enforce X as it helped us achieve Y

- we took features we liked from X, Y, Z...and came up with this

It is definitely easy to get overwhelmed by so many choices. In the end Pareto still applies, 20% of fewer of technologies are being used in 80% or more of the projects. Right now, React, Angular and Ember are probably leading the pack.

It's already starting to calm down. On that list, you have a few broad categories, with some overlap.

Angular and Angular-ish (I use the term loosely) frameworks like Aurelia.

Observable based frameworks (Cyclejs, Angular 2?)

React and React near-dropin replacements (Preact, React, Inferno, etc)

A bunch of MVC/MVVM frameworks that are not Angular-like (Mithril, Vue, Ember...). This is where there's the most variety, namely because many of these are a bit older (even though many kept up with time).

So there's not really nearly as many as you'd think, and in some cases migrating from one to the other as need be isn't crazy.

I find Vue being very angular-like.

Same here

DOM manipulation should be added to the title and before calling one faster than other one should understand that a JS framework is (should be) a lot more than painting views super fast.

Any chance/interest at adding http://riotjs.com to this benchmark?

I added riot 2.5, but I need help. Can you assist? https://github.com/krausest/js-framework-benchmark/issues/13

Seconded, I adore Riot and it bugs me that it's left out of most benchmarks.

FYI Riot's alpha 3.0.0 release is much faster than 2.5.0, and is almost production ready (in fact, I'm using it on a project and it's rather reliable).

I am not trying to be snarky with this question, but does it really matter with JS? Its always going to be run in a browser by one user, not having multiple people connecting to it like the server side does. As long as it seems snappy to the user it shouldn't make a great deal of difference. Counter opinions welcome.

I feel like these comparisons are interesting but a bit one dimensional. A broad generalisation you could make is that frameworks with more features tend to be slower, but mean developers have to write less code.

A LOC or character count for each benchmark would help add a bit of context.

I agree with some of that, but I'd be more interested not in how much code is written for the test, but how far into the private API the code must go before the actual action is done. Kind of like using Test Coverage techniques. I imagine theirs only like 2 or 3 ways developers are, say, appending lots of rows, and the difference maker is how efficiently those techniques are being implemented.

This is benchmark porn. You can create a fast-performing web app with any of these frameworks, as long as you focus on the right metrics, like “time to interact”.

I'm really surprised that d3.js isn't on the list. It should be. I _suspect_ it would completely own most of the tests, but be bad on memory allocation.

Ember looks really bad.

I'm not a fan of Ember, but it's focus on tooling, developer experience, and sticking to it's core principles in the name of consistency is something to be admired. It's not as shiny as Cycle or React, but if you started working with Ember years ago, you've had reasonably robust tooling all along (as opposed to the bleeding edge constantly breaking crap of the other frameworks), and have been able to reuse your knowledge for years, so you're potentially quite productive.

At least it's the impression I get.

There was a lot of breaking the world changes until 2.1. I am using Ember from 2014. It stabilized lately so no breaking changes.

Cycle doesn't look so good from those benchmarks tbh

It's not fun to develop for, either.

Why is that ? I've never used Ember, but it seems like the framework with the best tooling and backwards compatibility track record. Maybe I'm wrong, I don't know.

The templating is very inflexible and the routing becomes extremely complex as your application grows in size. There are also a lot of gotchas with the runloop that require time and experience to understand.

yes, but it's so cool though

I've heard good things about the performance of Knockout.js - isn't this popular anymore?

Knockout not considered "framework", just a lowly "library". Like jquery.

Frameworks are those that force their own opinion/style on your code structure and architecture.

But isn't React also a library rather than a framework by this measure? I think knockout is no longer in development so like Backbone it doesn't really merit inclusion.

That's news to me. The last release was just couple months ago.

Knockout is pretty much feature stable with most stuff in place, so it won't have as frequent releases as younger frameworks.

Interesting. Awhile ago it seemed it has a support from Microsoft.

I'd have liked to see modern things like nunjucks and marko in this comparison list.

> modern things

Heh, so a bunch of unreleased beta-frameworks don't count as modern any more?

These frameworks are all so different it renders these results pretty moot. Vanillaja is fastest but it doesn't offer half of what React or Angular do.

Actually everything you do in Angular or React you can do in vanilla JS, so your point is moot.

Everything you can do in Python you can do in assembly language.

How is Knockout.js compared to these?

I welcome you to add a pr for knockout.

Take a look at the sample implementations here:

https://github.com/krausest/js-framework-benchmark/blob/mast...

I've just finished my first project in KnockoutJS, I'm up for contributing to a Knockout version.

I'd like to collaborate with others, so we don't end up with 3 different Knockout implementations ;-)

Ok, let me regex your email out and I'll ping you.

Why is React performance slowing?

I suspect there's a perf bug in React 15 that causes the 'clear rows' test to be an order of magnitude slower than react 0.14. Otherwise, React is getting faster.

edit: My hunch was correct, see here https://github.com/facebook/react/issues/7227

Reply on news.ycombinator.com