Comment by fenomas | Hacker News Reader

Comment by fenomas | Hacker News Reader

Every time I see JS performance data like this, I'm always mystified to see lots of error ranges of a couple percent. Working on a JS game has made me dive pretty deeply into JS performance (on Chrome at least), and I constantly find that the hardest part of performance tuning is getting any kind of consistent results to tune.

For a random example, I tried reproducing the "Swap rows" test for Vanilla.js. The author found this took 7.83 +/- 0.18 ms, with five warmup iterations. On my machine, running the same command six times consistently gives results like these:

    swapRows took 10.944999999999709
    swapRows took 1.3250000000007276
    swapRows took 1.3200000000006185
    swapRows took 1.269999999999527
    swapRows took 11.875
    swapRows took 1.2750000000005457

Also, and orthogonal to the above, I've always found that even when performance is consistent across repeated trials, it varies widely between page reloads. (This isn't at all surprising, since v8's optimizing compiler isn't necessarily going to make the same choices every time - but it still makes things very hard to tune).

As an example of this, I tried repeating the "append to large table" test. The author's results were 225.3 +/- 1.89ms. On my (similar) machine, trying this five times (reloading the page between) varied between 225 and 329ms.

Am I missing something here? I've looked pretty hard into v8 perf tuning and various benchmarking libraries, and I've just never seen the kind of consistent results one always finds in comparisons like these.

You're basically right. Thus we've decided to perform 10 runs for each benchmark and drop the worst 4. This eliminates cases where the gc or a background system process causes a slow run. In consequence this also strongly reduces variance, but is blind on the gc eye. It would be interesting to include gc, but I have no idea how to do it in a real good way.

The results are comparable (but nowhere equal) between runs and the difference between the frameworks is in most cases large enough that the ranking stays consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01 better result in the average slow down, would you? For ember, mithril and cycle.js v6 performance might be something to consider depending on your use case.).

If you look at the console output this is just an approximation, the real measurement is performed on data from chrome's timeline using selenium. The console measurement uses a timer to guess when rendering was performed. Of course this is more inaccurate than using the timeline.

I think an equally interesting result would be to keep only the worst 1-4 benchmark results and compare those. In my experience some frameworks put much more long-term memory pressure on the GC, and by discarding those, your benchmark turns a blind eye on that.

In another words, I'd be interested more on the worst possible performance of a given framework, than their best (assuming the app is written in a reasonably good way otherwise).

Also, +1 to the sibling post that asks for longer runs, that would solve the GC question by including it altogether!

Do the benchmarks take a long time? Why not as many runs as necessary to give you a tight confidence interval?

Running all the benchmarks takes a few hours.

I'll keep that in mind for the next round. Let's see if that works nicely. Would you still drop outliers?

Thanks for the explanation! That sounds like as good a methodology as any to me.

It's hard to get consistent performance when you have any form of memory management, caching, or a JIT behind the scenes. I feel like this is just something you have to accept when writing code in a high level language.

Sure, and in tuning my game I've tried to work out ways of accounting for this. My point is basically that JS performance data with really small error ranges makes me suspicious about whether the data is meaningful or not.

Reply on news.ycombinator.com