HTML5 Games 0.1: Speedy Sprites (and interesting browser performance analysis)

The benchmark results in this article (as well as the benchmark itself) are not very representative. There are just too many use cases to consider and the benchmark seems to only deal with a tiny subset - one that IE9 happens to handle nicely.

I believe that some browsers struggle with this benchmark, because it uses 2 canvases on top of each other (one for the background, one for the sprites).

Here's a benchmark I just put together, that only uses one canvas:

http://www.phoboslab.org/crap/notjsgamebench/

Chrome 10, Firefox4 beta 7, IE9 and Opera 11 are quite on par in this one. They all draw about 1000 - 1400 sprites at 30fps on Windows 7.

And here's an example where IE9 performs very poorly (press F during gameplay to see the fps):

http://playbiolab.com/hqx.php?s=4

I just have about 20fps in IE9, whereas all other browsers draw at a steady 60fps. This happens because IE9 seems to only accelerate the drawing of IMGs into a canvas - drawing canvases into a canvas is still extremely slow.

A couple odd things about this benchmark: IE9 runs different tests than the rest of the browsers. I'm not sure why, and there's no mention of this anywhere in the documentation. You can confirm this by running the benchmark - IE9 completes it much quicker than the other browsers, and if you click to see the results, you'll see it only ran 4 tests (while the other browsers run way more tests). There's a chance this artificially skews IE9's test scores up.

It's hard to reproduce the other scores from the benchmark (IE9 is certainly fast, though)... Chrome 8, 9 and 10 all put out a miserable 350 sprites by default on my machine, so I suspect he had to change some settings to get it to perform well. If only he had provided those settings along with his results, I could reproduce them.

Likewise, I cannot reproduce his Firefox results either - Firefox 4 manages around 1800 sprites on my machine. This is probably because he ran it under a VM - VMWare does a pretty good job of pretending to be a 3D accelerator, but it's not a real video card. I wouldn't be shocked if this negatively affected the performance of other browsers on the benchmark as well.

The difficulty of reproducing his results, along with how utterly bizarre the benchmark is - I need to compile node.js to run a canvas benchmark??? - makes me wonder whether it's of any value for actually measuring real-world performance.

Ultimately, though, an even more important factor is that this benchmark does nothing to measure the consistency of framerates - modern browsers can suffer from some pretty severe garbage collection pauses while running games and applications written in javascript. Even if your game is running at 60fps, if it pauses for 50ms every 5 seconds to collect garbage, it's going to be a pretty terrible experience. At present, both Firefox and Chrome suffer badly from garbage collection pauses in many browser-based applications, but this is not represented in most modern benchmarks.