Introducing the JetStream benchmark suite
webkit.org
webkit.org
(For example, did you know that every major JS engine now has a daylight savings offset cache, something which is entirely useless for any real code, but substantially speeds up the date benchmarks in SunSpider? Bleh.)
The state of C/C++/Fortran benchmarking has been much better -- not perfect, but much better -- for years, thanks to the SPEC benchmarks, which are created by an independent company that invites submissions for real-world programs that are suitable for benchmarks.
I wish there was something similar for JavaScript and HTML, though the fact that you have to pay for copies of the SPEC benchmarks isn't good.
I agree with you that a neutral and fair benchmark would be good. Unfortunately, the non-browser-vendor-created browser benchmarks out there tend to be uniformly much worse. I think the only way to get something that is high quality and also neutral is to create a neural group that multiple browser vendors join.
I stand by my opinion that SunSpider is a poor benchmark suite that should have been retired years ago.
Octane also has some bad benchmarks, and I agree that splay is one of them. I've written about this before at https://blog.mozilla.org/nnethercote/2012/08/24/octane-minus....
And you admit that at least some of the SunSpider and Octane benchmarks are bad, and yet you included them anyway? Argh.
I must say I also dislike this trend of creating benchmark suites that have entirely new names yet consist mostly of tests from prior suites.
> I think the only way to get something that is high quality and also neutral is to create a neural group that multiple browser vendors join.
I agree 100% with you there. Robohornet was an attempt at this, but was fatally flawed by the fact that the benchmarks themselves were terrible.
> The classic Dhrystone benchmark program by Reinhold P. Weicker, translated from Ada to C by Rick Richardson, and then compiled to JavaScript with Emscripten.
Oh, for heaven's sake. An ancient, entirely synthentic benchmark (the very worst kind, as Hennessy and Patterson say), that's been translated into a new language twice. Double-argh.
Did Robohornet get non-Google browser vendor involvement? It doesn't seem so currently, and I don't recall any particular push to get involvement. It seems like it was single-browser-vendor in practice (and also not very well designed).
I will hedge this by mentioning that there is a totally neutral benchmark that is actually good and more realistic than any of these. JSBench is created from the JS in real websites, was not created by any browser vendor, and has had advice and input from multiple vendors. <http://jsbench.cs.purdue.edu> It's not very good at modeling advanced / bleeding-edge webs though. Also, Safari spanks other browsers so hard on this benchmark that it's not a useful optimization tool.
There was a stewardship committee that had lots of non-Google people on it: https://github.com/robohornet/robohornet/wiki/Committee-Memb.... There was even a Mozilla person on it at one point, I can't remember who (someone I didn't know) but they asked to have their name removed once we discovered and explained how bad the benchmarks were -- it turns out that person had only had tangential involvement anyway.
But I just found the technical advisory committee, which consisted of three Google Chrome people: https://github.com/robohornet/robohornet/wiki/Technical-Advi.... So it was more Google-heavy than I realized.