Hacker News Reader

Comment by samps | Hacker News Reader

One possible takeaway from these graphs is the urgent need for better JavaScript benchmarks. Both SunSpider and the V8 benchmarks are sets of microbenchmarks and the difference between them is enormous: JM+TM is significantly faster than plain JM on V8 but is a little slower on SunSpider. The fact that the conclusions can vary so greatly among these two (both deeply flawed) benchmark suites suggests that the true "real-world" performance of the engines is all but entirely unmeasured. Which loads Gmail fastest? That would be a step in the right direction.

View on HN

About a month ago, Mozilla released their' "Kraken" benchmark suite, designed to better reflect what actual web-apps actually do:

    http://blog.mozilla.com/rob-sayre/2010/09/14/release-the-kraken/

It's not listed on arewefastyet.com, but apparently Firefox 4 handily beats Chrome and IE9:

    http://weblogs.mozillazine.org/asa/archives/2010/09/javascript_performan.html

The tests are: astar, beat-detection, dft, fft, oscillator, gaussian-blur, darkroom, desaturate, parse-financial, stringify-tinderbox, crypto-aes, crypto-ccm, crypto-pbkdf2, crypto-sha256-iterative.

One big thing that's missing here is DOM interaction, which is a real bottleneck for current web apps (and handling 'foreign' API well is JS engine's job too).

Here's an alternative: http://krakenbenchmark.mozilla.com/kraken-1.0/driver.html

I have to agree, that things like machine translated scheme and OS scheduling are not representative of real world applications. (Part of the V8 suite: http://v8.googlecode.com/svn/data/benchmarks/v6/run.html - specifically Richards, and EarleyBoyer)

Keep in mind that V8 is not only running in the browser, but has completely different usecases as well (see node.js).

That it's a javascript benchmark only means that.. it's used by google to finetune their code.. not created to compare javascript in browsers among several different browsers.

And yet that's what people seem to like using it for.

I still don't see why machine translated scheme is a good benchmark for fine tuning code. Javascript is a fine language to work in - especially if you are wanting to use it for things other than a browser (then you must really like it!).

I haven't seen the code but based on the description it's a benchmark that was originally in scheme and ported to javascript.. sounds fine to me. And "kernel simulation benchmark" may have a usecase too, just because you can't think of it, doesn't mean it isn't there ;) I can think of tests for scheduling, threading, asynchronous code, I/O when i think of "kernel simulation".

Again, people like to use V8benchmarks, when it's not made for browser comparisons. The fact that it's often used only shows that there is a lack of a real cross browser benchmar suite.

"...Florian Loitsch's Scheme2Js compiler"

No, Florian Loitsch isn't a human translator for hire - he wrote a compiler.

Yes, but the benchmark is not "how fast does V8 translate scheme to JS", it's "how fast does V8 run benchmarks in javascript that where translated from scheme some time ago". Atleast that's how i understand it.

That's true, but machine-generated code can be pretty nutty, depending on what exactly the Scheme2JS compiler does. It's legitimate JS code, but it might not be representative of what JS-written-in-JS looks like.

True, but i see that this is a valid testcase. For example GWT generates JS out of Java, so it might be useful to also benchmark machinegenerated JS.

I think it'd be useful to benchmark GWT-generated JS (and also Objective-J), but not Scheme2Js, which is rarely if ever used in practice. Compilers for different languages will emit code with completely different performance characteristics and feature usage patterns. (For instance, Objective-J particularly stresses the call site caching mechanisms in modern JavaScript engines with its "objc_msgSend" equivalent, since the indirect call in that function branches to so many different call sites.)

If you create a benchmark that measures X, it will be used to measure Y. It's the rule of benchmarks.

the richards benchmark is not only a simulation of OS scheduling though, it's a classic benchmark of method dispatch performance (at least from what I recall, which is why everybody is using it via a Smalltalk inheritance and not reimplementing the bcpl code)

Reply on news.ycombinator.com