Testing JavaScript performance on Windows XP is a crapshoot, at best | Hacker News Reader

Testing JavaScript performance on Windows XP is a crapshoot, at best

ejohn.org

I don't get this.

Quote: To better understand the results I was getting I built a little tool that runs a number of tests: Running an empty function, looping 10,000 times, querying and looping over a couple thousand divs, and finally looping over and modifying those divs. I ran all of these tests back-t0-back and constructed a histogram of the results.

What's the point of doing this? It's a microbenchmark, and it's pretty unclear what you're measuring. Why put so much effort in writing a blog post with graphs and the whole shebang just for this?

Even if you _did_ get constant results the results would be uninteresting. The results are either uninteresting and unreliable, or uninteresting and reliable. Shortcircuit logic keeps me from worrying about reliability.

And that the Windows GetTickCount/GetTime functions have a bad precision is common knowledge. No test is needed to establish that.

I'm pretty sure Resig is a smart guy, so I'm probably missing the point here. But I really don't understand what he's trying to accomplish here, or why the results are worth blogging about.

The point isn't to measure something specific at a rate - it's to measure anything to see if the timer results continue to bin the results. It doesn't matter if it's a microbenchmark, or not, at that point.

The issue here is that Chrome and Firefox 3 are perfectly capable of producing accurate time results on Windows - so there obviously is a better solution out there. In the ticket in which the Firefox team [1] fixed the bug they detail the solutions that they used to fix it.

I would disagree that this is a knowledge issue here, as well. It certainly wasn't obvious to the WebKit, Firefox, and Chrome teams who were all happily testing on Windows XP using bum tests - and those are some pretty smart people (IMO).

In reply to your comment further down the thread: "Measure browser performance.... why? Why do you care how fast your browser is?"

Because it's better to have fast JavaScript than to have slow JavaScript? Any performance tests are better than no performance tests - but I will definitely agree that quality is a huge factor. For Dromaeo [2] I included a number of DOM tests that provide a more-real-world result. I'm working to try and provide more (especially related to CSS and rendering).

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=363258#c19 [2] http://dromaeo.com/

No. No. No. It doesn't matter. Not one bit. Exception: for FF maintainers, opera programmers. But ordinary javascript programmers never have to worry about a ms here or there.

Let me quote again from your blog post:

This was quite odd since the tests should've taken, at least, a couple milliseconds to run and getting consistent times of "0" is rather infeasible, especially for non-trivial code.

Conclusion 1: The code _is_ trivial! Why? it takes < 64ms to execute.

Conclusion 2: The actual time spent by the code doesn't matter, because you can't notice the difference between the reported 0ms and the actual 3ms or so that it takes to execute.

Next paragraph:

Second, I was running some performance tests, on Internet Explorer, in the SlickSpeed selector test suite and noticed the result times drastically fluctuating. When trying to figure out if changes that you've made are beneficial, or not, it's incredibly difficult to have the times constantly shifting by 15 - 60ms every page reload

Why does it matter if loads 15ms to 60ms faster/slower? It's not noticeable. And once you see that the difference is roughly 15ms a quick google search will reveal it's a simple timer accuracy issue.

So why the graphs? Why test different browsers on different platforms when the outcome can't be anything else than the 60ms fluctuations you've already seen? But you knew the second you saw a single time reported as 0ms that that the time accuracy is not on the 1ms precision level, which means that they use GetTickCount or something which has a 15ms precision.

> It certainly wasn't obvious to the WebKit, Firefox, and Chrome teams who were all happily testing on Windows XP using bum tests

If they used the tests they would've noticed the 0ms results. Therefore, they either thought 0ms means it's <15ms, which is fast enough, or they tested it primarily on other platforms.

I don't believe for a second that the WebKit, Firefox and Chrome teams have done ms comparisons with tests that don't report data on the ms level. 0ms results stand out, after all. They probably are pretty smart people, and they realized it doesn't matter.

> Any performance tests are better than no performance tests

Nope. Silly performance tests are worse than no performance tests.

The results are either uninteresting and unreliable, or uninteresting and reliable. Shortcircuit logic keeps me from worrying about reliability.

Your shortlogic is going the wrong way. The results are either unreliable, or reliable. That is what Resig was trying to find out.

And that the Windows GetTickCount/GetTime functions have a bad precision is common knowledge.

Yet the SunSpider benchmark is often used to measure browser performance. If, as Resig says, the error rate for any test run in [Internet Explorer 8, Opera, Safari, and WebKit Nightly] would be huge then SunSpider benchmarks don't have a lot of value.

The wrong way? My point was that you don't need to test if the results are going to be uninteresting. Benchmarking javascript is useless because it's gui level code. Even if it takes 100ms to run you don't care.

Measure browser performance.... why? Why do you care how fast your browser is? Your web apps have to run in any browser, so -if- you care you just have to test if it runs fast enough in the slowest browser.

So yes, I'm claiming that SunSpider is trying to solve a problem that doesn't exist (except for javascript compiler writers). Let me quote from the site:

"[Sunspider is for measuring speed of] parsing JSON input, a 3D raytracer, cryptography tests, code decompression"

Great. Fast javascript client-side cryptography. Just what I need.

>Why do you care how fast your browser is?

JavaScript runs outside of the browser as well (I'm using it for my current project). So, yeah, I care about the accuracy of the performance tests.

Well, I'm not a Javascript coder, but you use accurate tick counts for performance profiling, not the coarse-grained time of day clock.

I thought everyone knew that?

I don't know of a way to access those from browser JavaScript though.

Javascript benchmarking: a precise estimate of a wild ass guess.

I have to wonder if part of the problem is that he's running Windows in a VM. VMs are notorious for their odd timing behaviour. He's running the comparison tests in a native installation of OSX, not in a VM.

From the article:

I was worried that these results were from the virtual machine (I also loaded up Parallels but saw similar results to running VMware) so I just loaded Windows XP proper: Nope, the results are the same as using the VM.

Reply on news.ycombinator.com