Hacker News Reader

Comment by ctvo | Hacker News Reader

After sadly verifying Microsoft's claims, author goes on to find third party benchmarks where they'd lose to make the point he initially wanted to make.

View on HN

"Well these tests don't meet my expectations so lets keep testing using other benchmarks until I'm validated".

Sounds like FDA approval process.

I'm afraid you're going to get downvoted, but that pretty much is the FDA approval process: throw things at the wall until something sticks.

Authors explanation:

'It seems like Microsoft has been targeting their optimization effort for the competitor benchmarks in order to show impressive results for their new product. When it comes to more intensive and complex HTML5 benchmarks they are still miles behind the competition.'

Reasonable explanation to me.

I think instead they worked in JavaScript performance, so in js benchmarks like sunspider and octane they do well. Meanwhile their page loading was not as optimized, so benchmarks that test loading whole pages such as peacekeeper don't perform as well.

Peacekeeper itself says they focus on js.

http://peacekeeper.futuremark.com/faq.action:

Peacekeeper measures your browser's performance by testing its JavaScript functionality. JavaScript is a widely used programming language used in the creation of modern websites to provide features such as animation, navigation, forms and other common requirements. By measuring a browser’s ability to handle commonly used JavaScript functions Peacekeeper can evaluate its performance.

While it is obvious that the author was hoping to discredit microsofts speed claims, the result on running the original tests was that an agreement on sign but not on magnitude (assuming the uncertainty on the MS claims is entirely in the last digit presented).

The new tests don't seem particularly hand picked given that the are among the top results I get when I google for "Browser speed test", arguably a sign that they aren't overly obscure.

As an aside, all the tests benchmarks are made by parties other than microsoft so they should count as third-party. And their being third party is typically seen as a virtue due to the tendency of vendors overturning on their own benchmarks.

So while this was obviously a bit of hostile benchmarking, it doesn't seem like there was any need for special effort to get a discrediting result, simply benchmarking with other suites proved enough. The fact that the author is clearly hostile to microsoft does not in itself disprove the results presented, and is not ground for dismissing them.

Looking around the site and other pieces written by the same nick, I find it unfair to label him as "Microsoft hostile". After all, he has written a quite favorable review of a cheapo 7" Windows 8.1 tablet.

http://cd-rw.org/t/hands-on-with-the-affordable-linx-7-lamin...

This kind of reflexive cynicism is annoying and pointless. Especially because we can just as easily flip it around and accuse Microsoft of cherry picking and optimizing for specific benchmarks.

Neither of these accusations are in any way constructive. I'd be much more interested in real world examples where one browser or another is noticeably slow.

Reply on news.ycombinator.com