Hacker News Reader

Comment by tanousjm | Hacker News Reader

10.8.1 improved things a little bit, but running time is still short of where it was in Lion.

We only have one Mac in our test suite that can support it, but we're going to try to test the 2011 MBP with Snow Leopard and compare that to the results on the same machine for Lion and Mountain Lion.

View on HN

Would also love to see a test of 10.7.0 as a number of us have already been experiencing battery problems since switching from Snow Leopard to Lion that were (partially) sorted out with 10.7.4 (See: https://discussions.apple.com/thread/3194235)

How many tests do you run with each configuration? I'm wondering what sort of variation these test have, but it doesn't appear to be discussed.

You're right; I'm sorry I missed discussing that. Our testing methodology is to run each test at least 2 times and if the second test is within 5 percent of the first, to average the two results. If there's a greater than 5 percent difference, we run multiple tests until we can determine which result was abnormal and why.

It's not THE most scientific methodology, but it's a decent balance between time and accuracy. In the case of our most recent round of tests, none of the second results were more than 4 percent off the first test.

I'm really pleased to hear that. Given the lack of discussion I had assumed the worst, but I'm glad to be wrong, and sorry to presume otherwise. Thanks a lot for following up.

Reply on news.ycombinator.com