Comment by e12e | Hacker News Reader

Comment by e12e | Hacker News Reader

Parent Full thread

It's great that you're doing this, and listing stuff like standard deviation in the tables -- but I'd say your focus/interpretation of the data isn't quite right. At least provide the option to sort by standard deviation -- as that might well be more interesting than requests/second?

Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC licenses, which I didn't find):

http://zedshaw.com/essays/programmer_stats.html

For instance, you state that:

  > In this week's tests, we have added latency tab (available using the rightmost tab at the top of this panel). On i7, we see that several frameworks are able to provide a response in under 10 milliseconds. Only Cake PHP requires more than 100 milliseconds.

Only cake php requires more than 100 milliseconds on average. But look at Django: Average around 60 ms, standard deviation around 90 ms (!). Not to mention a "worst" score of 1.4 seconds.

Note that the results data is available as JSON if you want to play with different sorting yourself: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

As pfalls said, we're definitely open to different ways to present the data. More sorting would be cool.

Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.

As one of the people complaining about statistics last week (and also, by coincidence, citing Zed's rant), I'm glad to see you are working on it and open to more ideas and improvements!

Also, I like the "sportsmanlike benchmarking game between different communities" vibe I'm getting from all this.

Would be nice if the community helps turn this into the de facto example of how to benchmark correctly.

Now I'll just have to wait and see how Go 1.1 compares ;).

Go 1.1 is looking very strong! Pat (pfalls) just showed me some very preliminary numbers and we are extremely happy with them.

Thanks for the input and constructive criticism, vanderZwan. It has been very helpful to get feedback from yourself and everyone else. I too am particularly happy with the sportsmanlike competition vibe. You have no idea how fulfilling that is to us.

Reply on news.ycombinator.com