The Go Language is faster than the Computer Language Benchmarks Game thinks
klaig.blogspot.be
klaig.blogspot.be
The central complaint seems to be that the language shoot-out uses the 'go' compiler. This doesn't seem surprisingly, seeing as this is what the go website tells users to use.
There is however a gcc frontend for go, complete as of July and still not in many distributions (this isn't a go problem, man distros haven't updated yet to gcc 4.7.1).
So the real headline should be 'Computer Language Benchmarks uses google's go compiler, gccgo is faster'
Now that there is only one implementation for each language allowed, some important information is missing from the site.
Not that they respect that rule of course: http://shootout.alioth.debian.org/u32/benchmark.php?test=all...
2. This is old news. The Shootout was debatable before, but 12~18 months ago it suddenly decided that only one implementation of each language (picked completely arbitrarily and according to no clear cut rules, and sometimes allowing two implementations e.g. Python gets CPython 3, Lua gets Lua, but Ruby gets both MRI 1.9 and JRuby, Erlang gets HiPE and Javascript gets V8) would be allowed after — as far as I understand — a spat with the pypy team.
No, JRuby is a JVM-based implementation of Ruby.
> Not only is it built on a different ecosystem of libraries (anything on the JVM vs anything for Ruby)
Irrelevant, you can make the same claim about any impementation of a language: GHC has extensions to Haskell, why isn't JHC benched as well? IronPython can use .Net libraries but not Python C-API libraries, why isn't it there?
> there are language extensions
Which are not used in the shootout and — again – are irrelevant anyway.
Actually implementations are not fast (or slow) either, hardware is. Not like that matters. When somebody says a language is 'slow' they mean on typical hardware, under some common operating system, using the implementations available, and so on.
Everybody knows this. You're dissembling because your favorite language is slow.
So when somebody says a language is "slow", it's as when somebody says a language is "weakly typed", he picks arbitrary and unspecified parameters fitting what he wants the result to be, and then declares he was right?
> Everybody knows this.
Yes, just as everybody knows what "weakly typed" is (it's just completely different from one person to the next).
> You're dissembling
That word. I'm not sure it means what you think it does.
> because your favorite language is slow.
Not even sure where that one comes from.
That's natural language for you, but if you try to understand what the assumptions are behind a statement then you have a good chance of coming to an agreement.
"It's a nice day today"
"Actually, a day is a unit of time it's the weather that is nice"
That's what you are doing.
The tests in the benchmark game are totally useless anyway. When given a load that does not stress the L1i, modern CPUs perform pretty much completely differently than they do on real loads with more than 32kB of code. Given such a small code snippet, the CPU will never fail a (theoretically predictable) branch, will never spill L1, and, on SNB, will in fact never even have to decode an instruction after the uOP cache is primed.
Even if that's the case, one would assume that would hold for ALL languages, so you get a fair comparison at that...
[I'm not saying this about the L1i cache case specially, which is a good point, but rather for the prevalent response to any benchmark in Go-land, a defensive attitude which I have not witnessed in any other language community. Usually Python, JS, Ruby, Rust etc guys get on to fixing such microbenchmark behavior or explain why it's as it is. Go guys just propose you forget about it and rewrite your code in another way].
I don't care about a specific real world case of getting from A to B, or how it can be done faster in another way. I care about measuring sprinters. The case that's important here is benchmarking itself. How fast each and every instruction of a similar type executes in a language.
E.g if I do:
for i in range(10): print i
and
for i:=0; i<10; i++ { fmt.Print(i); }
I don't care if this kind of code is not representative of an actual program, I don't care if the code inside the loop might take more time in most cases, I don't care if I can write some particular program using some other structure.
I only want to know why this takes X time in Python and c*X time in Go, and if the c factor can be improved.
Suggestions about "real world programs" and "write this another way and then measure" in this regard are counter-productive, because they focus not on raw benchmarking the language but on specific cases.
A notable example is C++-style templates (typically every codepath is expanded independently) vs java-style generics (one codepath with checks for types). C++-style absolutely dominates in small benchmarks, because it provides the best possible code. However, if you use a lot of them on different types, it's very easy for them to totally ruin your performance due to resource exhaustion. Some of the very largest performance gains I've ever gotten from a small optimization (on the order of 10x real improvement for a spot change to a few dozen lines, with no change to algorithms) involved removing C++ templates, storing tags, and adding a few case statements based on them.
> I'm not saying this about the L1i cache case specially, which is a good point
Note that L1i is not the only, or even the biggest offender. BTB slots are another very critical one. Predicted branches are essentially free on modern OoO cores, and business logic often contains inhuman amounts of what's essentially if trees. Have enough branches in your main loop that the BTB overflows, and thanks to LRU replacement, all those ifs turn from costing half a cycle to costing ~10 cycles or so. Properly optimizing for that case is very different from optimizing for some microbenchmark with a 100 bytes of code.
And again, this is not praise for on an indictment against some language. I'm not saying that C++ is slow, it was just an easy example of a case where things change when the codebase gets bigger. What I am saying is that the performance shootout is essentially useless, and promotes the kind of optimizations that either don't really help, or even hurt speed in the large. If performance actually matters to you, you are much better off measuring things like competing mature XML parsers or the like. While there are a lot of things wrong with using that to measure the speed of your favorite language, it's still less wrong than using a five-line microbenchmark.
Please tell us where we can see that comparison!
>>less wrong than using a five-line microbenchmark<<
The trouble with that hyperbole is that someone's already mentioned their five-line microbenchmark in this discussion -- http://news.ycombinator.com/item?id=4544096
The alternatives to the benchmarks game that people come up with are usually worse, not better.
meteor-contest Go -- 0.16s
meteor-contest Java 7 #2 -- 0.22s
One example of use ;-)
Now,the default Go compiled program is going to fire right up and get going while the JVM is still warming up. (Yes, I know there are countless things one can do to make Hotspot do amazing things) But the average default user? No... they're still going to say "the JVM is slow" and from their perspective, they're not wrong... just not knowledgeable enough to configure a ton of settings.
It looks like HotSpot VM is preety much optimized and that Go still needs a lot of compiler optimizations...
I know that this test isn´t representative and complete, but it is a good smoke test for first comparison. C and Go should be faster than Java in pure integer numerics (or at least as fast as Java)...
No it isn't.
http://www.oracle.com/technetwork/java/hotspotfaq-138619.htm...
http://www.azulsystems.com/events/javaone_2002/microbenchmar...
http://www.ibm.com/developerworks/java/library/j-jtp02225/in...
I am a bit of a fan of dark backgrounds but you got to be careful with the colours for links etc. Dark grey on Darker Grey is not very visible.
Additionally the article is not clearly laid out and if it didn't have "Go" in the title would not have had a chance at high front page spot on HN.
Oh well such is life at the moment.
After having spent a little while looking over the site, I've got to give igouy props for fixing many of the worst problems that existed on the site a few years back (some of the micro-benchmarks that used to exist were questionable, at best; some of the implementations were even worse). It is a far better presented, qualified, and curated site than it used to be.
I still don't think it's a useful site and that people would be better off ignoring it, but with the changes made, I don't think it's a harmful site anymore.
"#1. To show working programs written in less familiar programming languages"
>>trustworthy<<
Are you saying that you believe the measurements to be falsified?
Do I believe that your measurements are falsified? No. But generalized benchmarks (that is, without meaningful context to the problem that you're trying to solve) are truthy, at best—sort of like statistics presented out of context (such as the 47% figure floating around in political circles).
As I said, I think the way that the CLBG is presented now is much better than it was presented as the Shootout. I just don't think that it's useful toward real software development.
The measurements are The Truth about their very specific context -- they are not a Prophesy.
>>limited utility<<
I don't think there's anything on the website that suggests the benchmarks game provides some sort of perfect, definitive and ultimate statement about anything at all.
On the contrary -- "Here you'll find provisional facts about the performance of programs written in ≈24 different programming languages for a dozen simple tasks."
>>useful toward real software development<<
That would be depend on how well informed the "real" software developers are, and there seem to be plenty of programmers with strange ideas about languages they haven't used.
My guess is that your mistaken view of what the benchmarks game promotes reflects a lack of familiarity with what the website says about itself on the homepage and the Help page and the Conclusions page and ...
The "benchmarks game" is largely the same as when it was called the "language shootout" now with more caveats plastered about. Please explain to me which language tribe I am promoting by calling the "benchmarks game" by its original name? Is it the tribe of skeptics? Perhaps use an HTTP 301 to redirect <http://shootout.alioth.debian.org/>; to <http://microbenchmarks.alioth.debian.org/>; and expunge "shootout" from all names?
My guess is that your mistaken view of what the benchmarks game promotes reflects a lack of familiarity with what the website says about itself on the homepage and the Help page and the Conclusions page and ...
Instead of guessing what my "view" "reflects", consider what the "benchmarks game" has promoted in TFA:
Title: "The Go Language is faster than the computer language benchmarks game thinks it is"
Well, this isn't what the "benchmarks game" says it measures anywhere (in fact, the site explicitly decries this interpretation) and yet this is what it indirectly promotes in various language communities (and linkbait tastes good).
Conclusion: "So Go should be between C and Clean, smoking scala, clojure, java and lisp."
Again, this isn't what the site says it measures (it measures some aspects of some implementations, not languages) and yet this is how it is portrayed by third parties. I understand that it is difficult to give people a lot of 'easy' data and simultaneously educate them about the reasonable limits of inference based on those data. I understand that the benchmarks game has taken great pains to try to educate its readers and inform their conclusions. I don't know if it will ever be enough -- people (as we can see) like easy answers and aren't willing to 0.) do the work to get more complete data sets and 1.) temper their findings with objective reality instead of drawing misinformed conclusions.
Do you see how the easily-digestible plots (indirectly) promote tribalism over rationality? Do you see how a plot is worth a thousand words?
>>Perhaps use an HTTP 301 to redirect<<
Perhaps that was understood 5 years ago, and there are obstacles to that approach. (Not that I think that's actually why people like to talk about the shootout.)
>>promoted in TFA<<
Here you are -- http://news.ycombinator.com/item?id=4544096
He's talking about this -- https://groups.google.com/d/topic/golang-nuts/KqA1Jlpu2nM/di...
He came up with it all by himself, the same way people make factorial performance comparisons.
>>(and linkbait tastes good)<<
This kind of traffic-less blog post isn't a benefit -- stackoverflow is by far the main source of link traffic, and much more traffic comes from direct search.
>>Do you see how the easily-digestible plots (indirectly) promote tribalism over rationality?<<
Do you see how you can intervene and explain what the plots actually say?
The benchmarks game is a resource. There are enough examples that when someone draws a misinformed conclusion, we can usually show them a counter-example and ask them to think.
That's what Go advocates always say, but, for a static language with tight memory structures, it's not that fast at all.
Now, he does compile with gcc go, which is faster, but using the go tool I frequently find for lots of use cases it's about 50-100% faster than Python, or 2-5 times slower than Java.
Not every impressive -- it probably needs a better compiler with more optimizations and a better GC.
When you point that out the the golang mailing list, it's mostly met with la-la-hands-in-the-ears denial, and suggestions to rewrite your program in a smarter way and micro-optimize it. But I don't want to find the optimal algorithm for something -- I want to compare speed, and for that same-ish algorithms in same-ish languages are fine.
(Though, Russ Cox said the current HEAD has a lot of optimizations and is quite faster, so things might be improving there).
I, for one, found that for some kinds of work I do, Go doesn't give that much of a performance advantage, so I'm exploring other languages.
That said, even if Java gives you 2x, that doesn't mean that you shouldn't complain about Go.
E.g you might like Go's semantics and syntax, or it might have other benefits for you. That you like those things about Go, doesn't mean you don't get to say "shame about the performance".
a) writing in C for performance (and having to setup glib et al to make it comfortable)
and:
b) carrying the JVM with me to run something.
(And, no, I won't consider AOT compilers for Java. I want to deal with reasonably mature and widespread technologies and tooling).
So, there are my constraints. Given those, maybe C++ is the ticket. D is not mainstream enough, and Rust I've played with and I like very much but is still in flux.
Let's be specific. What are you building or considering, where your current options are C and the JVM? How does Go fall down on that work load?
I've got a fair bit of professional experience in both C (~10 years shipping code continuously in it) and Java, and I've spent the last couple weeks in Go --- not long enough to be a true believer, but enough to have some practical experience.