Hacker News Reader

Comment by minimax | Hacker News Reader

Their code is on github [1]. I am a C++ programmer so I'm happy that C++ came out on top, but the comparison is obviously flawed (as these sorts of comparisons always are). The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version. See Russ Cox's response [2] to a Google paper [3] of a similar nature comparing C++, Go, Java, Scala for an example of a similar boondoggle.

The R code, for instance, is obviously unidiomatic. If you have a four levels deep nested loop in R, you're doing it wrong, and it's taking every ounce of willpower to keep from spending the rest of the afternoon trying to fix up their code and make it faster.

1. https://github.com/jesusfv/Comparison-Programming-Languages-...

2. http://blog.golang.org/profiling-go-programs

3. http://research.google.com/pubs/pub37122.html

View on HN

The stochastic growth model, which is the workhorse of modern economics, cannot be easily vectorized, and usually requires tight inner loops to get good performance, or rewrite the computational kernels in C. Codes such as Dynare (http://www.dynare.org) do exactly that. In my opinion, this is where Julia shines - it provides high performance in a dynamic language, much at the level of R or MATLAB. Then again, being a co-creator of Julia, I am probably biased, but this is exactly the kind of thing that motivated work on Julia in the first place.

The two things I like about Julia are actually using it and all the smart people I have met who are also using it. It's awesome. After playing with this particular problem a bit, I do see how it is a pathological case for R due to the recursive nature of the value function. I still think I could do better than 500x slower than C++.

The stochastic growth model is the workhorse of a narrow subfield of economics. It is not in any way representative of the research we do. Further, you will not find any examples that are better suited to compiled languages or less suited to using R than the one in this paper.

To my knowledge, value function iteration isn't even used much in this area of research, so the claim that the stochastic growth model is a "workhorse" is a bit misleading - the stochastic growth model gets used, but it is not solved very often using the methodology in this paper.

Hi Viral, do you have an intuition for why Numba would be faster than Julia here? It appears Julia spends a lot of time calculating the log on line 71.

(not Viral, but) yes - log was the first thing that came up when this paper was discussed on the Julia mailing list. Julia's openlibm is intended to provide cross-platform accuracy and consistency, but the log function turned out to be slower than log in the system libm used by C++ and Python. The other big reason that Numba was faster was GC performance. There is an outstanding pull request adding incremental GC to Julia, which should help when it is fully integrated, and several other improvements are planned.

>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version

One of they authors addresses this in the comments on the OP:

> As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.

> Yes, we know how to code better and prettier R code with the plyr package and related tools. Again, that was not what we wanted to do in this exercise. But we will be thrilled to see alternative versions on github. That is why we posted all our code there.

> My personal guess is that, given how large our capital grid is, plyr will go less far than some people would have forecast, but I’d be happy to be proven wrong.

And if zissou is correct below in his assertion that the "vast majority of economists are horrible programmers", a comparison of what a naive user would write may be the most useful.

The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests.

They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their tests on consumer desktop general purpose operating systems instead of a more specialized system designed for the task... or at the very least a more specialized OS such as a flavor of BSD or Linux without the desktop and other generalized packages.

I don't really consider this a well done test, and the results are only surface-level... certainly nothing you should base a serious decision on.

JIT'ed Java code is lacking a major facility that truly performant native code has, and that is cache locality. That is, it isn't enough that the code to perform the algorithm is "well tuned" if the data structures are not.

(Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)

While I agree Java makes writing cache-friendly code much harder than C, it is no-way impossible. Nor is any C or C++ code implicitly cache-friendly. Generally, in both cases you must know what you're doing. I also hope some of those problems are going to be addressed in Java 9.

Fair enough. My point was supposed to be more limited, saying that a JIT'ed solution is not a cure all for performance considerations. Sure, it is great that there is a JIT compiler helping out. It just is not necessarily enough on its own to even the performance gap between native and not.

It is enough. You can perf test the same non-trivial program (such as a biz application) in C and Java, and given you allow the JVM to warm up, discover hot spots, and JIT your code... they will run just about the same performance level... every time. The JVM is impressive.

JVM is impressive until closures are involved, in which case when inlining is not done (either because it cannot be or ever-changing heuristics are not met), you can drop back to virtual calls and performance suffers a great deal.

Compare to C++ where closures can be used with virtually no penalty and aggressively optimized. Even C with its trick of passing an extra *void for environment comes out shining performance-wise in comparison.

Edit: There's a good discussion of the problems that JVMs have in this area at http://www.azulsystems.com/blog/cliff/2011-04-04-fixing-the-....

"The Problem is a hard one to solve; (i.e. it’s not the case that we’ll wave our wands and do a quick fix & rebuild HotSpot and the problem will go away). John Rose, Doug Lea and I all agree that it’s a Major Problem facing JVMs with long and far reaching implications."

Maybe progress will be made soon though, with more pressure caused by increasing use of closures in the new streams library and with anonymous function syntax available now.

Actually, C++ has worse problems with this. Consider separate compilation and dynamic library loading. There is no way a static C++ compiler can inline accross dynamic module boundaries nor megamorphic call sites when the called function tree is not inlined completely up to the caller level. In rare cases, when it is, both Java and C++ do it now.

JVM inlining is unmatched by static compilers now and is going to only improve once techniques described in the Azul paper are used. Sure, JVM has a ton of other problems compared to C++, but inlining is not one of them.

BTW: I agree with the point that JVM optimizations are not enough and you often have to know what you're doing and optimize by hand appropriately.

Only if you do an impressive job of designing the data in such a way that you aren't constantly thrashing. Which, more and more, means non idiomatic Java code. Consider, even with massive advances in JIT capabilities, you do get performance benefits from an array over an ArrayList.

Now, can this be improved? Of course. Scala, for example, does a good job of not requiring primitive arrays. This is done with compile time tricks, though, not runtime (JIT) ones.

They also didn't account for the possiblity that a Java programmer would use an ArrayList<Double> instead of double[]. You can't JIT that away; JIT does not change the semantics of the Java code so you have to code Java in a sympathetic way. I can imagine this may be more idiomatic in numeric intensive fields though.

The authors do have a comment on the Github project that code needs to be allowed to run for a while so the JIT can kick in, although I didn't see any statement of how they did that. The paper also mentions a decision to turn a loop body into a function to allow a JIT to optimize things.

But, generally, I'm not interested in whether "for" loops run faster in language X or language Y. I am interested in whether some languages encourage code styles that happen to be easily optimized (I'm also interested in whether some languages encourage code styles that are easily maintained, easily extensible, or easily run into a ditch).

This paper is meaningful to somebody, but it's clearly written by people who spend most of their thinking about things other than programming. That's not an insult; it's a simple fact that a lot of programs are written by people who are experts in other fields. I happen to work with a former physicist who has experience writing programs to run on one of the nation's largest supercomputers. The depressing thing (to me) is that while he knows how to write numerical programs that run close to a computer's theoretical maximum speed, that information is only well-known among a small group of people (namely people who have to show that if they're allowed to use a supercomputer, they won't make it run wasteful code). I wish it were shared more broadly.

>JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed

No way. It may be native C, but it's not native C you've hand-coded yourself.

C doesn't enter into it, any way you look at it. Unless you are using some obscure Java -> C compiler.

No, the optimizing compiler inside the JVM does produce a lot of native code for your system... c/assembler. It just has to discover hot spots then JIT them.

> One of they authors addresses this in the comments on the OP:

>> As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.

Sure, but this makes the paper less useful for answering the question, "which programming language should I use?" Or, "which programming language should I learn better?"

But, yes, most economists are pretty bad programmers, so "which language should I use poorly?" is probably a useful question to answer.

Which language should I learn to use better? is probably a useful question to answer.

We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.

This is rather half-baked reasoning. The only ways I can think of to control for programmer skill or at least account for it is to

    1.  Use "idiomatic" code for all languages.
    2.  Use "optimized" code for all languages.
    3.  Use "real" code for all languages with a large and detailed sample set.

In number 3 you aren't "controlling" the variables so much as diligently including them in the observations. The baseline skill of a programmer sounds to me more like an unknown than a variable-- something more worth studying first than controlling.

The closest to #3 available seems to be -- "Plat_Forms - The web development platform comparison"

https://www.plat-forms.org/

It is unfortunate that the authors of this paper didn't reach out to experts in each programming language in order to write high-quality code against which they could have run the analysis. This seems like the perfect use-case for open-source: academic study, subject-matter experts in one area who are stepping outside their areas of expertise and an interesting subject matter.

They should have crowd-sourced the code for this study with a Top Coder-like contest.

I don't know about other languages, but the authors did reach out to us to sort out performance issues with the Julia code.

That is very good to know. That reassures me a little.

I was really impressed with the speed difference they noted between Python and R and the fact that Numba wasn't too much slower than C++ implementations. However, given the state of the R code I became wary of accepting the benchmarks at face value.

I haven't made the jump from R to Python for statistical applications yet. So, this article gave me something to think about. In terms of programmer happiness, I'm willing to live with a little slower performance in Python and not have to spend hours in C++.

I've used Numba to speed up some tight numerical computations written on NumPy arrays and it's incredibly good - about as fast as rewriting it from scratch in Cython, but much, much faster.

The flipside is that it only really results in a speed up for pure numerical code - if you have to use any Python objects, then you lose everything.

From the paper...

Second, to make the comparison as unbiased as possible, we coded the same algorithm in each language without adapting it to the peculiarities of each language (which could reflect more about our knowledge of each language than of its objective virtues).

They have invited others to fork the code on github and write the fastest possible for each language.

Thank you for the note. I hadn't visited the github yet.

I will be interested to see whether people can whittle down the respective benchmarks.

^^ This is not "unbiased"... in their attempt to make it unbiased, they introduced a huge amount of bias.

They wrote the code for each language based on prior knowledge about each language, and/or from likely googling how-to's... far from performant code with any language.

A mild understanding of each language could yield dramatic performance improvements over not understanding it at all.

This is not "as unbiased as possible" -- it is biased towards the initial form of algorithm description.

To make the comparison "as unbiased as possible" would be to code the same algorithm but adapt it to the peculiarities of each language implementation.

It is also unfortunate that several of the language developers and related dev communities have come out against compilers (python and jython in particular). This is the (religious) downside to open source fundamentalism.

PyPy is a counterexample: it's a JIT compiler for Python, pretty efficient.

Compiling dynamic languages is not easy: their semantics of having everything dynamic and changeable at runtime resists it. Python is internally compiled into a pretty compact bytecode, but something like a method call is much more involved business in Python than even in C++, and is many times slower.

WRT fundamentalism: most sane open source communities are pretty pragmatic; Python's in particular.

That would answer the interesting question 'which language gives the fastest programs in the hands of someone who is an expert in that language?'

In fairness to the authors, though, it seems to me they were asking the equally interesting and rather different question 'which language gives the fastest programs in the hands of someone who is an expert in something other than programming?'

http://benchmarksgame.alioth.debian.org/u64q/which-programs-...

Wouldn't that make the results arbitrary - nothing more than a reflection of the ignorance of the participants?

I almost gasped when I saw that their R code[1] does the bulk of its computations inside a while(...for(...for(...for(...if(...){...}else{...})))) block, instead of using functionals like apply and 'vectorized' operations like ifelse.

If this criticism seems harsh, compare the authors' R code to Hadley Wickham's advice at http://adv-r.had.co.nz/Profiling.html -- or just search Google for "speed-up loops in R".

[1] https://github.com/jesusfv/Comparison-Programming-Languages-...

Many of the bioinformaticains I know use R. Not knowing how to use it properly wouldn't surprise me in the slightest. I struggle to get some of them to indent their Perl code.

Given that their code is on github and many people criticized the unidiomatic nature of their code, wouldn't it make sense to set up a competition to improve the code? Everyone can submit pull requests with their improved codes and the authors should simply rerun the codes on a standard machine to get a benchmark. Obviously, a couple of rules should apply:

1. when you improve the code, do not move any elements outside the benchmarked language, i.e., don't use mex files in Matlab, c++ in R etc.

2. maybe create two or three different versions of your code - single processor, multiple cpus, gpu and run it on a single machine with a predefined set up.

This will remove at least some of the criticism thrown at the economists and has an additional benefit: researchers trying to find the best-suited language for their problem can not only use speed but also expressiveness, aesthetics, code length, boilerplate and ease of use to guide their decision.

http://benchmarksgame.alioth.debian.org/

I'm hoping that -- and will be pleased by the irony if -- the HN discussion has less language war than the MR discussion.

It's worth knowing, for economists, how well economists can write code in different languages. But I'm sure I, too, would write better C++ than R, since I've not written any R at all. They would need to ask economists skilled in each language to write the code to get a better answer, and there would still be a lot of noise in the signal.

EDIT last sentence was gibbgersh

It would have been nice to see R with c++. Using the Rcpp package you can write inline c++ for the heavy loops like a boss.

Reply on news.ycombinator.com