Comment by jb1991 | Hacker News Reader

Comment by jb1991 | Hacker News Reader

Honest question, just how fast is Julia anyway? Because performance is typically the main reason anyone would want to use it over alternatives. A lot of people justify Julia because, as noted in other comments, it is both high-level yet claims to be nearly as fast as C:

> "Julia is the fastest modern open-source language for data science, machine learning and scientific computing...with the speed, capacity and performance of C, C++..." [4]

That's a bold claim! And there is much evidence to the contrary. I often find that the language falls into a similar trap of many other languages where the creators are evangelists not quite sharing the whole picture. Writing fast Julia code is not always a pleasant experience as you often need to fight the easier idioms. It is marketed as fast, but really, how fast is it?

[0] http://www.zverovich.net/2016/05/13/giving-up-on-julia.html

"What’s disappointing is the striking difference between the claimed performance and the observed one. For example, a trivial hello world program in Julia runs ~27x slower than Python’s version and ~187x slower than the one in C."

[1] https://www.ibm.com/developerworks/community/blogs/jfp/entry...

"We can code in Julia in a way similar to Python's code. However, that code is slower than it should. One way to speed Julia is to take into account the Fortran ordering it uses by looping on j before looping on i in the second loop. We also add decorators to speed up the code."

[2] https://www.codementor.io/zhuojiadai/julia-vs-r-vs-python-st...

"...comparing R's sorting speeds to Julia's is not the complete story, even though on the surface R appears faster, and from a users' perspective, (once the data is loaded) R is still the king of speed."

Stackoverflow has quite a few user posts [3] from folks trying to get their code to perform at the advertised speed vs popular alternative languages. It's not so easy.

[3] https://stackoverflow.com/questions/20613817/julia-julia-lan...

[4] https://juliacomputing.com/

Regarding [0], startup time is a commonly acknowledged issue which affects the 'hello world' benchmark. (Can you even call it a benchmark? Maybe if startup time is what you're comparing then it makes sense, other wise I don't agree.)

I used to call Fortran code from Python for a particularly complex function (Mittag-Leffler function) from Python, I then rewrote everything in Julia and it runs approx 10-20x faster. So there you go, I'm a primary source.

Edit: just for clarity: The Mittag-Leffler function itself runs just ever so slightly faster in Julia than Fortran-called-from-Python (due to the probably inefficient way I was sending and receiving the data to/from a Python subprocess, which itself is an example of the difficulty of combining two languages). It is the program as a whole which runs 10-20x faster.

My experience tells that computationally intensive codes are roughly 10% slower that their C++/Fortran counterparts. So fare we are implementing a data analysis pipeline for a cosmological telescope (LSPE/STRIP), and we are migrating a number of codes my team developed for the Planck/LFI instrument. As data formats and a few details of the instrument changed, we decided to rewrite everything from scratch instead of manually fix and recheck every line of code, and we chose Julia 0.6.

The 10% figure comes from a comparison of the performance of the codes whose logic has been left virtually unchanged, of course.

What is awesome is the ability to run codes within Jupyter (like Python), and the ability to run "for" loops on large arrays fast and with little memory consumption (unlike Python). The 10% increase in computational time is fully absorbed by the reduced development/validation time!

This is mirrored in my experience with dynamical simulations and mathematical optimization work.

This 10% is due to some aliasing optimizations that haven't been added to the compiler yet, at least for the differential equations code we've looked at. But now that the compiler guys have some more free time...

Concerning the "Giving up on Julia" article, it's very dated and dubious. Please read [1].

About the second quote: this is not about Julia being slower than Python; it's about avoiding cache misses. Obviously you should exploit contiguity of the data in memory. Secondly, the 'decorators' like "@inbounds" just ensure that Julia does not redundantly checks bounds, otherwise the inner loop has branches and cannot exploit SIMD.

[1] https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...

That's a fair rebuttal. But if the original author struggled to get fast Julia working, maybe it's a similar experience to the very many users of Stackoverflow who ask how to get their code to run at marketed speed. It appears to not be obvious or idiomatic in Julia to really get the speed it advertises.

Fortunately there is documentation about performance [1], which is about everything from type stability to SIMD.

[1] https://docs.julialang.org/en/latest/manual/performance-tips...

Last week's discussion on matrix multiply:

https://news.ycombinator.com/item?id=17164737

I guess the claim is that you could write all that in Julia, and hope to be competitive, while you could never do this in (pure) Python. It would still be a lot of work, and would require you to think about the cache, there's no way around that.

The selling point here (it seems to me) is that you can easily doodle up the most naive version of whatever algorithm you're thinking about, and then start optimising to the degree needed, where needed, without having to start over in C or something.

I thought Python offers an entry into the performance space via Cython, which seems to nearly map directly to C. Is Cython not a viable alternative to Julia?

Yes, this is aimed at the same people, I believe. Doodle in Python and then work harder on the critical part.

I think this works best when you have some loops pushing arrays of Float64 around. And worst if you'd like to pass some library a million little functions to optimise which depend on some strange data type you just defined... Cython is quite a limited sub-language. But certainly many current Julians made a living this way in the past.

One of the main issues is that, because you compile things separately with a context-switch managed by python in the middle, if you pass a Cython compiled function to code that is calling compiled code (C/Fortran/etc.), then you still get a huge overhead. We tested this with ODE solvers, and things like Numba+SciPy odeint were still about 10x slower than they should be because of this phenomena ([1] mentions some of our tests). In the end, we found a very good reason to make sure the whole stack can compile together!

[1]: http://juliadiffeq.org/2018/04/30/Jupyter.html

I wouldn't say so. Cython is quite literally C for python, basically making all your python code look like C with most disadvantages, while not always reaching C performance.

Julia in comparison is a fully featured language, with a lot to offer - you can write highly generic + fast code in a very high-level way. E.g. have a look at https://medium.com/@Jernfrost/defining-custom-units-in-julia... You can see, that Julia can be more elegant than python while being a lot faster.

To be fair, the majority of the stackoverflow questions on how to speed up julia are from people brand new to the language coming from Python or whatever. Its relatively easy to optimize julia code, but I think its understandable that someone fresh out of Python might struggle for a little while to get all the performance benefits one might expect.

Especially when you consider what would result from that same user spending that same amount of effort trying to learn C.

I actually could hardly care less about julia's speed. It is fast enough.

What matters to me is that it is actually really nice to write. The syntax is elegant, the community is excellent, and the core of the language's semantics being multiple dispatch is a game changer.

This is absolutely correct. The benchmarks they claim are misleading, and under fair conditions, Julia simply isn't the speed/efficiency solution they market themselves to be.

I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting.

> I think that the ML community's decision to write core operations in C or C++ and provide Python wrappers is the way to go about it if you want the flexibility of scripting.

Isn’t Google’s “Swift for TensorFlow” move in direct opposition to this statement? I think you are right that it gets you 95% of the way, but that the final 5% of performance and portability simply will not be there.

https://github.com/tensorflow/swift/blob/master/docs/DesignO...

It's not just 5%. If you have any API which takes a higher order function, for example a differential equation or optimization package, then even if your code is compiled and the user's input is compiled, you still have to hit Python in the middle, and that context switch can be the most costly part of an optimized code, making Numba+SciPy about 10x slower than it should be. So yeah, Python + compiled code is not a viable solution in all cases, or in fact what seems to be most of scientific computing (but not data science where things tend to not include function input).

http://juliadiffeq.org/2018/04/30/Jupyter.html

There can be a significant hit in performance due to using Python. I generally prefer developing pure C++, but being able to test/prototype in Python is great. pybind11 makes interoperability easy and efficient.

What about using Cxx.jl? You can have a nice little REPL for your C++ code. I haven't seen pybind11, does it give you that ability?

No, but cling [1] does.

I might be interested in trying cxx.jl, thank you for the pointer.

[1]: https://github.com/vgvassilev/cling

Hello world program measures startup time which is not what julia is optimized for right now. I''ve heard that they are looking into it but it is not as important as startup time in python where we have a lot of python scripts as binaries which are executed repeatedly or interactively(like "hg status").

Julia v0.7/v1.0 comes with an interpreter which doesn't have any JIT startup time (because it's an actual interpreter). It's slow because it cannot use precompiled code yet, which is why it's not documented, but it's on the "more to come" list.

Do you have a link to share? Sounds interesting.

It's one of the things being developed on the down low right now, but you can find PRs like:

https://github.com/JuliaLang/julia/pull/24473

which are making this:

https://github.com/JuliaLang/julia/blob/master/src/interpret...

a compilation-free version of Julia. At the same people there is discussion about how to cache native compilation results after precompilation, so that way packages can store all of their compiled code and users can directly use it without compilation through the interpreter. This second part isn't done yet, but it'll be interesting to see what happens when it has a truly dynamic + precompiled form.

That is absolutley false. Look closely at the 'benchmarks' shown in the above article and tell me if they are at all compelling. They measure Julia's startup time and compilation time. IF you wanted to compare Julia to C there, you'd want to include the time it takes to compile your hello_world script.

Julia's performance optimizations have to far been mostly focused on intensive numerical computations where the startup time and compilation time are merely a constant overhead that are irrelevant to high performance numerics.

No, it's not. My comment is unrelated to startup and the issue in this article but more generally about Julia as a language of failed promises.

The biggest issue with the standard Julia benchmarks is that they're not using what would be optimal C code to compare against.

In addition, if you look at the Julia issues on GitHub, you'll find hundreds of performance regressions where code performs more than 10 times as slow as what they expected/claimed at one point. It's not reliably fast, even when written by Julia experts/developers.

> The biggest issue with the standard Julia benchmarks is that they're not using what would be optimal C code to compare against.

Yes, the C code benchmarks are not optimal. Neither are the julia benchmarks or any of the other languages for the matter. The julia devs made a hard decision with those benchmarks and decided that if they allowed arbitrary optimization, the benchmarks would become more a measure of who spent the most time and knowhow writing the benchmarks for ______ language. Instead, they tried to keep the code for all the languages at a reasonable level and avoided super specialized magic. That may make some uncomfortable, but keep in mind that the julia code used in the benchmarks also has a lot of room for improvement. Some julia devs are absolute wizards and getting performance out of julia code if you let them go crazy.

> In addition, if you look at the Julia issues on GitHub, you'll find hundreds of performance regressions where code performs more than 10 times as slow as what they expected/claimed at one point. It's not reliably fast, even when written by Julia experts/developers.

Julia 0.7 (which is still in its alpha build by the way) included a ground-up replacement of julias iteration protocol and a reworking of a ton of code optimization routines. If you think you or anyone else in the world could take a codebase as large as julia's and replace fundamental parts of it without seeing performance regressions anywhere you're delusional.

There are a number of performance regressions, some mysterious and some not mysterious and they will all be worked on because the julia devs take regressions very seriously. It won't be instantaneous, but I do not doubt that the bulk of them will be eliminated promptly.

There are also orders of magnitude more performance improvements in 0.7-alpha than there are regressions, they just aren't filed as issues. Some of these performance improvements, especially with broadcasting, are state of the art and not seen in other languages.

Calling julia a "language of failed promises" because the alpha build of a pre 1.0 version has some performance regressions is sensationalist and disingenuous.

This is ridiculous. On top of eigenspaces comment, the promise was never that any odd Julia code would be fast. You can write untyped/dynamic code and that will be faster than Python but not near C, or you can put a little work into ensuring type stability (from experience this is easier than writing in C from the start, especially as you already have a working prototype that you are iterating in the same language) and you get excellent results.

I was very careful to state this the people writing the code I was talking about were Julia experts/developers, not just anyone writing arbitrary Julia code.

When you're doing that much typing, you might as well write C or C++.

I can see how one might naïvely think that, but in my experience and the experience of everyone I’ve talked to who uses julia, gradually improving the performance in your bottlenecks using only Julia is much nicer that just working in C or dropping down from python to Cython.

You can improve much more gradually as needed, retain all the full language features and have much less menta overhead, needing only to keep Julia in your head.

Julia devs make performance regressions when replacing critical central code components just like everyone else does. The advantage is that julia makes it easier to reason about and fix those regressions

Okay, with this comment your comment on preferring C++(11?) and pybind11 [1] I think I am finally getting your angle. Let’s see if I can bridge the perspectives.

If I am reading you correctly, you are a library builder, an infrastructure creator, and comfortable caring about the bottom line when it comes to performance. You most likely prototype an algorithm in some high-level language, ensuring that it works, then push it down into C++ in order to make it scale to cool problems, lastly you may create bindings to a higher level programming language so that others without your C++ acumen can benefit from your labour. A lot of great code is written this way, some that I rely on in my day-to-day work would be OpenBLAS, TensorFlow, and PyTorch.

I can only really speak for myself, but I know that there are many like me, we are researchers and to us code is only incidental. We are judged based on our ability to churn out as many papers and results as possible, in as little time as is humanly possible (ever wondered why academic code can be absolutely awful?). We rarely know the structure of the solution a-priori, rather, we start throwing techniques at things and try to make the experiments run. At some point, a (wild?) performance bottleneck appears and we just want to get around it as soon as possible. Now, some like myself have been in the Python/Cython world for years and you can get around a lot of bottlenecks this way. However, it comes at the expense of additional boilerplate and mastering which parts of the Python programming model you must throw overboard, not to mention how you make your Cython code interact with pure Python code from libraries that others have written. This is where Julia shines, it allows you to much easier go between this “productive” and “performance” mode, to me, this is worth its weight in gold. It is not for everyone, but if I ever have a law named after me I would be happy if it was “Nothing is for everyone”.

[1]: https://news.ycombinator.com/item?id=17206370

You’re exactly right. Thank you for bridging the gap in perspectives. I used to do a lot in cython but found that the glue code was taking more effort than writing the whole application in C++.

Glad that someone managed the translation. Another note: I'm a trained mathematician, I'm not afraid of a type system (nor are the physicists I work with). In fact I was missing one dearly in Python. If that was all that was needed to write C++ instead of Python we all would.

Yet that's how Julia looks to us: Python + a sensible Type system.

I take it from your GitHub profile that you're an experienced C++ programmer, in which case you're probably correct.

Personally, I still find C++ to be effectively a black box, meaning that if I want to understand it or make changes, I'm at the mercy of the maintainers or willing colleagues.

I do not have to recompile my C program every time I run it. Suggesting I'd want to include that is just silly.

:shrug:

I don't have to recompile a julia function every time I run it so long as I don't close my julia session. If I do need to close and reopen my julia session a lot for some reason, I'd just statically compile the function. In practice, one rarely needs to close a julia session and recompile functions and if one does do it occassionally, the compile time while a little annoying is not too bad.

Compile time will also be actively worked on to reduce it post 1.0 once all the breaking changes are done and pressing bugs are squashed.

That's a sweeping statement without any backing argument. Others have debunked the blog post enough (even if you seem to want to cling on to it).

I know several efforts that are transitioning Python + C Libraries to Julia. It's simply much much nicer and simpler to write fast code in Julia than in the Python/C paradigm.

This has been my experience as well: Python-style Julia code often does not run faster than in Python. Only of you rewrite it in C-style do you get C-like performance.

This is not necessarily a bad thing. Just an observation from me translating my Python code to Julia.

Reply on news.ycombinator.com