How to Make Python Run as Fast as Julia
ibm.com
ibm.com
We have written some documentation for Octave in order to guide people towards writing faster Octave code:
https://www.gnu.org/software/octave/doc/interpreter/Vectoriz...
But look at how much we have to explain, and look at all the hoops we have to jump through in Python and Octave in order to write faster code. Matlab used to have similar guides telling people don't write this, write that instead.
What Matlab eventually did was look at what people were writing and making that fast. Julia did the same and made it even faster.
This is a lesson from C and C++ compilers that seems to be taking a long time to trickle to other programming languages: as long as you're writing reasonable code, speeding up your code is your compiler's job, not yours. Your compiler usually knows better than you how to unroll loops, how to cache results, how to use multiple cores, how to elide unnecessary intermediate results, how to completely remove dead code. You should focus on writing easy to understand, maintainable, high-level code. That's why you're using a programming language and not machine code.
I don't have much to offer to Julia on technical terms. Octave's codebase started as pre-standard C++, and it's been a long and arduous job to very slowly modernise it. Julia has the advantage of starting modern.
I think the biggest lesson I've learned is, don't underestimate Matlab. It may be old, it may be an ugly language, but it also is gigantic, very well-funded, and everywhere. Julia and Octave are approaching Matlab from different angles, Julia by providing a better language and Octave by providing a free drop-in replacement, but ultimately we are both trying to replace Matlab.
Matlab users greatly value having a complete package with an editor, debugger, visualiser, profiler, and system modeller all in one. They don't generally see Simulink or the symbolic toolbox as being separate components, or indeed, as the completely separate programs that they really are. For the devout Matlab users, these conveniences greatly outweigh a better programming language. Going further, many Matlab users don't even consider their activity "programming" at all, and in recent years Matlab even writes the code for you if you just push a few buttons.
We're seeing some academics slowly switching over to Julia. This is great. The Mathworks knows very well that captivating young, malleable minds in university is how you get loyal paying customers, namely, the future bosses of these newly-molded minds. Julia is doing a good job appealing to profs. I am optimistic that within five years or so, new generations of Julia-trained graduates will slowly spread the love of the new language.
That's a very good point. Julia's debugger and profiler are both horrifyingly bad.
https://groups.google.com/forum/#!topic/julia-dev/-LTsBVRv1d...
You figure out who's who.
I was merely trying to understand why thebooktocome thought the profiler was so bad.
Have a nice day.
Also your words with the easy usage of "stupid" led me - wrongly! - into the direction that you might not be interested in understanding why users prefer Matlab, but more in expressing your opinion about the topic.
And to be frank, again perhaps due to my weak understanding of English, your usage of "You figure out who's who." and "Have a nice day" to me do not sound as the intention of trying to understand things but more on making clever remarks. But as I've pointed out, this might be due to the fact that English is not my native tongue.
If you mean a packaged script more like a mathematica notebook, I don't believe so.
Relevant to the topic at hand, Julia does have that functionality already, and it in fact (afaik) uses the same engine that's behind iPython, the Jupyter environment. There's even a web-based community version for people to try stuff out: https://www.juliabox.org/
In my experience (15 years), Matlab has been the best for exploratory data analysis. If you change a script, you don't need to worry about re-importing issues like in python; you can even change code DURING a debugging session, EVEN with GUIs. I have been using python+numpy+pandas for a few years now, and python just feels like there is an impedance there that I don't get from Matlab, primarily with graphics/matplotlib. But, deploying and sharing python is easier. I admit it could just be my experience with Matlab that biases me. I usually find that people bitching about Matlab don't like the language, and that is usually because they are viewing Matlab AS the language (like python), and not as a data exploration and analysis environment.
Except in practice, to write fast Matlab code you need very deep understanding of Matlab’s internals and years of experience. Seemingly trivial patterns end up slowing your code down by multiple orders of magnitude, and there’s more “guess and check” involved when trying to write code that can be effectively JIT compiled by Matlab than careful reasoning. Even worse, the JIT and the profiler don’t get along, so it’s often impossible to get any insight into the reasons for JIT-related performance differences.
In general, Matlab is an extremely unpleasant and frustrating environment compared to almost any other language I’ve worked with. The only thing Matlab has on Python/Numpy is a nice quantity of publicly available code for various technical functions. Most of this code is hacky academic prototype stuff, but that’s much better than nothing if you’re trying to follow someone’s algorithm written up in a paper.
The Matlab GUI and tooling is a buggy and unpolished Java turd from the 90s which fits in poorly with any modern operating system.
It's still pretty immature, but promising.
That said, matlab's linear algebra syntax is still better than python's.
In practice you can't rely on the compiler, because you don't know what the compiler is doing.
You certainly can't assume it's going to make the smartest possible decision for all of your code. There's no standardisation for optimisations, and they're often a context-dependent trade-off anyway.
The only way to write fast code is to learn the quirks of the tool chain and profile production binaries to find the bottlenecks.
And yet every time I try to get Matlab users to use something else (Python or Julia) the one thing they almost immediately complain about is the lack of a GUI IDE as good as the Matlab one.
Spyder offers a similar environment. It has a variable explorer, script editor, console, etc all in one window (which I think is what matlab users are looking for).
Pycharm[0] has become my prefered Python IDE about 2 years ago (when they went free commmunity/pay for pro) and it's a way better environment than MATLAB. It now has "Scientific tools" too[1], and I understand some good features, though I haven't used them yet.
The only good thing I still recognize of MATLAB is that most toolboxes work together without (too much?) hacking. Working with Python/NumPy you eventually find stuff that you need to adjust to work with that other library you just found implements that thing you absolutely need.
[0] https://www.jetbrains.com/pycharm/
[1] https://www.jetbrains.com/pycharm/features/scientific_tools....
Let me just clear one thing: I am not trying to represent Julia in any way. I wouldn't be legitimate for that at all. I tried to not write anything negative about the Julia language. Let me know if you think i did, in which case I'll modify my text.
To your point about compilers in charge of speeding up code, I see Numba doing more and more. I hope it will cover all of Python soon.
I think the flaw in the article is when it switches to answering, "did the Julia team [write] Python benchmarks the best way for Python?" Then it rewrites the fib implementation to use a cache, which makes the comparison to the julia version completely ridiculous. It also does all sorts of optimization which clearly deviated from the spirit of the benchmark, naive python vs naive julia.
I wish the conclusion had been written in a way that clearly answered the original premise, should we ditch python for julia? The article clearly showed that there are a lot of good ways to speed up python code. Looking for algorithmic complexity wins (like in the fib example), using cpython, using numba, and profiling all can be used to speed up python code to the level of naive julia code. Which leads to the conclusion, if all you want is faster code there is no need to ditch python for julia.
I made it clear however that I was going to answer a different question: did the Julia team wrote Python benchmarks the best way for Python?
That's all what the post is about: how to run Python code faster.
I will need way more Julia experience to be able to even think about answering the first question. The only thing I am 100% sure is that these micro benchmarks do not help answering that question.
EDIT Also in the scientific fields I've encountered not using the various libraries like numpy is non-idomatic. I personally think that python should absorb numpy into the standard library but effectively people I know treat it as if it were already there.
You can argue that numpy etc are implemented in C. But is the Julia interpretor not implemented in C?
Most of the Julia standard library is implemented in Julia.
Arguments about liking to loop over arrays to operate on them are a matter of taste. I like whole array operations, it's just what I learned first and best (Fortran 9x+ and numpy). Someone else may prefer to write loops (maybe they started doing numerical work with F77 or C). There is no universal correct way to express things -- but there is sometimes a language specific best way and to benchmark ignoring this is not good.
Reimplementing LAPACK or BLAS would be a lot of unnecessary work, but I think that one goal of the language is to be fast enough that a reimplementation would not be worse than the existing versions. (I'm not affiliated w/ the project, though, so I'm just guessing.)
[1]: https://github.com/JuliaLang/julia/blob/master/base/arraymat...
But in general, unrolling loops, caching (some) results, and eliding redundant code rightfully is the compiler's job. That's a different kind of speedup.
I pretty strongly disagree that with the meta-parent that it's the "compiler's job to speed up your code". It just doesn't have enough context to be able to do it effectively. I would actually prefer a predictable compiler to one that tries to do all sorts of magic tricks that can be perturbed by seemingly trivial code changes.
That's not what jordigh is talking about. Let's say you figured out a clever way to get your problem to O(log n). That's definitely your job. But in practice, constants matter - that is, 10 log n is way, way worse in practice than 2 log n, even though they are both O(log n). And that's the kind of optimization that compilers may be better at than you. Once you have convinced yourself that the performance of some component matters, a good principle is to figure out the best complexity class you can (using the "right" data structures and algorithms), then write the most idiomatic code possible.
https://lists.swift.org/pipermail/swift-evolution-announce/2...
Therein lies your problem. If I took some of my python code and literally translated it to Julia would it run like a pig as well? Could I even do a literal translation?
Julia's like scientific Python, but where the naïve implementation also runs fast. "Fast by default.", perhaps.
"type unstable", IO and text processing code is also slower.
You can do something clever in any language. There are plenty of really, really smart people that spend a lot of time writing incomprehensible (to me) Haskell that outperforms C.
The question is, do you have to do something clever to get performant code in the language of your choice?
In Julia -- not often. I've written around 50kloc of Julia; almost all of it is first-pass prototype code that manages to be performant despite itself. The most polished code I've written in Julia is about 100x faster than the MATLAB it replaced.
IMO, the main advantage of Python is its massive library of modules. As a prototyping language, on the other hand, it just seems to me that Julia is more flexible.
It is currently more of a domain specific language, kind of like Matlab, because it's not really optimized for other things and has no libraries in the other domains.
On the other hand, I implemented a prototype neural network in it, and it went very smoothly. Will have to eventually rewrite it in Python though.
You can verify this for yourself at https://github.com/JuliaLang/julia/blob/master/test/perf/mic...
If it's really idiomatic Python 3 to always use arbitrary precision integers for everything, then it's not really Julia's fault that Python 3 makes it more difficult to use performant arithmetic.
I have tried Julia, and really like it, but am still using Python for most of my prototyping. Maybe I should force myself to use Julia for some time :)
Of course, this is currently being worked on in julia as well, but I don't see your point.
- In scientific computing, we can either use better algorithms (yes, that makes a lot of difference) and dropping to C as necessary. The canonical example of the second alternative is Numpy.
In my humble opinion, the expressive power of python is what makes it an excellent language for scientific computing. For these kind of problems you cannot afford to worry about buffer overflows and memory allocations. You need a free mind to think about mathematical algorithms.
My own approach is to use python whenever I can do it and then use cProfile to determine whether to port some parts to C.
- If you are using python in application server, most of the time is spent waiting for data. Mostly, the job of application server is to collect data and make some kind of response which is not CPU intensive.
What you should optimize for in this case is data access patterns and creation of data objects. On the other hand, if you have any CPU intensive work, write it as a separate service outside of your application.
import subprocess
with subprocess.Popen(["julia", "tongue-in-cheek.jl"], stdout=subprocess.PIPE) as proc:
print(proc.stdout.read())Author misunderstands this. If you say that you're optimizing python by calling C, then something's wrong here.
Please let me know if Numpy is supported now in Pypy. I'd be happy to add Pypy in the mix in that case.
What would motivate me would be that Pypy supports the packages I need for my day job, including Pandas, Scipy, and Scikit-learn. Do you know if there are plans to have these on top of Pypy?
tl;dr, Yes there are plans. But funding is needed.
Python 3 has been out for 7 years and I refuse to use anything that doesn't work in Python 3, it's just ridiculous to keep building stuff for Python 2, it's hindering the language and keeping it back in the past.
Also, thinking like that makes more sense about the naming of JavaScript.
On the smaller PR, I had requested fixing the mandel benchmark that in Java is doing lesser work than the Julia and Lua benchmarks, which gives it an unfair advantage. That should be easy enough to fix too - but I didn't get a reply.
Let's get it merged though, and continue the discussion on the PR.
I would agree with the article if Cython and Numba would support 100% Python or Numpy wasn't used to achieve similar execution speed.
Also try IO or text processing in Julia. Python is known to be faster right now.
Aside- Do you know if multiple inheritance/traits will happen at some point? I need this for modeling, even though it can be worked around for general software architecture.
What about the dataframe and stats infrastructure? its currently in shambles. Any Idea when this can be expected to be fixed?
Back when I used Python (early 2000's), that was actually C code, if I remember correctly.
The question is with IO and text processing implemented in pure Julia and pure Python, except for the OS FFI, which JIT compiler provides the best implementation?
def benchmark_sort_numpy():
lst = np.random.rand(5000)
np.sort(lst)
For me the main question is not "will the compiler optimize well in the general case" but rather "will you naturally reach for the right libraries which are optimized well". For me/Python I'd say more often than not the answer is yes. I understand that that's not the point the Julia team is trying to make but it's a decent practical approach (imo)
I don't know how many folks who use MATLAB care that much about loop performance that they will be inclined to look at Python or Julia just because someone found an amazing improvement there.
The one thing that made me finally go over to Scipy from MATLAB was that I changed focus and no longer have to do analysis involving nonlinear/stochastic optimization. Several years ago (maybe 2009?), Octave was really struggling with speed here, and numpy still felt too new. I vaguely remember needing to wrangle with the mathematics a lot more (like approximating the Jacobians or Hessians) in order to get Octave to work. On the other hand I can only remember a handful of times where I needed to get MATLAB to do loops like these benchmarks (like maybe Runge Kutta or FD ODE solutions). Loops are really quite unnatural/unidiomatic in MATLAB, if you can keep your algorithm as close to linear systems as possible it's quite fast.
Has it changed much since? I know right now most of the important nonlin opt algorithms are available in scipy and if you want more there are external packages, but you still have to tinker a bit for the best solution. There's nothing like mindlessly using fmincon for every single problem in the world. I am only a layperson at nonlinear optimization, so I can't tell you why MATLAB is so much better out of the box.
If you care about constrained optimization, Julia has leaps-and-bounds more sophisticated tools than anything Matlab or Python have to offer. Check out http://www.juliaopt.org and especially JuMP.jl. http://www.optimization-online.org/DB_FILE/2015/04/4891.pdf has some detailed comparisons. Macros and fast generic programming make Julia a very well-suited language for doing automatic differentiation (https://en.wikipedia.org/wiki/Automatic_differentiation).
I've recently been writing satellite image processing code, and profiled it thinking the algorithm was the problem. It turned out that even on a SSD nearly 90% of the program time (~30 minutes) was reading from & writing to disk intermediate image files.
More could be kept in RAM, but high-memory cloud machines aren't cheap, and our local development machines only have 16-32GB.
Welcome to academia...
Rcpp is really nice too, and definitely is a win for R.
So, I don't see where you got that from.
What he did was VERY MINIMAL changes. In some cases a mere annotation.
What does he think Cython and Numba are?
> I am not writing any C code either
No, he's just using libraries written in C and Fortran in a ridiculous attempt to praise Python.
> Writing better Python code to avoid unnecessary computation
You don't improve a language benchmark by changing the algorithm. If you don't understand why, you have a lot more to learn before teaching people how to "make Python fast".