Hacker News Reader

Comment by zintinio5 | Hacker News Reader

For many users of Julia, long-running performance matters more than microbenchmarks. Having converted a naively written Python program to Julia (there was a huge amount of computation being done over a large search space), I experienced a massive speedup even against PyPy. My Python scripts ran for about 10 hours before I called it quits (.6% of the work had been completed). Converting to Julia allowed me to finish within 3-4 hours, AND it was easy to parallelize.

View on HN

Just how naively was the Python written? ~1600 hours vs. 4 hours of execution time sounds like some extremely naive starting code. Is it fair to even compare them?

I've gotten 400x speedups going from python to c++.

I've gotten 200x speedup going (badly written) Javascript to (better written) pure python, despite python being a nominally slower language according micro-benchmarks. Comparing run time without knowing anything about the code doesn't say much.

Were you using a js runtime without a JIT? Were your python algorithms better? Otherwise a 200x speedup sounds completely unbelievable. That would basically indicate a bug in the js runtime causing degenerate performance under your scenario.

That's very believable.

Example: popping from the beginning of a list in Python is O(N). Popping from the end is O(1). Initial poorly written code can have a lot of such obvious optimizations.

It's a common programming thing - you write your code, it compiles, it runs, it runs properly on your test case, you try it on your full data, it's too slow. You run your profiler and you spot some problems, you solve them, it's still too slow, repeat until fast enough, declare victory.

I replaced some inefficient code with some better O(n log n) code. Make n large and you'll get to 200x. Which was kind of my point, without making sure you've held every else the same, saying I rewrote X in Y and got Z improvement doesn't always say that much.

I've gotten 400(+)x speedups by going from python to python.

Original program was 50min in python, 1 min pypy

After figuring out some inefficiencies that were O(Nsquared), it was 3 seconds in python

This was timing for a case where N was much larger than my normal cases, where run time had been a couple seconds with the original version and optimization didn't really matter.

The Julia version was a literal translation of the Python version, the only difference being that I changed the entry point in the Julia version to allow restarting searches in different ranges. That made parallelization very easy, which is where I got most of time saved. I distributed the work across 3 machines. However, even without parallelization, the Julia version progressed much further than either CPython or PyPy. I also attempted a C version before Julia, but I ran into some issues, which Julia handled without me having to write extra code (int128 support out of the box, and various functions for prime numbers). The int128 support was the main reason I didn't write it in C, as I came across a lot of contradictory information online for __int128 in Clang or GCC.

>AND it was easy to parallelize

This is what is killing me when I use Python. So hard to run things in parallel.

I have had luck with Go trying to do this and set up a concurrent application.

PyPy doesn't get you that much -- were you not using Numpy to begin with, and if not, why not?

Correct, I was not using Numpy. I did not think of a good way to fit my program into a Numpy shaped solution. The main point is that I didn't NEED to try, as an almost literal translation to Julia worked much better. I also did not try using Numba, which I suspect would have given me similar results to Julia. I am a fan of Python overall, but sometimes you just need to choose a tool that gets the job done with the least mental effort expended on the tool. This was a one-time script that needed to be written, no maintenance was needed: I just needed an answer, and within a certain amount of time.

What kind of computation were you running?

Reply on news.ycombinator.com