Hacker News Reader

Recent versions of Python (3.8+) have introduced a SharedMemory class for sharing memory between different processes (see https://docs.python.org/3/library/multiprocessing.shared_mem...).

The implementation may still be a bit buggy though, so use with caution! https://bugs.python.org/issue38119

Thank you, I'll definitely check it out!

There are also some libraries built on top of it that might be useful https://github.com/dillonalaird/shared_numpy

I've been using JAX (https://jax.readthedocs.io/en/latest/) for scientific computing in general (in particular MCMC algorithms), as it's really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases).

The main selling point of the library is automatic differential and compilation to XLA, but I've been using it even when I don't need gradients, as it's really fast (due to compilation). I also really like the random number generator as it's very good for reproducibility.

I've played around with Julia in the past and really liked it, but in terms speed Jax has pretty much solved that problem for me

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax.

You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

I had the same experience about 2 years ago. Maybe it's changed since then? It was nice when you had a pure math function to write, but otherwise seemed to be unreliable, especially in multithreaded and multiprocess situations.

I still have some issues in nojit mode, but jit is most of the time fine. It's still a bit iffy with lists, but most of the time I can use numpy arrays.

TBF I am mainly using it for mostly pure path functions.

We have a very large portion of production code written in numba we’ve been running for about 3 years and I made a small contribution to the library. There are a lot of gotchas to numba when the codebase gets large but the benefits far outweigh the downsides. I highly recommend numba.

Edit: also note that a big part of the umap library is written in numba.

I just implemented both a CSV parser and an address standardizer in numba (both CPU and GPU) running in parallel feed through a message queue with a bunch of workers subprocs.

It takes a bit of getting used to but the performance gains on impressive. Basically, my bottlenecks shift from compute to i/o.

I think you have to balance it against writing in C/C++. Mentally, it is basically the same work as writing in C (you manage memory/you write complicated for-loops) but you have good array support with numpy. The primary advantage for me that everything stays in the python runtime environment. You just run the code without any extra steps.

...

What is missing from the timing type 'toy' benchmarks is an understanding that there is typically more than one bottleneck in a real problem and it is easy to choose the wrong one to optimize and get little gains.

After starting C (30 years ago now), spending a long time in C#, then switching to Python a few years ago, I think the unappreciated advantage of python is that I have to abandon all pretense of caring about speed and just get stuff working. It basically solves the pre-mature optimization problem for me by being a fast interpreted language rather than a slow compiled language.

> I think the unappreciated advantage of python is that I have to abandon all pretense of caring about speed and just get stuff working. It basically solves the pre-mature optimization problem for me

I feel the same way. With Python I just write the simplest algorithm that first comes to my mind, even though I know that it is not the most optimized way of doing things. But most of the time I am surprised that it works so fast that I realize I actually don't need to optimize it.

And being able to create and easily manipulate dictionaries and tuples also allows me to create efficient data structures very quickly.

I have a 25,000+ loc scientific codebase in Python, with lot of use in my specialized domain. I gave up on numba, because it's speedups vanish when transitioning to real problems. And secondly, it's bugs are not that intuitive.

I haven't tried numba but I've heard good things about it! Nice linked tutorial. If I understand correctly, you pass in jitted functions (using numba, and jax) into iminuit which does the optimisation?

With Jax you can write native for loops that can also be jitted (I imagine you can also do this in numba?); this can then be really fast. Though in that case you would have to write the optimisation algorithm yourself which is not always practical!

Another big speedup in Jax is due to vmap/pmap, which allow to vectorise/parallelise computation. For example you can build a massive gram matrix really quickly using vmap.

Another point: Jax can also run on GPU (like numba :) ) without having to rewrite anything.

> I haven't tried numba but I've heard good things about it! Nice linked tutorial. If I understand correctly, you pass in jitted functions (using numba, and jax) into iminuit which does the optimisation?

Yes, you can just pass the function and it runs the optimization, in severe cases, you can start by doing a grid `.scan` or `.simplex` (Nelder-Mead simplex method), then `migrad` to minimize, and `.hesse` for 1 sigma bound.

You can also provide a gradient function that, well computes the gradient instead of computing it numerically.

> I imagine you can also do this in numba

Yes! It compiles to native code.

> Another big speedup in Jax is due to vmap/pmap, which allow to vectorise/parallelise computation.

It's possible to compile some vmapped functions with numba too, if the link is any indication, you may see even greater speedup than just jax.grad

However, I do concur that jax's vmap is absolutely fantastic and I found it very useful on many occasions.

Nice to know that numba has these features; I'll have to check it out at some point!

Numpy operations release the GIL (usually at least) so you can use a threadpool and, indeed, share memory. Just try it and you may be pleasantly surprised.

Dask is great if you’re processing large amounts of data, and it recommends and supports threads for this reason.

I didn't know that. So let's say I have 4 regular Python threads calling the same function, and in this function, let's say I call numpy.add on the same array (but different parts of the array), then will it actually use different cores for these 4 different threads? I will try it out, if it works, then that's actually great and would be super easy.

Great, then my comment was worth it!

Not sure there’s anything to prevent you from foot-shooting if you’re operating on the same array. I’d expect that this is only safe to do if, as you say, arrays are not overlapping. Let me know what you find.

The answer is No, isn't it?

I wrote a small code to test it. Unfortunately it didn't work, but I don't know if the problem is me, maybe I made a mistake...

What I do is write single-threaded programs that process only a section of data, and manage them with HTCondor. It's some extra overhead and complexity but this way you can easily scale to hundreds of machines without changing anything. You also get mature queuing and job management tools that work independently of your program. If your application is easy to paralellize and you might need more than one machine i highly reccommend this route.

> We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price.

I understand the sentiment, but in this case I wonder if it is less a python issue and more an issue with numpy. I don't see any reason why numpy couldn't execute the last line of the following multi-threaded:

    >>> import numpy as np
    >>> a = np.arange(1000000000)
    >>> b = 2 \* a

The same thing applies to matrix multiplication. I'm guessing numpy has chosen not to for good reason though.

edit: Actually it seems numpy is multi-threaded under certain circumstances:

https://stackoverflow.com/questions/16617973/why-isnt-numpy-...

Numpy will perform a parallelized calculation when using one of the underlying BLAS routines, if you have the appropriate libraries installed on your machine. So to take advantage of your cores, try to write the calculation as a call to a linear algebra function. Loops or expressions with arithmetic operators as in your example will run on one core. Or, just use Julia.

Try dask

Distribute your data and run everything as dask.delayed and then compute only at the end.

Also check out legate.numpy from Nvidia which promises to be a drop in numpy replacement that will use all your CPU cores without any tweaks on your part.

https://github.com/nv-legate/legate.numpy

> I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily.

Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.

> Also a big downside with JavaScript.

Can't Node run multiple processes (or multiple workers, not sure of the proper terminology) on multiple cores? As discussed in this thread on StackOverflow: https://stackoverflow.com/questions/61893497/node-js-on-mult...

> Can't Node run multiple processes

Of course, but you can also run multiple Python processes.

It’s way different though, node itself is multithreaded (just under the hood)

> Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.

Modern JS engines are all primarily JIT-based, generally only interpreting code when it's faster than waiting on the JIT. JS JITs are quite good now, and in many cases will produce optimized C-equivalent compiled code from fairly naive JS.

Why not use the multiprocessing module? (if you can partition your problem space). I find memory sharing much more difficult to reliably implement an algorithm, and speed gain might be lower through cache conflicts.

> cache conflicts

This can be an absolute sinkhole for performance, but if you have a large dataset you shouldn't be sharing lines that much one would hope.

Then you shd be able to do process parallelization.

A real hard requirement is going to be resource pools such as session linked connection pools with associated data. But there are just plenty of places where you don't want to be forced into a message passing architecture for a service and shared memory may be a better fit.

> Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily.

Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.

I haven't run into any issues with that package honestly (mclapply ftw!). Other than not being able to use it on Windows of course.

It’s process based, so communication overhead is heavy and the infrastructure is minimal - if you’re distributing jobs which are individually large it works fine, but it can’t really be used to replace multithreading in a shared memory model.

I do as much data work as possible in data.table, which parallelizes things under the hood, and doesn't seem to require me to configure anything.

If you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it

The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places where we absolutely need speed. But it would have been so convenient if we could just have native threads, so that we don't even need that 5% C code.

Do you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit)

Also, if you are running on Intel you might have some luck with Intel's profiling tools as, although I've never used them for python they are by far and away the best things in this area, they do support Python.

I don't know what details they give you in from Python code directly, but if you've used perf to profile code at a low level before, vTune is like a Tom & Jerry shotgun in comparison.

> Compiler Explorer

Never seen this before. Thanks!

https://xkcd.com/1053/

Enjoy

You don't have to replace the core python parts. We moved a decent amount of our data pre-processing into Scala, Golang, and Data Base queries and then just store the processed data for the python models to use to train/run (which ends up using the super efficient C/Fortran/Cuda code).

Try Dask