Hacker News Reader

Comment by KeplerBoy | Hacker News Reader

Actually ran into this problem this week, toyed around with multiprocessing.shared_memory (which seems to also rely on mmaped-files, right?) and decided to just embrace the GIL.

Multiprocessing is not needed when all of your handful subprocesses are just calling Numpy-code and release their gil anyways.

Also some/most Numpy functions are multithreaded (depending on the BLAS implementation, linked against), take advantage of that and schedule huge operations and just let the interpreter sit idle waiting for that result.

View on HN

I have seen this movie before, it always ends the same way.

Look to go parallel for speed increase, burn a bunch of time and decide its way too hard and just wait the extra few minutes instead.

It's actually the opposite. My current job is making a piece of code faster my predecessor tried to make faster using multiprocessing. I now try to pull stuff back into a single process.

The code for example loads a file from disk in a separate process to have the main process available for number crunching. The problem: This data is then sent to the main process and another copy is created which takes time. Doing the same with threads is way faster, since data i/o operations are releasing the gil anyways.

Actually, you get the speed increase if using threads on python with numpy because numpy will release the GIL internally, so, operations using numpy will actually run in parallel (i.e.: c libraries can release the GIL and do get the performance benefits when using multiple processors -- although yes, when you get back to running python code that won't be parallel due to the GIL).

My genuine deep love for numpy extends to them doing these things without me knowing anything about it.

That's something I worry about when reading about future Python releases without GIL.

Will people just throw more and more threads at unfathomably slow Python code to get somewhat fast results?

Reply on news.ycombinator.com