Hacker News Reader

Comment by blitzar | Hacker News Reader

I have seen this movie before, it always ends the same way.

Look to go parallel for speed increase, burn a bunch of time and decide its way too hard and just wait the extra few minutes instead.

View on HN

It's actually the opposite. My current job is making a piece of code faster my predecessor tried to make faster using multiprocessing. I now try to pull stuff back into a single process.

The code for example loads a file from disk in a separate process to have the main process available for number crunching. The problem: This data is then sent to the main process and another copy is created which takes time. Doing the same with threads is way faster, since data i/o operations are releasing the gil anyways.

Actually, you get the speed increase if using threads on python with numpy because numpy will release the GIL internally, so, operations using numpy will actually run in parallel (i.e.: c libraries can release the GIL and do get the performance benefits when using multiple processors -- although yes, when you get back to running python code that won't be parallel due to the GIL).

My genuine deep love for numpy extends to them doing these things without me knowing anything about it.

That's something I worry about when reading about future Python releases without GIL.

Will people just throw more and more threads at unfathomably slow Python code to get somewhat fast results?

Reply on news.ycombinator.com