Hacker News Reader

Comment by xioxox | Hacker News Reader

Just in case someone is interested, I've written a simple fork-based module for processing jobs for Python: https://github.com/jeremysanders/forkqueue

This has the advantage of allowing work done inside a nested function, allowing large initial datasets to be shared and not have to be passed over pickle.

View on HN

See the article for an explanation of why this is a bad idea. Note that on macOS Python disabled fork()-based multiprocessing as the default long ago (Python 3.8?) because it's so broken, and it will stop being the default for multiprocessing on Linux in 3.14, deprecated in 3.12.

It works really well if you avoid threads, and I've found it pretty easy to avoid them in my code.

Yes, but... it's harder than one might think to avoid threads. Merely importing NumPy starts threads, for example.

Yes, and that's a good thing. Let numpy do its threading unless you know exactly why you don't want it to parallelise your matrix multiplications and what havbe you. Numpy releases the GIL so that works exactly as it should.

Threads aren't bad: threads that cause resource contention with the GIL is (possibly) bad. That is almost always done explicitly by the developer and almost never done without your knowledge.

Unfortunately NumPy's thread pool is only for BLAS, the underlying library for numpy.linalg functions, mostly. Other operations are single-threaded. So you need your own thread pool (or process pool) if you want to parallelize anything else.

Yes, I know. But your statement makes it sound like numpy using threads is somehow bad or undesired (it can be; but if it is, you'll know how to tell numpy not to thread.)

Reply on news.ycombinator.com