Easy parallel Python with concurrent.futures | Hacker News Reader

Easy parallel Python with concurrent.futures

gist.github.com

50 pointsedward11 comments

If you need this in python 2.7, you can get it from pypi (pip install futures)

Parallel python is almost an oxymoron. Python has a GIL that prevents more than one thread from running Python code at the same time. You can get around that by spawning multiple processes each running an independent interpreter, but that's not as cheap or easy as spawning threads. You need to reuse those processes for multiple units of work, or chunk your units of work into big enough pieces to offset the startup costs. Sharing large amounts of memory between the processes requires copying or shared memory (which also isn't that friendly to use from Python.) Granted copying is harder to get wrong, so it's the recommended solution anyway. If you wanted extreme performance Python is a lousy choice in the first place.

CPython the implementation has a GIL; Python the language does not. Plenty of other implementations (PyPy, Jython, Cython) avoid having a GIL or have options to disable it.

> If you wanted extreme performance Python is a lousy choice in the first place.

Using Python for glue code and C-modules for performance critical work is a popular option.

I'm aware of that, but CPython is by far the most popular implementation. Add IronPython to the list without a GIL as well.

However, Jython, IronPython and PyPy must all variously slow down even single-threaded code without the GIL because Python having the GIL made lists, dicts, and sets threadsafe. The locking (or complex lock-free algorithms or STM in the case of PyPy) is not free.

The Python language is just a poor choice for concurrency. It's also a poor choice for a glue language, as Lua is much better designed for that, but Python seems to have become popular in that role anyway (likely due to a richer ecosystem.) Python, even with all of PyPy's JIT magic is still one of the slower languages. If you're using it for real work (not just as glue) and you need extreme performance, then it's a poor choice in the first place.

Keep in mind that I have a Python day job, and Python has been my favorite language for over a decade. I'm not hating on Python, I'm just acknowledging its failings.

Through Cython you can write extensions that release the GIL to make them easily parellizable. However, given that I mostly work Python in Windows (and 64 bit version), I get that Cython might not be a viable option for your use cases.

As for high performance Python, I always recommend Ian Ozsvalds talks at various conferences. Python can be used to reach high performance, even work well in distributed environments and very paralell architectures. It's owed to the great librarys that were developed to work in those circumstances (Numpy, numba, SciPy, OpenCV[1], to name a few), but it's still something posible within the language. Add that to the rapid development cycle that is posible within it, and the flexibility that it can achieve, and you've got a great language for HPC -- despite it's shortcommings.

[1] I will acknowledge here that OpenCV is not Python-specific, if anyone wishes to point that.

I'll grant you all of those points, if you use Cython to compile your code to native or use a native library to do the expensive parts of your code, then it works out fine. Both of those get around Python's inherit slowness by doing the costly things outside of Python.

Cython isn't a separate implementation: it's merely a language that compiles down to CPython API calls. Therefore, it's bound by the GIL insofar as CPython extensions are.

PyPy does really have a GIL, it's only the experimental PyPy-STM which doesn't.

> If you wanted extreme performance Python is a lousy choice in the first place.

Agreed; assuming the problem is "we need blazing performance here," I don't think you should choose Python.

However, that doesn't preclude looking at improving performance. E.g. I have a python process that takes 10 minutes to run. If I rewrite it into pure C, I can get it down to less than a second, but if I (minimally) rewrite it to take advantage of concurrency, I can get it down to two minutes, I see that as potentially worth it. As in, I need to run this process 7 or 8 times a days, the hypothetical concurrent Python makes a lot of sense. If I need it to run 1000 times an hour though...

I see where you're coming from but I don't think parallel python is an oxymoron, I see it more as the old range() vs. xrange() issue. It's been awhile, but as I recall, there is no reason to ever use range(), but if I see it in someone's code, I'm not going to start freaking out. Similarly, I think rewriting a Python 'subroutine' to something faster when the optimization is found to be necessary is well-covered ground, and is incidentally more essential than parallel python, but I see value here.

I guess I can summarize my position to considering better performance (parallel python) versus extreme performance...

Another Python concurrent library?? To boot one which makes Python look like Java? This.that.that.this.whatever()?

How is this better than a few simple lines of multiprocessing module exactly?

Yes I'm being negative. People need to stop already adding yet more parallel libraries to Python. Contribute to existing 15+ options or stay away. Unless you're GvR (and even that's debatable) you're not helping the ecosystem - you're confusing it even further.

Python is not a good first choice for multicore anyway because it is at least 10x slower than compiled languages so the best you are hoping for is 8-core parallelism to get you back to the performance of single core C (or whatever compiled language you want). If you're serious about performance there is no other way than to abandon Python. Only Numpy (and Pandas) are keeping it competitive, and in a number of fields, it's looking more and more R-like as a prototyping-only language, with production in anything-but.

concurrent.futures isn't really _another_ parallel library... its just the standard libraries higher level interface to both the multiprocessing and threading modules.

See the PEP: https://www.python.org/dev/peps/pep-3148/

concurrent.futures is not new, it has existed for years and is not an alternative to multiprocessing but an abstraction that can be used with threads and multiprocessing.

The purpose is primarily to have a solution to promises/deferreds in the standard library.

Reply on news.ycombinator.com